Learning to Count Everything: Transformer-based Trackers are Strong Baselines for Class Agnostic Counting
Authors: Tsung-Han Chou
Brian Schweitzer Wang
Han Ru Chen
Jun-Cheng Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: class agnostic counting
transformer
tracking

Abstract: Class agnostic counting (CAC) is a vision task which can be used to count the total occurrence number of any given reference objects in the query image. The task is usually formulated as density map estimation problem through similarity computation among few image samples of the reference object and the query image. In this paper, we show the the popular and effective similarity computation operation, bilinear similarity, actually share high resemblance with self-attention and cross-attention operations which are widely used in the transformer architecture. Inspired by this observation, since the formulation of visual object tracking task is similar to CAC, we show the advanced attention modules of transformer-based trackers are actually powerful matching tools for the CAC task. These modules allow to learn more distinct features to capture the shared patterns among the query and reference images. In addition, we propose a transformer-based class agnostic counting framework by adapting transformer-based trackers for CAC. We demonstrate the effectiveness of the proposed framework with two state-of-the-art transformer-based trackers, MixFormer and TransT, with extensive experiments and ablation studies. The proposed methods outperform other state-of-the-art methods on the challenging FSC-147 and CARPK datasets and achieve new state-of-the-art performances. The code will be publicly available upon acceptance. 

Under review as a conference paper at ICLR 2023
LEARNING TO COUNT EVERYTHING: TRANSFORMER- BASED TRACKERS ARE STRONG BASELINES FOR CLASS AGNOSTIC COUNTING
Anonymous authors Paper under double-biee blind review
ABSTRACT
Class agnostic counting (CAC) is vision task which can be used to count the total occurrence number of any given reference objects in the query image. The task is usually formulated density map estimation problem through similarity com- putation among few image samples the reference object and the query image. In this paper, show the the popular and effective similarity computation oper- ation, bilinear similarity, actually share high resemblance with self-attention and cross-attention operations which are widely used in the transformer architecture. Inspired by this observation, since the formulation of visual object tracking task is similar to CAC, we show the advanced attention modules of transformer- based trackers are actually powerful matching tools for the CAC task. These modules allow learn more distinct features to capture the shared patterns among the query and reference images. In addition, we propose transformer- based class agnostic counting framework by adapting transformer-based trackers for CAC. We demonstrate the effectiveness of the proposed framework with two state-of- trackers. MixFormer and TransT, with extensive experi-
the-art art transformer-buse ments and ablation studies. The proposed methods outperform other state-of-the the- art methods on the challenging FSC-147 and CARPK datasets and achieve new state-of- of-the-art performances. The code will be publicly available upon accep- tance.
INTRODUCTION
Object counting is popular research topic in the vision community with wide spread of appli- cations, including visual surveillance, intelligent agriculture, etc. It aims to count the occurrence number of target objects in an image. The object counting methods can be classified into two ma- jor categories: class-specific object counting and class-agnostic counting. For class-specific object counting, it usually focuses counting specific category such car, animal, people, etc. where crowd counting to count the number of a crowd of people is well studied by [Song et al. (2021): []: [Cheng et al. 2022 2019) Li et al. (2018). However, it requires train individual model for each cat- egory with tremendous efforts on collecting thousands of training images with annotations and fail work for unseen classes.
In contrast, class-agnostic counting (CAC) studied by Lu et al. (2018): Ranjan et al. (202 ); Yang al. (222) arises recently and aims count any novel objects within the query image, especially for those objects of unseen classes during the training stage. Given several reference object images of the target class, the CAC model able predict the number of occurrences within the query image. Current models share similar network architecture, consisting of feature extractor, matching module, and density head. Once the query and reference feature maps are extracted from the feature extractor, they are fed into the matching module to compute the similarity map followed by a density head to yield a density map estimation. The sum of the values in the density map is used as the final estimated object count. Nevertheless, the major drawbacks of traditional CAC models are their fixed matching framework, which performs template matching in the same fashion regardless of the variations in the patterns of the reference patterns. Moreover, the extracted features for matching are not discriminative enough across categories. Last but not least, traditional models are supervised by pixel-wise root-mean squared error (RMSE) between ground-truth and predicted density map,

---

Under review as a conference paper at ICLR 2023
Query Features
cnt 56.
Quary Image
Stage1
Stago2
Stage3
Densily Head
Matcher
Reference
Reference
Backhone (ResNetVIT)
Feature Extractor
Matcher
Density Head
Figure 1:
A general pipeline CAC models consists of feature extractor, matcher, and density head. Given query image and at least one reference image, the model learns predict density map for counting.
which fails to recognize predictions with respect each ground-truth object as unit. The state- of-the-art model BMNet proposed by (hh et al. (2022) alleviates such weaknesses by introducing: (1) bilinear similarity (or dot-product attention) in matching (2) applying self-attention on feature maps after feature extraction, and (3) additional contrastive loss function. Yet, these revisions are inadequate for more accurate object counting.
In this work, we overcome the above problems of matching and feature extraction by integrat- ing transformer-based object tracking for CAC feature extractor. We first show the connections of the popular bilinear similarity matching module with the prevailing attention modules used by transformer-bast transformer- seaeddrrer--aeedd methods, since the visual object tracking share similar formulation to localize the target object in the query images (or upcoming video frames) as the CAC task, we demonstrate he actually powerful matching tools for
methods proposed by Vaswani et al. (2017). In addition, among different
advanced attention modules of transformer-basee trackers the CAC task. These attention modules allow to learn more distinct features to capture the shared state-of-the. art transformer-basee trackers, MixFormer by Cui et al. (202)) and TransT by Chen et al. (2021), for the CAC task by replacing the original prediction head for tracking with the U-Net-like density head for density map estimation. With extensive experiments and ablation studies, the results not only demonstrate the effectiveness of the proposed methods but also show the proposed method outperform other state-of-the-art methods the challenging FSC- 147 and CARPK datasets with new state-of-the-art performances. These further show the CAC models based transformer-based trackers strong baseline for the CAC task.
patterns among the query and reference images. We verify our idea by adapting
RELATED WORKS
Due to large amount of related works in the literature, we briefly review the relevant ones on class-specific and class-agnostic object counting as follows.
2.1 CLASS-SPECIFIC OBJECT COUNTING
Class-specific counting focuses on the task to count the occurrence number of the objects of specific classes, such as crowd counting. addition, the task usually approached two paradigms: detection-basse and regression-baate based methods. The detection methods, including the methods proposed by Leibe al. (2005): Hsiih et al. (2017), perform explicit object detection over the input image using visual object detectors and then get the count. However, similar to the object detection methods, their performances are also sensitive to the situations when the objects are overlapped, occluded, or crowded. To address these problems, the regression methods proposed by Thanasutives et al. [2021;; Ma et al.] (2021): Cheng et al. (202) instead predict the density map input images where each pixel value can be interpreted the fraction the confidence level of the target object present in the query image. The sum of these values then used as the estimated object count. In addition, the ground truth density maps for training generated by convolving point annotations
2

---

Under review as a conference paper at ICLR 2023
of the training images with properly selected Gaussian kernel. Although class- counting methods have achieved satisfactory results, they usually require to train an individual network for each class of objects and can only be applied for counting the classes that have been seen in the training data.
2.2 CLASS-AGNOSTIC COUNTING
With respect to class-specific object counting, the class-agnostic counting aims to perform object counting for every class. In its setting, the models usually take a query image and several refer- ence object images from the same class as inputs and predict the count of the class appeared in the query image through the density map regression. A typical CAC model consists of three compo- nents: a feature extractor, a matcher, and density head. The model is also supervised by ground truth density map and mean square loss. Lu et al. (0188) propose GMNet which is the first CAC framework, where the reference and query feature maps are extracted independently through the feature extractor based on the ResNet backbone proposed by He et al. (2016). These features are then directly exploited as pixel-wise regression task perform object counting. To better cap- ture the interactions between the query and the references during the matching stage, |aanjan et al. (202T) propose FamNet which introduces template matching for the CAC task. FamNet enhances the matching framework by convolving the reference feature maps across query feature maps. Each pixel value the resulting query feature map represents the similarity between the query image and reference images that specified location and therefore supports better localization. Building upon the existing structures, CFOCNet proposed by Yang et al. (2021) matches query and references fea- ture maps from different stages of backhone for multi-scale strategy. Moreover, CFOCNet integrates self-attention to strengthen the distinct features of the query image. Furthermore, Shi et al. (2022) propose tfate-of-the-art CAC model called BMNet which adopts bilinear similarity for matching. The bilinear similarity is a special case of generalized inner product and provide more flexibility for matching than previous methods using convolution. CounTR proposed byLiu et al. (2022)
troduces rranformer-bbsee architecture into class aansosa counting. Different from Coun'TR, we focus on the relation between the transformer tracking and CAC with template matching and propose a general framework for introducing transformer tracker into CAC model.
Similar to CounTR, since high resemblance of the formulation between the bilinear similarity and the attention module of the transformer framework, we believe more advanced attention modules can help the model focus on important patterns shared among the query and reference images and further strengthen the matching for better counting performance. Since the models of object tracking excel at the task of localizing the target objects from the query images which is similar to CAC, the capability of localization have intrigued our interest to integrate with CAC model. Therefore, different from CounTR which uses transformer encoder for CAC, we adapt the transformer-bas select TransT proposed by Chen al. (202T) and Mixformer proposed by CCen et al.) (202T) to illustrate how transformer-basee
tackers for CAC since In addition,
backhones can perform well on CAC model (c.g.. BMNet), where both trackers have have been specialized in localizing target objects through strengthening target features through self-attention and atoss-attention as means to accurately match and localize target location within the query image.
3 METHOD
In this section, we first introduce the general pipeline for class agnostic counting (CAC) and present the connection between transformer-basee tracking and CAC. Furthermore, we demonstrate the details of how to adapt wansformer-based trackers for the CAC task and to achieve the state-of- the-art performance.
3.1 CLASS AGNOSTIC COUNTING
As shown in Figure I standard CAC models consist of three components: (1) feature extrac- tor which is used to extract visual features from the reference and the query images. (2) a cross matching module which performs template matching through computing a similarity map among the references and the query. (3) a density map estimator which estimate the density map based on the similarity map for counting.
3

---

Under review as a conference paper at ICLR 2023
Given query image, X E IR #xxxxxx3 K reference images, Z = (= #2xxxxx3 and ground truth label, Y E R or Y E Bxxxxxx (i.e., the number or the density map of the target class.), a CAC model aims to solve the regression problem as follows to predict the number of the references occurring in the query image.
min E (L 0,0.1
where Fol.) is pretrained feature extractor (e.g., the backbones of ResNet-50 proposed by He ct al. (016)) or vision transformer (ViT) proposed by Dosovitskiy et al. (2021). etc.). S.(:.)) is the matcher to compute the similarity map between the query and the references, Ro is the regressor (e.g.. fully connected layers) or density map estimator (e.g.. transposed convolutions with bilinear upsampling), and -), -) is the loss function (e.g., mean-squared error (MSE)). The predicted density map. Y. can be easily converted to the estimated count through computing the expectation, C, xxx Wx
j=1 (i, and ji, represents the fraction of reference objects occurred at the pixel
location (i,
3.2 GENERALIZED MATCHINN MODULE
To precisely predict the occurrence count of the reference class in the query image, the prevailing
CAC models developed by Yang et al. (202T); Shi et al.] (202)); ; Ranjan et al. (202I) have pointed
the important role of the matcher. Given the query feature f4 (X) R and the reference features f, (Z) x prevailing CAC matching frameworks usually realize the template matching as the dot product of the reference features and the query feature to compute the similarity map. E xW,, Without loss of generality, for the illustration purpose, we follow the setting of BMNet proposed by [Shi et al. (2022) in the rest of the paper. Moreover, we reshape f4 as fg E dd, xW,) and perform the global average pooling upon fR along the spatial dimension which results in f. E IR Kxd and let f-(()) E R be the feature of (-th reference. Then, can compute the similarity map, π as follows:
K
1 K
where x is further reshape from 1D vector 2D map for the following density map estimation once the computation is finished. It is worth noting that distinct methods process fy and f. differently.
Bilinear Matching: The state-of-the-arr CAC model, BMNet, further introduces the bilinear matching for better performance which is a special case of generalized inner product.
K
T BM
Σ
(W + hg) + b,).
(3)
K
1=1
where W,, and W, are trainable transformation matrices, and by and b, are trainable biases. By replacing fixed template matching with bilinear matching, it allows to model flexible interactions among the query and reference features. Furthermore, we can also perceive bilinear matching as to project the features onto a latent space and to better captures the similarity among the query and reference features.
Cross-attention Matching: Recall the attention module of the transformer which is computed as follows:
Attn K, V) = sof softmar( QKT V
(4)
where Q, K., and V are the query, key, and value matrices, respectively. d is the feature dimension. We can easily find the bilinear similarity shares high resemblance with the dot-producct attention (W, (W.f++,,) and V as the identity matrix. Inspired by this observation, we thus propose to replace bilincar matching with self-attention and cross- attention modules in the literature of transformer since this allows us to exploit various advanced attention modules for versatile matching. On the other hand, before computing the final similarity stack multiple attention
computation by letting @ = (W,fg +b,)
maps, let Q (W, +bb)" and both IC and V (W, fr + b,,)T, and layers to amplify the reference features in regions that share larger similarity between the references and the query for better matching.
4

---

Under review as a conference paper at ICLR 2023
1x1 Conv (64)
1x1 Conv (64 -> 32 1)
Density Map
Stage1
CR
Up
Stage2
1x1 Conv (128)
CR x
Concat Channels
Features after Matching
Bilinear Upsampling
1x1 Conv
Stage3
CR
(256)
3x3
ReLU
Figure 2: U-Net-like Fusion of Multi-Scale Features. We utilize the expansion path (decoder) of U-Net, which fuses features among two adjacent scales with learned convolutions, Numbers within convolution blocks denote the output channels.
3.3 TRANSFORMER-BANE TRACKING CAC
As shown in the previous section, attention modules in the transformer frameworks could be used to compute the self- self-similarity and cross- sooss-immilartty among the reference features and the query feature. Recently, Liu et al. (2022) propose CounTR to adapt transformer-based architecture onto CAC mod- els and achieve the state-of-the-ar performance. We concurrently propose to utilize iitention-based modules that enhances both feature extractor and matching. Furthermore, we claim that transformer- based trackers better candidates for object counting since they not only excel localizing target objects in the next frame with robust features and matching but also the setting and architecture of transformer- based trackers are similar with existing CAC models, which contain a feature extractor, matcher, and prediction head. To support our idea, we take two recent state-of-thc-as transformer based trackers, TransT proposed by Chen et al. (202T) and Mixformer proposed byCCui al. (2022). for further studies to demonstrate the practicality of exploiting the modules of transformer based trackers for the CAC task. We mainly use their backbones and matchers with the pretrained weights and replace the prediction head for tracking with our density map estimator with the similar de- sign as the decoder of U-Net as shown in Figure 2 More details of the density head can found in Section 3.4.1 The adapted trackers are denoted as TransTCAC and MixformerCAC. A brief introduction of both trackers is shown in the Appendix.
3.4 IMPLEMENTATION DETAILS
Then, we show how to adapt the existing pretrained transformer-based tracker for CAC and the training details effectively train the models.
3.4.1 MULTI-SCALE ENHAACCMENT FOR MATCHING AND DENSITY MAP ESTIMATION
After features and matching, most of prior works adopt simple density head consisted with several upsampling and convolution operations on single-scale feature maps and achieved acceptable per- formance. Yet, utilizing feature maps from the last layer/stage as BMNet and stacking convolution layers are insufficient to capture small objects. To preserve the information of fine features from early layers/stages across scales, we respectively perform matching on features across stage 1, 2, and 3, where the feature dimensions are decreased by half as the stage progresses. Then, we utilize U-Net-like expansion path as the density head to fuse query and similarity-relater features across scales. After fusing feature maps, utilize x 1 convolutions compress multi-chann feature maps into a single final density map as shown in Figure]
3.4.2 NETWORK ARCHITECTURE
Here, we further explain the details of which network components of TransT and Mixformer are used to substitute for the corresponding ones in the standard CAC models. For TransTCAC, we utilize the ResNet-50 backbone and the featureFusion matcher from TransT. We further tune the featureFusion matcher of TransT by setting the number of the encoder layers and compress the input features to the matcher with 256 channels. For MixFormerCAC, we use the pretrained Mix Formerrr222 and replace the prediction head with our density head for the CAC
and heads 1,
5

---

Under review as a conference paper at ICLR 2023
task. Regarding the multi-scale version of TransTCAC and MixFormerCAC, follow the same architecture definition of stagel, 2 and as used in MixFormer proposed by Cui et al. (2022) and for our TransTCAC, denote the features from layerl, 2 and 3 from the ResNet-50 backbone of Trans'' as stagel, and 3, respectively.
3.4.3 TRAINING DETAILS
Training Data Preprocessing FamNet developed by Ranjan et al. (2021) initiated conventional data preprocessing for the FSC147 dataset. During training stage, reference images are resized to the size of 128 x 128. Query images are processed with (1) zero-padding image dimensions multiples of 32, then (2) zero-pads images to match the dimensions of the largest image within the batch. During evaluation stage, reference images are processed similarly as training but query images not zero-padded. Given that the image dimensions from the FSC147 dataset vary from 384 to 1,584, excessive paddings are imposed have the same image sizes within the same batch. The excessive zero paddings could mislead transformer-based models as they learn pixel relationships and localization globally through attention blocks. We then propose a new dataset protocol for training. Instead of adding zero-paddings, query images are (1) randomly flipped horizontally and (2) randomly cropped to the same size of 384 384. To keep the aspect ratio of the reference images, reference images are then zero prro-ppdeed to square dimensions and then resized the resolution of 128 x 128. During the evaluation stage, the reference images are processed in a similar fashion as training. while the query images fed into the model without cropping. Concurrently, CounTR, another transformer-hant based CAC model, performs the same query image setup by replacing padding with random cropping.
Training Loss Currently, most CAC approaches based on density map estimation employ the pixel- wise L2 loss (or MSE loss) to train the models where the ground truth density map is generated by converting the point annotations of each training image through convolving with Gaussian kernel. However, the counting performance is sensitive to the hyperparameter selection of the Gaussian kernel since it greatly affect the generated ground truth density map. Recently, Ma et al. (2021) propose generalized loss (GL.) which uses optimal transport (OT) measure the transport cost between the predicted density map and the ground-truth point annotations directly. In addition, it also has shown that the L2 loss usually results in suboptimal solutions and is a special case of GL. Thus, we mainly exploit GL to train the proposed models.
Optimizer and Training Hyperparameters Our model is trained with 300 epochs, batch size of 8, optimized with AdamW optimizer, and weight decay of 5e-4. For both TransT and Mixformer, we leverage the tracker's pretrained backbone and matcher onto CAC model. The backbone learning rate is lc-4 and con-backbone learning rate is lc-5. We use Pytorch implementation framework. We as well train on generalized loss function with a ground-truth density map downsampled to 1 ratio in terms of original height and width. We follow the default setting of generalized loss, utilizing mean-absolute error (MAE) as the point-to-point loss and mean-squared error (MSE) for pixel-wise loss. The hyperparameters of the loss function are set as scale 0.6, reach 0.5, blur=0.01, 0.1.
EXPERIMENT
In this section, present the details of experimental setup and then show extensive evaluation results and ablation studies of the proposed approaches on the widely used CAC benchmarks.
4.1 DATASETS
We utilize two datasets to evaluate our model performance. To compare the model performance, FSC147 dataset collected Ranjan et al. (202I) a standard class-agnostic object counting dataset composed of 147 categories and 6,135 images across different scales, each image annotated with three reference objects. Given that the dataset contains decent annotations, we use the FSC147 dataset to train and evaluate our models. To ensure the generalizations of our models unseen dataset, we also evaluate our ESC147-traine models on CARPK collected by (Hsich et al. (2017), a dataset that contains 1,448 photos of cars in parking lots from bird-view perspective.

---

Under review as a conference paper at ICLR 2023
Methods
Backbone
Pre-
Val
Val
Test
Test
trained
MAE
RMSE
MAE
RMSE
GMN Lu et al. (2018)
ConvNets
MSE
29.66
89.81
26.52
124.57
FamNet
202
ConvNets
MSE
23.75
69.07
22.08
99.54
CFOCNet Yang
2021
ConvNets
MSE
21.19
61.41
22.10
112.71
BMNet et et (202))
2022)
ConvNets
MSE
19.06
67.95
16.71
103.31
BMNct+ Shi ct al
(2022
ConvNets
MSE+CL
15.74
58.53
14.62
91.83
CounTR. Liu
(2022)
VIT
MSE
17.40
70.33
14.12
T08.01
CounTR
2022))
ViT
MSE
13.13
49.83
11.95
91.23
BMNet+ Shi ct al.
2022
ConvNets
GL
15.12
53.62
13.50
91.60
BMNct+*
Shi ct al.
2022
ConvNets
MSE+CL
17.37
61,36
15.75
94,44
MixFormerCACIOuus
CVT
GL
11.17
46.69
11.82
92.78
TransTCAC(Ours)'
ConvNets
GL
11.66
43.91
12.46
82.89
Table 1: Comparisons with state-of-the-ard CAC models, and CL denotes contrastive loss
4.2 EVALUATION METRICS
We follow the same evaluation protocol as the previous CAC works, which evaluate their model performance based MAE and RMSE. For dataset of N testing samples, the vector of ground- truth counts denoted as C E RR,, and the vector of predicted counts denoted as CP E RRN.
N
MAE
C
C RMSE
C -
(5)
N
4.3 EVALUATION RESULTS WITH OTHER OATEE-FFTTE-S ART CAC MODELS
Model Comparison We compare models with other CAC models the FSC147 validation and test sets. For the 3-shot counting task, we compare the proposed models TransTCAC and Mix- formerCAC with recent CAC models. We are specifically interested the comparison with B.Nett+ (officially current state-of-the-an and CounTR (recently released state-of-the-art). Given that our model does not perform test-ting test-time normalization, we categorize CounTR into two settings, where the plus version integrates test-time normalization. We denote beside the method name as the model performance under our dataset protocol. We especially run BMNet on our dataset protocol for the fair comparison purpose.
Quantitative Results As shown in Table 1 the proposed CAC models based on transformer-based trackers achieve the state-of-the-arr performance comparing to BMNet, BMNet+ and CounTRR. In comparison to BMNet+, the official state-of-the-art, our models yield an improvement of 29.03% on VAL MAE, 24.98% VAL RMSE, 15% TEST MAE, and 9. 9.74% TEST RMSE. Even without multi-scale setting, our models are able to achieve the best performance, proving the validity of exploiting transformer-bis bassed trackers baselines of CAC task. Without adopting the same test- time normalization strategy as CounTR+, the proposed approach models still outperform CounTR and CounTR+.
Qualitative Visualizations As shown in Figure our models are able to locate objects more accu- rately with higher confidence, especially in dense scenes. While BMNet+ is able identify position of target objects, the confidence regions are spread more widely. On the other hand, our models capable of placing point annotations right at the center of the target objects with smaller radius, modelling with less ambiguity. When inspecting the first row in Figure 3 the proposed models are better at locating birds in dense scenes and predict a density map that aligns closer to the ground truth and wouldn't It create incorrectly distributed patches.
Pross-Dataset Generalization A strong CAC model should excel in counting objects on scenarios that differ from the training dataset in terms of perspectives, scales, illumination, etc. It should be able to perform well on unseen scenarios without additional exposure on similar scenarios. To eval- uate the ability generalize on other datasets, and then evaluate on CARPK testing set without the use fine-tuning on the CARPK training set. Specifically. since the CARPK dataset mainly consists of images, remove the category from the FSC147 dataset during training stage. As shown Table 4.3 both our models outper- form the non-fine-tuned baseline. Moreover, MixFormerCAC is able to win over other fine-tuned
train the proposed method on the FSC147 dataset

---

Under review as a conference paper at ICLR 2023
GroundTruth
BMNet+
MixFormerCAC
TransTCAC
1092
569.7843017576125
571.1722412109375
637. 637.56396484375
105.43711853027344
119.03854370117189
126.39823913574219
211
230.99246215820312
217:31951904296875
725.76397705078125
33
24.659990042114258 33.5667724609375
37.83891296386719
Figure 3: Qualitative Visualization. We visualize the predicted density map results of the proposed approaches with BMNet+ the FSC147 testing dataset. The leftmost column shows the ground- truth density maps overlaid onto the original images.
models and achieve the state-of-the-art performance. This further demonstrate the proposed models better excel in various counting scenarios than other methods.
Model
Fine-tuned
MAE
RMSE
BMNet Shi et al. 2022
8.05
9.7
BMNet+ Shi et (2022)
5.76
7.83
CounTR Liu et al.
2022)
/ 5.75
7.45
BMNet Shi et al. (2022)
14.61
24.60
BMNet+ Shi et al. (2022)
X
10.44
13.77
CCanSTCAC(Ours
X
9.61
12.65
MixFFormerCAC(Ours)
X
3.6
4.67
Table 2: Cross-datased evaluation results compared with other state-of-the-ee models where 'fine- tuned' means the model is further fine-tuned using the CARPK training data.
4.4 IMPACT OF NUMBER OF REFERENCES IMAGES
We investigate our model performance in relation the number of reference images, where S 3. As shown in Table 3 the results indicate that one reference image is adequate for the proposed models to reach comparable performance with current state-of-the-ar performance, given ability of the attention-based modules within transformer-based tracker to extract representative reference features through global interactions among query and reference features. Moreover, the multi-scale fusion also allows the model to effectively capture objects of different sizes. Given more reference images, the proposed models are able to yield better results and excel in each of scenarios of n reference images.
8

---

Under review as a conference paper at ICLR 2023
n
Trans CensTCACE CAC(O)us))
BMNet+ Shi et (2022
Val MAE
Val RMSE
Val MAE
Val RMSE
Val MAE
Val RMSE
14.77
62.45
12.82
45.14
17.89
61.12
2 12.65
52.68
12.17
45.39
16.03
58.65
3
11.17
46.69
11.66
43.91
15.74
58.53
n
MixFormerCAC(OUue
TransTCAC CACC(Ours))
BMNet+ Shi et al. (2022)
Test MAE
Test RMSE
Test MAE
Test RMSE
Test MAE
Test RMSE
T
12.63
106.41
13.62
82.44
16.89
96.65
2
11.90
97.21
12.75
85.37
16.16
97.18
3 11.82
92.78
12.46
82.89
14.62
91.83
Table 3: Evaluation results of the proposed methods using different numbers of reference images.
Bold font indicates the best model performance of MixFormerCAC and TransTCAC.
No.
Feature
Val MAE
Val RMSE
Test MAE
Test RMSE
S1
B
18.79
66.9
15.82
104.6
S2
B+GL
12.29
56.88
13.09
99.58
S3
B+GL+MS
11.66
43.91
12.46
82.89
Table 4: Ablation studies It shows the ablation results of the proposed TransTCAC where 'B' denotes using backbone and matcher of TransT followed by simple density head using the features from the last layer, 'GL' denotes replacing the MSE loss with the generalized loss, and 'MS' means to use multi-scale features instead with the U-Net-like fusion.
4.5 ABLATION STUDIES
As the architecture of TransTCAC is similar to BMNet, we perform ablation studies on TransTCAC
to examine the functionality of our proposed framework in relation to the baseline model B.Net. (1) Transformer-basee backbone: We integrate transformer-basee backbone (ResNet--0) and matcher with the CAC density head. The essential difference between $1 and BMNet is the substitution of the BMNet matcher with the Trans'T matcher. The results show that the performance of CCCC in SI surpasses BMNet by 0.89 in test MAE and 0.29 in val MAE, suggesting the strength of transformer-trakker matching. (2) Generalized Loss: The introduction of generalized loss yields relative improvement of 6.5 in val MAE and 2.73 in test MAE. The loss function provides stronger supervision, therefore allowing TransTCAC to effectively exploit attention modules and identify regions of interests. (3) Multi-scale Feature with U-Net-like Fusion: The addition of multi-scale feature extraction with U-Net-like fusion allow to capture the objects across scales. It thus enables TransT to further improve the counting performance by large margin and to achieve 12.91 in val RMSE and 16.69 in test RMSE.
5 CONCLUSION
In this work, we show the connection of the popular and effective bilinear similarity matching for the CAC task with the attention modules which are widely used in the transformer-based methods. Furthermore, inspired by this finding, since the visual object tracking shares similar components with CAC, we demonstrate the advanced self- and cross-attention modules used in transformer- based trackers are powerful matching modules for the CAC tasks and can greatly help the model learn more distinct features to capture the shared pattern among the query and reference images. We demonstrate the effectiveness of this idea by adapting two state-of-the-art transformer- trackers and achieve new state-of-the-art performances for the CAC task. Moreover, we also find that the widely used zero-padding strategy for the query images during training as used in previous CAC models hurts the inference performance when we adapt the transformer based trackers for CAC. We instead propose random cropping strategy for the query images to avoid performing any zero-padder pero-padding for improved performance. With extensive experiments and ablation studies, we show that the CAC models based on transformer-hasee tracker are strong baselines for the CAC task.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer track- ing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 8126- 8135, 2021.
Zhi-Qi Cheng, Jun-Xiu Li, Qi Dai, Xiao Wu, and Alexander Hauptmann. Learning spatial aware- ness improve crowd counting. In IEEE/CVF International Conference Computer Viston (ICCV), pp. 6152-6100 6121, 2019.
Zhi-Qi Qi Cheng. Qi Dai, Hong Li, Jingkuan Song, Xiao Wu, and Alexander Hauptmann. Rethinking spatial invariance of convolutional networks for object counting. In IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 19638-19648, 2022.
Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 13608 13618, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An im- age is worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations (ICLR), 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In IEEE/CVE International Conference on Computer Vision and Patlern Recognition (CVPR), pp. 770-778, 2016.
Meng-Ru Ru Hsich, Yen-Liang Lin, and Winston Hsu. Drone-based object counting by spatially regularized regional proposal network. In IEEE International Conference Computer Vision (ICCV), pp. 1145-4153,, 2017.
Bastian Leibe, Edgar Seemann, and Bernt Schiele. Pedestrian detection in crowded scenes. In IEEE International Conference Computer Vision and Pattern Recognition (CVPR), volume 1, pp. 878-885, 2005.
Yuhong Li, Xiaofan Zhang, and Deming Chen. Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In IEEF/CVF International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1091-1100, 2018.
Chang Liu, Yujie Zhong, Andrew Zisserman, and Weidi Xie. Countr: Transformer-based gener- alised visual counting, arXiv preprint arXiv:2208. 13721, 2022.
Erika Lu, Weidi Xie, and Andrew Zisserman. Class-agnostic counting. In Asian Conference on Computer Vision (ACCV), pp. 669-684,30 2018.
Zhiheng Ma, Xiaopeng Hong, Xing Wei, Yunfeng Qiu, and Yihong Gong. Towards a universal model for cross-dataset crowd counting. In IEEF/CVE International Conference on Computer Vision (ICCV), pp. 3205-3214, 2021.
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3394 -4033, 2021.
Min Shi, Hao Lu, Chen Feng, Chengxin Liu, and Zhiguo Cao. Represent, compare, and learn: A similarity- -waare framework for class-agnostic counting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9529-9538, 2022.
Qingyu Song, Changan Wang, Zhengkai Jiang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Feiyue Huang, and Yang Wu. Rethinking counting and localization in crowds: A purely point- based framework. In IEEE/CVE International Conference on Computer Vision (ICCV), pp. 3365- 3374, 2021.
Pongpisit Thanasutives. Ken-ichi Fukui, Masayuki Numao, and Boonserm Kijsirikul. Encoder- decoder based convolutional neural networks with multi-scale-aware modules for crowd counting. In IEEE International Conference Pattern Recognition (ICPR), pp. 2889-2389, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems (NeurIPS), 30, 2017.
Shuo-Diao Yang, Hung-Ting Su, Winston H Hsu, and Wen-Chin Chen. Class-agnostic few-shot object counting. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 870-878, 2021.


Reviews:
[{"review_id": "VlE5d3U6E2", "reviewer": "Reviewer_eXTk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to apply two existing Transformer-based trackers to class agnostic counting and evaluated their performance on public datasets.", "summary_of_the_review": "This paper's novelty by applying siamese trackers is incremental. The used networks are directly borrowed from existing tracking methods without adaptively redesign for counting task. Besides, the performance improvement largely benefit from Transformer-based attention block which brings more computational cost. Based on these concerns, I tend to reject this paper.", "strength_and_weaknesses": "\nWeaknesses\n\n-The largest concern is the limited contribution. The authors directly apply two existing trackers to fix the counting task with minor changes by extending them to multi-scale architecture with an off-the-shelf generalized loss.  Besides, the core idea of transferring the CAC problem to a matching task between reference and query is the same with pervious arts. This paper only replaces the simple inner-production with more complex Transformer modules. I can't see any new insights for solving counting tasks in this paper. \n\n-Based on the comparison between Table 1 and Table 4, the improvement largely benefits from the used GL loss. The baseline model by using only TransTCAC is even not comparable with 'BMNet+* Shi et al. (2022)'. The current results can't support the advantage by using Transformer Trackers. \n\n-The experiments are not fair according to the current statement since the input resolution, model size, computational cost and the usage of multi-scale structure are missing in Table 1.\n\n-The statement 'Even without multi-scale setting, our models are able to achieve the best performance, proving the validity of exploiting transformer-based trackers as baselines of CAC task' in Quantitative Results of Sec. 4.3  is confusing. It seems that the listed TransTCAC(Ours)* already used the multi-scale setting according to Table 4.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I-ePDKyS6ho", "reviewer": "Reviewer_8qRK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper tackles the task of Class Agnostic Counting (CAC) in a few-shot setting, which involves predicting the overall count for the object of interest in an image, given few exemplars of object of interest from the same image. Previous works on CAC such as Ranjan et al, FamNet, rely on correlation between the exemplar features and input image features to predict the density map (heatmap), and the sum of the density map serves as the overall count. Similar to the work of Shi el al, CountTR, authors replace the Resnet-50 backbone from earlier works with Transformer based backbone. Unlike CountTR, which uses Imagenet pretrained ViT backbone, authors use Transformer backbones designed for Visual Object tracking task. Drawing inspiration from BMNet, which uses bilinear matching between the exemplar features and image features, authors first draw similarities between bilinear matching and attention, and propose a attention based approach for matching the two sets of features.", "summary_of_the_review": "My main concern with the paper is the limited novelty of the proposed approach, as stated in the Strengths and Weaknesses section.", "strength_and_weaknesses": "Strengths:\nThe main claim of the paper, that Transformer based approaches designed for object tracking are suitable for exemplar based CAC is reasonable. And the significant boost in performance achieved by the proposed approach provides some weight to the claim.\n\nWeaknesses:\n1. Original technical contribution of the paper is somewhat limited, since previous work like CounTR  has already advocated the use of Transformer for Class Agnostic Counting task.\n2. Most of the previous approaches on FSC-147 use Imagenet pretrained backbone such as Resnet-50 and ViT. Proposed approach uses backbones trained on Visual Object tracking datasets. This extra training data being used could be one of the reasons behind the superior performance of the proposed approach.\n3. Ablation study in table 4 would have been more complete had there been a row corresponding to ViT backbone (similar to the first row S1, but using ViT backbone instead of TransTCAC backbone).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "96GPyMdl4K", "reviewer": "Reviewer_zJ5d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper present a novel perspective for the task of Class Agnostic Counting (CAC), drawing similarities with the task of Visual Object Tracking (VOT), where a pattern (VOT: patch / CAC: reference) is localized in an image (VOT: search space / CAC: query image). \n\nIn particular, the authors leverage:\n - previous finding in VOT models including self- and cross- attention mechanisms (from TransT, Chen et al. (2021) and MixFormer, Cui et al. (2022)), \n - previous findings in CAC models, including the general structure of the models and a Bilinear Matching (from BMNet, Shi et al. (2022)), \n - previous finding in crowd counting models, including the generalized loss based on optimal transport (from SDNet, Ma et al. (2021)), and\n - novel modules such as the Multi-Scale enhancement for matching and density map estimation\n - novel preprocessing of the dataset (inspired by FamNet, Ranjan et al. (2021))\n\nto build a CAC model showing state-of-the-art performances on the datasets/benchmarks FSC-147 and CARPK.\n\nThe authors provide a complete analysis of their method, with an ablation study for all the modules they have integrated.\n", "summary_of_the_review": "The analysis of the similarities between tracking architectures and CAC architectures is interesting to be noted and analyzed. It appears that the methodology leads to SOTA results on the latest benchmarks, which is remarkable.\n\nYet, the authors claim that transformer-based trackers are strong baselines for CAC, but it appears that most of the improvement in the experiments originate from the Generalized Loss based on Optimal Transport, introduced by Ma et al. (2021) in a different task. The ablation study confirms such a statement. It is misleading to claim that the results originate from self-/cross- attention mechanisms and the transferability of trackers’ knowledge to the task of class-agnostic counting.\n\nThe paper is very borderline, as I believe the analysis with trackers is a fresh perspective, yet it needs further investigations as it is misleading in the current form. As a result, I would recommend a score of 5.\n", "strength_and_weaknesses": "Strengths:\n\nThe method presented by the authors reached impressive state-of-the-art performances on the FCS-147 dataset for CAC.\nThe method is able to transfer to CARPK (with state-of-the-art performances) without the need for any extra fine-tuning.\nThe similarity of the architectures for CAC with the architectures for VOT is interesting and worth noting.\n\nWeaknesses:\n\nThe similarities drawn between the CAC and VOT tasks omitted an important analysis. In particular, VOT methods are trained to discriminate similar objects (same class) from the same image, while CAC actually wants to look at those exact objects. In VOT, those extra samples in a frame are referred to background clutter, and previous works focus on learning to discard those distractors (e.g. “Context-aware correlation filter tracking”, Mueller et al., CVPR’17 / “Distractor-aware siamese networks for visual object tracking”, Zhu et al., CVPR’18 / “Exploring Motion Information for Distractor Suppression in Visual Tracking”, Liu et al., CVPR’22). Because of the nature of the tasks, I would respectfully disagree with the authors who state that “the formulation of visual object tracking task is similar to CAC” (abstract), “visual object tracking formulation share similar formulation [...] as the CAC task” (introduction), “object tracking excel at the task of localizing the target objects from the query image which is similar to CAC” (related work), and “the setting and architecture of transformer-based trackers are similar with existing CAC models” (methodology) among other places in the remainder of the paper.\n\n\nWhile the CAC and VOT architectures show significant similarities, the knowledge a tracker is expected to learn is different from the knowledge needed for a CAC task. The learning process aims at different goals, and as a result, one would not expect the knowledge pre-trained in a VOT task to transfer to a CAC task. TransT and MixFormer were both pre-trained on a tracking task, and the paper misses an analysis comparing the performance with and without transferring the negative knowledge learned from the tracking task.\n\n\nFrom the ablation study, it appears that most improvement originates from the optimal transport technique borrowed from Ma et al., and very little from the attention mechanisms, and tracker architecture. As a result, I would recommend putting the focus on that generalized loss, rather than biasing the reader into believing that the improvement originates from expensive transformer-based mechanisms or a tracking formulation. Moreover, the paper only mentions that loss without properly defining it. It is a pity and frustrating for the reader to not fully grasp and understand that loss, that contributes to most of the improvement in the ablation study.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject