Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Paper tackles the task of Class Agnostic Counting (CAC) in a few-shot setting, which involves predicting the overall count for the object of interest in an image, given few exemplars of object of interest from the same image. Previous works on CAC such as Ranjan et al, FamNet, rely on correlation between the exemplar features and input image features to predict the density map (heatmap), and the sum of the density map serves as the overall count. Similar to the work of Shi el al, CountTR, authors replace the Resnet-50 backbone from earlier works with Transformer based backbone. Unlike CountTR, which uses Imagenet pretrained ViT backbone, authors use Transformer backbones designed for Visual Object tracking task. Drawing inspiration from BMNet, which uses bilinear matching between the exemplar features and image features, authors first draw similarities between bilinear matching and attention, and propose a attention based approach for matching the two sets of features.
strength_and_weaknesses: Strengths:
The main claim of the paper, that Transformer based approaches designed for object tracking are suitable for exemplar based CAC is reasonable. And the significant boost in performance achieved by the proposed approach provides some weight to the claim.

Weaknesses:
1. Original technical contribution of the paper is somewhat limited, since previous work like CounTR  has already advocated the use of Transformer for Class Agnostic Counting task.
2. Most of the previous approaches on FSC-147 use Imagenet pretrained backbone such as Resnet-50 and ViT. Proposed approach uses backbones trained on Visual Object tracking datasets. This extra training data being used could be one of the reasons behind the superior performance of the proposed approach.
3. Ablation study in table 4 would have been more complete had there been a row corresponding to ViT backbone (similar to the first row S1, but using ViT backbone instead of TransTCAC backbone).

clarity,_quality,_novelty_and_reproducibility: Paper is clearly written, and the proposed approach is well explained. Paper has somewhat limited novelty, given paper follows previous works like CounTR and BMNet.
summary_of_the_review: My main concern with the paper is the limited novelty of the proposed approach, as stated in the Strengths and Weaknesses section.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed to apply two existing Transformer-based trackers to class agnostic counting and evaluated their performance on public datasets.
strength_and_weaknesses: 
Weaknesses

-The largest concern is the limited contribution. The authors directly apply two existing trackers to fix the counting task with minor changes by extending them to multi-scale architecture with an off-the-shelf generalized loss.  Besides, the core idea of transferring the CAC problem to a matching task between reference and query is the same with pervious arts. This paper only replaces the simple inner-production with more complex Transformer modules. I can't see any new insights for solving counting tasks in this paper. 

-Based on the comparison between Table 1 and Table 4, the improvement largely benefits from the used GL loss. The baseline model by using only TransTCAC is even not comparable with 'BMNet+* Shi et al. (2022)'. The current results can't support the advantage by using Transformer Trackers. 

-The experiments are not fair according to the current statement since the input resolution, model size, computational cost and the usage of multi-scale structure are missing in Table 1.

-The statement 'Even without multi-scale setting, our models are able to achieve the best performance, proving the validity of exploiting transformer-based trackers as baselines of CAC task' in Quantitative Results of Sec. 4.3  is confusing. It seems that the listed TransTCAC(Ours)* already used the multi-scale setting according to Table 4.


clarity,_quality,_novelty_and_reproducibility: The authors promise to release the code depending on the acceptance.  
summary_of_the_review: This paper's novelty by applying siamese trackers is incremental. The used networks are directly borrowed from existing tracking methods without adaptively redesign for counting task. Besides, the performance improvement largely benefit from Transformer-based attention block which brings more computational cost. Based on these concerns, I tend to reject this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper present a novel perspective for the task of Class Agnostic Counting (CAC), drawing similarities with the task of Visual Object Tracking (VOT), where a pattern (VOT: patch / CAC: reference) is localized in an image (VOT: search space / CAC: query image). 

In particular, the authors leverage:
 - previous finding in VOT models including self- and cross- attention mechanisms (from TransT, Chen et al. (2021) and MixFormer, Cui et al. (2022)), 
 - previous findings in CAC models, including the general structure of the models and a Bilinear Matching (from BMNet, Shi et al. (2022)), 
 - previous finding in crowd counting models, including the generalized loss based on optimal transport (from SDNet, Ma et al. (2021)), and
 - novel modules such as the Multi-Scale enhancement for matching and density map estimation
 - novel preprocessing of the dataset (inspired by FamNet, Ranjan et al. (2021))

to build a CAC model showing state-of-the-art performances on the datasets/benchmarks FSC-147 and CARPK.

The authors provide a complete analysis of their method, with an ablation study for all the modules they have integrated.

strength_and_weaknesses: Strengths:

The method presented by the authors reached impressive state-of-the-art performances on the FCS-147 dataset for CAC.
The method is able to transfer to CARPK (with state-of-the-art performances) without the need for any extra fine-tuning.
The similarity of the architectures for CAC with the architectures for VOT is interesting and worth noting.

Weaknesses:

The similarities drawn between the CAC and VOT tasks omitted an important analysis. In particular, VOT methods are trained to discriminate similar objects (same class) from the same image, while CAC actually wants to look at those exact objects. In VOT, those extra samples in a frame are referred to background clutter, and previous works focus on learning to discard those distractors (e.g. “Context-aware correlation filter tracking”, Mueller et al., CVPR’17 / “Distractor-aware siamese networks for visual object tracking”, Zhu et al., CVPR’18 / “Exploring Motion Information for Distractor Suppression in Visual Tracking”, Liu et al., CVPR’22). Because of the nature of the tasks, I would respectfully disagree with the authors who state that “the formulation of visual object tracking task is similar to CAC” (abstract), “visual object tracking formulation share similar formulation [...] as the CAC task” (introduction), “object tracking excel at the task of localizing the target objects from the query image which is similar to CAC” (related work), and “the setting and architecture of transformer-based trackers are similar with existing CAC models” (methodology) among other places in the remainder of the paper.


While the CAC and VOT architectures show significant similarities, the knowledge a tracker is expected to learn is different from the knowledge needed for a CAC task. The learning process aims at different goals, and as a result, one would not expect the knowledge pre-trained in a VOT task to transfer to a CAC task. TransT and MixFormer were both pre-trained on a tracking task, and the paper misses an analysis comparing the performance with and without transferring the negative knowledge learned from the tracking task.


From the ablation study, it appears that most improvement originates from the optimal transport technique borrowed from Ma et al., and very little from the attention mechanisms, and tracker architecture. As a result, I would recommend putting the focus on that generalized loss, rather than biasing the reader into believing that the improvement originates from expensive transformer-based mechanisms or a tracking formulation. Moreover, the paper only mentions that loss without properly defining it. It is a pity and frustrating for the reader to not fully grasp and understand that loss, that contributes to most of the improvement in the ablation study.

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear and provide quality content. The methodology appear relatively novel, but the contributions are not clearly stated. The authors promised reproducible code upon publication.

The paper missed a proof-reading. Some sentences are incomplete or missing “...we adapt the transformer-based trackers for CAC since In addition, we select TransT …” (related work), as well as some prepositions all across the paper.

The exact contributions of this paper are not clearly stated, and most contributions appear to be borrowed from previous works in different fields. The paper would benefit from a clear statement of the contribution.

The hyperparameters scale, reach, blur and \tau are not defined in the paper.

The dataset FSC147 is named differently in the paper (FSC-147 vs FSC147).

The aggregation of the results obtained from multiple references (e.g. for the 3-shot counting task) are not explained in the paper, hence is it unclear how the results from the multiple references are merged together.

Figure 3: Would it be possible to visualize the qualitative results for the ablation study? It would validate the sharpness of the density map introduced by the GL based on OT.

Figure 3: The significant figure should be truncated or averaged.

Table 2: What are the performances on CARPK once fine-tuned? Also, the table is missing the SOTA method “An Accurate Car Counting in Aerial Images Based on Convolutional Neural Networks”, Kilic et al., (2021) that appears SOTA on CARPK.


summary_of_the_review: The analysis of the similarities between tracking architectures and CAC architectures is interesting to be noted and analyzed. It appears that the methodology leads to SOTA results on the latest benchmarks, which is remarkable.

Yet, the authors claim that transformer-based trackers are strong baselines for CAC, but it appears that most of the improvement in the experiments originate from the Generalized Loss based on Optimal Transport, introduced by Ma et al. (2021) in a different task. The ablation study confirms such a statement. It is misleading to claim that the results originate from self-/cross- attention mechanisms and the transferability of trackers’ knowledge to the task of class-agnostic counting.

The paper is very borderline, as I believe the analysis with trackers is a fresh perspective, yet it needs further investigations as it is misleading in the current form. As a result, I would recommend a score of 5.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The article proposes a counting method agnostic of the class. It is based on transformers and achieves state-of-the-art in several datasets. The method also shows transfer capabilities to other datasets. 

The main problem with the method is that the contribution is not clear. Other articles, like CounTR, have already proposed using transformers for this problem. It seems that the improvements come from pretraining on more datasets. Moreover, there are other problems, such as more ablation results needed, difficulties in comparison with baseline models, not fair experiments, confusing statements, and missing analysis.

The authors have addressed some of the reviewer's concerns. However, the contribution of the article is limited, and accordingly, I suggest rejecting this article.

justification_for_why_not_higher_score: The contribution of the article is very limited.
Decision: Reject