A Novel Unified Architecture for Low-Shot Counting by Detection and Segmentation
Authors: Jer Pelhan
Alan Lukezic
Vitjan Zavrtanik
Matej Kristan
Venue: NeurIPS 2024 poster (2024)
Keywords: low-shot counting
few-shot counting
object counting;small object detection; small object coutning;small objects;detection;counting;loss function
Primary area: machine_vision

Abstract: Low-shot object counters estimate the number of objects in an image using few or no annotated exemplars. Objects are localized by matching them to prototypes, which are constructed by unsupervised image-wide object appearance aggregation.
Due to potentially diverse object appearances, the existing approaches often lead to overgeneralization and false positive detections.
Furthermore, the best-performing methods train object localization by a surrogate loss, that predicts a unit Gaussian at each object center. This loss is sensitive to annotation error, hyperparameters and does not directly optimize the detection task, leading to suboptimal counts.
We introduce GeCo, a novel low-shot counter that achieves accurate object detection, segmentation, and count estimation in a unified architecture.
GeCo robustly generalizes the prototypes across objects appearances through a novel dense object query formulation. 
In addition, a novel counting loss is proposed, that directly optimizes the detection task and avoids the issues of the standard surrogate loss. 
GeCo surpasses the leading few-shot detection-based counters by $\sim$25\% in the total count MAE, achieves superior detection accuracy and sets a new solid state-of-the-art result across all low-shot counting setups. 
The code will be available on GitHub.

A Novel Unified Architecture for Low-Shot Counting by Detection and Segmentation
Jer Pelhan, Alan Lukežić, Vitjan Zavrtanik, Matej Kristan Faculty of Computer and Information Science, University of Ljubljana jer.pelhan @ ffri.umi-lj.si
Abstract
Low-shot object counters estimate the number of objects in an image using few no annotated exemplars. Objects are localized by matching them to prototypes, which are constructed by unsupervised image- wide object appearance aggregation. Due to potentially diverse object appearances, the existing approaches often lead to overgeneralization and false positive detections. Furthermore, the best- best-performing methods train object localization by surrogate loss, that predicts unit Gaussian at each object center. This loss is sensitive to annotation error, hyperparameters and does not directly optimize the detection task, leading to suboptimal counts. We introduce GeCo, a novel low-shot counter that achieves accurate object detection, segmentation, and count estimation in a unified architecture. GeCo robustly gener- alizes the prototypes across objects appearances through a novel dense object query formulation. In addition, a novel counting loss is proposed, that directly optimizes the detection task and avoids the issues of the standard surrogate loss. GeCo surpasses the leading few-shot detection- based counters by ~25% in the total count MAE, achieves superior detection accuracy and sets a new solid state-of-the-an result across low-shot counting setups. The code available on GitHub
Introduction
Low-shot object counting considers estimating the number of objects of previously unobserved category in the image, given only a few annotated exemplars (few-shot) or without any supervision (zero-shot) 22 The current state-of-the-an methods are predominantly based on density estima- tion [4] [14 32
26 22 31 31]. These methods predict a density map over the image and estimate the total count by summing the density.
While being remarkably robust for global count estimation, density outputs lack explainability such object location and size, which is crucial for many practical applications [3] 30). This recently
gave rise to detection-based low-shot counters (20) 19 35). which predict the object bounding boxes
and estimate the total count the number of detections. Nevertheless, detection-based counting falls
behind the density- -base methods in total count estimation, leaving a performance gap.
In detection- based counters, dominant approach to identify locations of the objects in the image involves construction of object prototypes from few (e.g., three) annotated exemplar bounding boxes and correlating them with image features (20 35 19]. The exemplar construction process is trained to account for potentially large diversity of object appearances in the image, often leading overgeneralization, which achieves high recall, but is also prone to false positive detection. Post-hoc detection verification methods have been considered (2) 35 to address the issue, but their multi-stage formulation prevents exploiting the benefits of end-to-end training.
Currently, the best detection counters [20] 35 the correlation map. During training, the map prediction is supervised by a unit Gaussian placed on
predict object locations based on the local maxima in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

GeCo, ERR
GeCo, ERR
2030H ERR
predicts object centers (red dots) biased towards blob-like structures, leading to incorrect partial detections of ants (bottom left), while GeCo(ours) addresses this with the new loss (top left). CDETR [99 fails in densely populated regions (bottom right), while GeCo addresses this with the new dense query formulation by prototype generalization (top right). Exploiting the SAM backbone, GeCo delivers segmentations as well. Exemplars are denoted in blue.
Figure 1: DAVE 20
each object center. However, the resulting surrogate loss is susceptible to the center annotation noise, requires nontrivial heuristic choice of the Gaussian kernel size and in practice leads to detection preference of compact blob-like structures (see Figure column 1&2). Recently, DETR [ inspired counter was proposed to avoid this issue [19], however, it fails in densely populated regions even though applics a very large number of detection queries in a regular grid (see Figure I column 3&4).
We address the aforementioned challenges by proposing new single-stage low-shot counter GeCo, which is implemented as an add-on network for SAM 12] backbone. A single architecture is thus trained for both few-shot and zero-shot setup. it enables counting by detection and provides segmentation masks for each of the detected objects. Our first contribution is dense object query formulation, which applies non-parametric model for image-wide prototype generalization (hence GeCo) in the encoder, and decodes the queries into highly dense predictions. The formulation simultaneously enables reliable detection densely- poopulllpp regions (Figure [ column 3&4) and prevents prototype over- generalizatinnn leading to improved detection precision at high recall. Our second contribution is a new loss function for dense detection training that avoids the ad ad-hoc surrogate loss with unit Gaussians. it directly optimizes the detection task, and leads to improved detection not biased towards blob-like regions (Figure) T column 1&2).
GeCo outperforms all detection-basee counters on challenging benchmarks by 24% MAE and the density-bane based long-stani standing winner 4 by 27% MAE, while delivering superior detection accuracy. The method shows substantial robustness to the number of exemplars. In one-shot scenario, GeCo outperforms the best detection method in 5% AP50, 45% MAE and by 14% a zero-shot scenario. GeCo is the first detection- counter that outperforms density based counters in all measures by using the number detections as the estimator, and thus sets a milestone in low-shot detection- based counting.
2 Related works
Traditional counting methods focus on predefined categories like vehicles cells [5), people [5], and polyps, (33 requiring extensive annotated training data and lacking generalization to other categories. necessitating retraining conceptual changes. Low-shot counting methods address this limitation by estimating counts for arbitrary categories with minimal or no annotations, enabling test-time adaptation.
With the proposal of the FSC147 dataset 24 low-shot counting methods emerged, which predict global counts by summing predicted density maps. The first method [24] proposed an

---

adaptation of a tracking backbone for density map regression. BMNet+ [26] tackled learning representation and similarity metric, while SAFECount [32 introduced a new feature enhancement module, improving appearance generalization. CounTR [1] utilized a vision transformer for image feature extraction and convolutional network for encoding the exemplar features. I.OCA 4 argued that exemplar shape information should be considered along with the appearance, and proposed an iterative object prototype extraction module. This led to a simplified counter architecture that remains top-performer among density-basee based counters.
To improve explainability of the estimated counts and estimate object locations as well, detection- based methods emerged. The first sho- detection based counter [99] was an extended transformer- based object detector 2 with the ability to detect objects specified by the exemplars. Current state-of-the-an DAVE 20 proposed two-stage detect-and-verit paradigm for low-shot counting and detection, wherein the first stage it generates object proposals with high recall, but low precision, which is improved by a subsequent verification step. PSECO 35 proposed a three-stage approach called www-----emeeeeddcneeee
which employs more involved proposal generation with better detection accuracy and also applies verification step improve precision. Both DAVE and PSECO are multi-stage methods that train network for the surrogate task of predicting density maps for object centers, from which the bounding boxes are predicted. Although irrection-based counters offer additional applicability, they fall behind the best density- based counters in global count estimation.
3 Single-stage low-shot object counting by detection and segmentation
Given an input image I E Ha Wax3 and a set of k exemplar bounding boxes B (br specifying the target category, the task is to predict bounding boxes B {by}j=ı:N for all target category objects in L, with the object count estimated as N B
Detection extraction
SAM
DQE
DQ,
Backbone
4h
4W
Count:
Prototype extraction
Figure 2: The architecture of the proposed single-stage low-shot counter GeCo.
The proposed detection- based counter GeCo pipeline proceeds follows (see Figure 2 The image is encoded by a SAM [12) backbone into f E Rhxwxd, where h = Ho/r, to Wo/r and is number of feature channels. In the few-shot setup, two kinds of prototypes (appearance and shape) are extracted from each annotated object exemplar. The appearance prototypes pd E Re are extracted by Rol-pooling
[ features f from the exemplar bounding boxes. Following 4. shape prototypes p Rxd are extracted as well, by p = ((W,, Hb,l), where Wb, and Hb, are the width and height of the i-th exemplar bounding box, and (--) is small MLP network. The concatenation of A and pa yields p E 22dx prototypes.
Note, however, that in a zero-shot setup, exemplars are not provided and the task is to count the majority-class objects in the image. In this setup, single zero-shot prototype is constructed by attending a pretrained objectness prototype p to the image features, i.c., p = CA(p² , f', f'), where CA(a, b, c) is cross-attention (28) followed by skip connection, with a, b and attention query, key and value, respectively.
The prototypes p (either from few-shot or zero-shot setup) are then generalized across the image, and dense object detection queries are constructed by the Dense query encoder (DQE, Section 3.1). These are decoded into dense detections by the Dense query decoder (DQD, Section 3.2). The final detections are extracted and refined by a post- post-processing step (Section 3.3). The aforementioned modules are detailed in the following sections.
3

---

3.1 Dense object query encoder (DQE)
To account for the variation of the object appearances the image, the current state-of-the-art 20 aims at constructing a small number of prototypes (e.g., three) that compactly encode the object appearance variation in the image, often leading to overgeneralization and false detections. We deviate from this paradigm by considering image- wide prototype generalization with non con--aamettic model that constructs w location-specific prototypes PN E RR=h Let Po = f' be the initial dense generalized prototypes (i.e., one for each location). The final dense generalized prototypes P PN, are calculated by the following iterative adaptation via cross-attention
P; CA(P.-1,P.))
where i (1...,,Pp) Np) Note that spatial encoding is not applied, to enable spatially- unniaael information flow from the prototypes p to all locations.
Next, dense object queries are constructed from the generalized prototypes by the following iterations
Q = CCASS((F), Q1-1),
where j E adapt the imput features the current queries. In both cross- and self-attentions, positional encoding query construction. In the remainder of the paper, the dense
Qo PAP Ny. and SA(.) is a self-attention followed by a skip connection to
is applied to enable vocaionnddppenee object queries QNo are denoted as Q for clarity
3.2 Dense object query decoder (DQD)
The dense queries Q from Section 3.1 are decoded into object detections by a dense object query decoder (DQD). Note that the spatial reduction of image by the SAM backhone may lead to encoding several small objects into the same query in Q. To address this, the object queries are first spatially unpacked into high-resolution dense object queries i.e., '#ftt E = xd, where I/ Ho/2. 2, W Wo/2 and d is the number of feature channels. The unpacking process consists of three convolutional upsampling stages, with each stage composed of 3 x 3 convolution, a Leaky ReLU and 2x bilinear upsampling. To facilitate unpacking of small objects, the features after the second stage are concatenated by the SAM-HQ features [1] fTQ before feeding into the final stage.
Finally, the objectness score yO E 1xxxxx1 is calculated by a simple transform, i.c., yo LRelu(Wo R) where Wo is a learned projection matrix and LReI.U(-) is Leaky ReLU. Each query also decoded into the object pose by three-layer MLP, i.e., BB (MMIP(Q#R)) where (.)) is sigmoid function and BBB E RR HR#xWxx are bounding box parameters in the tirb format 27
3.3 Detections extraction and refinement
The final detections are extracted from y and BB follows. Bounding box parameters read out from BB at locations of local maxima on a thresholded y (using a x nonmaxima suppression, NMS). The bounding boxes are refined by feeding them as prompts into SAM decoder [12] on the already computed backbone features ff. The boxes are refitted to the masks by min-max operation and finally mon-maxima suppression with IoU 0.5 is applied to remove duplicate detections. This process thus yields the predicted bounding boxes B P and their corresponding masks M".
3.4 A novel loss for dense detection training
GeCotraining requires supervision on the dense objectness scores y and the bounding box parame- ters y BB Ideally, network should learn predict points on objects that be reliably detected by NMS, and also from which the bounding box parameters can be reliably predicted. We thus propose a new dense object detection loss that pursues this property.
Following the detection step (Section 3.2) in the forward pass, a set of local maxima (i) is identified by applying a NMS on y and keeping all maxima higher than the median response, to ensure detection redundancy. The maxima are then labelled as true positives (TP) and false positives (FP) by applying Hungarian matching [13] between their bounding box parameters { (y"")>= 1: and the ground truth bounding boxes (B Jj=1:Nor. To account for missed detections, centers of

---

the non-minn non-matched ground truth bounding boxes are added to the list of local maxima and labeled as false negatives (FN). The new training loss thus defined as
=
Σ gloU(y!? RB BHUN(S) +
Σ (y: - 1)² + Ey. -0?
(3)
SEIP
SETPURN
EEFP
where g.oU(..-) the generalized IOU (25, and (UN()) is the ground truth index matched with the i-th predicted bounding box. Note that the new loss simultaneously optimizes the bounding box prediction quality, promotes locations with better box prediction capacity that can be easily detected
by a NMS, and enables automatic hard-negative mining in the objectness score via FP identification.
4 Experiments
Implementation details.
Using the SAM 12 backbone, GeCo reduces the input image by a factor r = 16, and projects the
features into d 256 channels (Section B). DQE (3.ction NN iterations are applied in
prototype generalization ( and No = 2 iterations in dense object query construction (2). Following
the established test-time practice 20: T9 26]. the input image is scaled to fit Wo Ho 1536
the average of the exemplars widths and heights is below 25 pixels, otherwise it is downscaled to
fit the average of the exemplar width and height to 80 pixels and zero pero-padded Wo Hg 1024.
As in (20), the zero-shot GeCo is run twice, first to estimate the objects size and then again on the resized image.
Training details. With the SAM backbone frozen, GeCo is pretrained with the classical loss 20 for initialization and is then trained for 200 epochs with the proposed dense detection loss (3) using a mini-batch size of 8, Adam W [16] optimizer, with initial learning rate set to 10-4, and weight decay of 10-4 The training is done 2 A A100s GPUs with standard scale augmentation 220 4 and zero-padding images to x024x 1024 resolution. For the zero-shot setup, the few-shot GeCo is frozen and only the zero-shot prototype extension is trained for 10 epochs. Thus the same trained network used in all low-shot setups.
Evaluation metrics and datasets. Standard datasets used. The FSCD147 [19 is detection- oriented extension of the FSC147 [24), which contains 6135 images of 147 object classes, split into 3659 training, 1286 validation, and 1 199 test images. The splits disjoint such that target object categories in test set are not observed in training. The objects are manually annotated by bounding boxes in the test [19], while in the train set, the bounding boxes are obtained from point estimates by SAM (35). For each image, three exemplars are provided. The second dataset is FSCD-LVIS [19] derived from LVIS and contains 377 categories. Specifically, the unseen-split used (3959 training and 2242 test images), which ensures that test-time object categories are not observed during training.
The standard evaluation protocol 24 26 52
with Mean Absolute Error (MAE) and Root Mean
Squared Error (RMSH) is followed to evaluate the counting accuracy. Following [19), Average Precision (AP) and Average Precision at loU=50 (AP50) is used on the same output to evaluate the detection accuracy.
4.1 Experimental Results
Few-shot counting and detection. GeCo compared with state-of- thee---ftthearttte density-based counters (which only estimate the total count) LOCA 4. CounTR [14), SAFECount (2), BMNet+ [26].
VCN (22) CFOCNet [3I] MAML (7), FamNet 24 and CFOCNet [31], and with detection- based
counters C-DETR 19,. SAM-C [18. PSECO
35 and DAVE (20). which also provide object
locations by bounding boxes. Results are summarized in Table
GeCo outperforms both recent state-of-the detection based counters DAVE 20 and PSECO 35 by a 24% and 39% MAE, and remarkable 27% and 51% RMSE on the test split, setting a new state- consity-based
of-the-art in detection- based counting. Notably, GeCo outperforms all single-stage counters (top part of Table [ by a large margin, which makes the first detection- counter that outperforms the longstanding total count estimation winner LOCA 4 by remarkable 27% MAE and 4% RMSE on test split. this respect, GeCo closes the performance gap that has been present for several years between ofte-of-the-art density-, and detection-based counters.
5

---

Table 1: Few-shot density-based methods (top part) and detection- -based methods (bottom part) performances on the FSCD147 19
Validation set
Test set
Method
MAE ())
RMSE(() AP(T)
APS0(1)
MAE(U)
RMSE()
AP(1)
AP50(1)
GMN 17
ACCVI8
26.52
124.57
MAMI.
ICMIL17
25.54
79.44
24.90
112.68
FamNet
CVPR2
23.75
69.07
22.08
99.54
CFOCNet
21.19
61.41
22.10
112.71
BMNet+ 26
15.74
58.53
14.62
91.83
VCN 22
19.38
60.15
18.17
95.60
SAFEC 32
15.28
47.20
14.32
85.54
CounTR 14
13.13
49.83
11.95
91.23
LOCA 4
JocV2s
10.24
32.56
10.79
56.97
C-DETR
BOCY22
20.38
$2.45
41.90
16.79
123.56
22.66
50.57
SAM-C 18
31.20
100.83
39.02
27.97
131.24
27.99()
49.17
PSECO
35
CVP06
15.31
68.363)
32.12
60.023
13.05
112.863
42.98
73.33
DAVE
20
CVPR3
9.75.))
40,30
24.20
61,08
10.45(2)
74,51
26,81
62.823
GeCo
(ow))
9.52
43.00.2)
33,51
62.51
7.91
54.28
43.42
75.06.D
In terms of detection performance, GeCo surpasses all state-of-thhe-arr methods, including PSECO 35 which uses both, SAM [12] and CLIP (1 backbones, by 1% AP, and 2% AP50. Note that GeCo also outperforms PSECO in count prediction by a large margin (~40%), which crucial, as an well feature good object localization. In addition, GeCo also outperforms SAM-C, which is a low-shot counting and detection extension of SAM by 70%/55% MAE/AP. To demonstrate the impact of the refinement step in existing methods, we modified DAVE (20) by feeding predicted bounding boxes to SAM [12] prompts, which results in a GeCo-like box refinement. Compared modified DAVE, GeCo achieves 21% and 16% higher AP and AP50, respectively, indicating that the reason for the excellent performance of GeCo lies in its architecture, rather than segmentation-bun based refinement.
ideal detection counter should deliver both accurate total count prediction
Figure visualizes detections for qualitative analysi GeCo predicts bounding boxes of superior quality for elongated objects (row 1), validating the selection of bounding box prediction locations. On detecting complex, non-blob-like objects (row 2), GeCo outperforms concurrent methods, by more accurately generalizing the prototypes. In densely populated scenes (row 3), GeCo achieves higher accuracy in both count and bounding box predictions. In comparison with state-of-the-Ae GeCo features better object discrimination (row 4), which can be attributed to better prototype generalization in DQE (Section 3.I and hard negative mining in the new loss from Section 3.4
We further evaluate GeCo FSCD-LVIS 19). Results in Table show that GeCo outperforms the best method by significant 178% and 73% AP and AP50, respectively, and performs on-par in terms of MAE. The experiment supports the results on FSCD147.
Table 2: Few-shot counting and detection on the FSCD-LVIS
19 "unseen" split.
Count
Detection
Method
MAE(()
RMSE(L)
AP(t)
AP50(1)
PSDetView- PB 29
TRAMI22
28.99
40.08
1.03
2.89
AutRN-PB
CVPR22
39.16
46.09
3.15
7.87
C-DETR 19
23.503
35.89
3.853)
11.28
DAVE 20 CVPR24
15.47
25.95
4.122
14.16
GeCo (ouns)
15.26
28.80.)
11.470
24.49
One-shot counting and detection. In the one-shot counting setup, single exemplar is considered. Table shows comparison with the recent density- and detection- methods. GeCo outperforms all state-of-the of-the-art single-stage density- specifically trained for the one-shot setup, by a significant margin of 35% MAE and 20% RMSE on validation and test split, respectively. GeCo also outperforms state-of-the-ee method PSECO 35 by
counters, outperforming LOCA shoot 4 version
'ee supplementary material for visualizations

---

DAVE
PSECO
CDETR
GeCo (ours)
1
ERR:
ERR:
ERR:
ERR: 51
ERR: 16
ERR: 25
ERR:
(3)
ERR: 65
ERR 79
ERR: 63
ERR:48
(4)
EAR:
ERR: 24
11
Figure 3: Compared with state-of of-thh--tttttttt few-shot detection- based counters DAVE 20 PSECO 65 and C-DETR [19], GeCo delivers more accurate detections with less false positives and better global counts. Exemplars are delineated with blue color, while segmentations are not shown for clarity.
4% AP and 5% AP50, and by significant 45% MAE and 49% RMSE on test split. These results show that GeCo features remarkable robustness to the number of exemplars since a single network (without re-training or fine-tuning) is used in both three- and one-shot setups. In particular, the performance drops by only 2%/1 % MAE/RMSE and 1%/1% % AP/AP50 on test split between both setups. In one-shol setting, GeCo surpasses state-of-the-ard three-shot models. Specifically, one-shot GeCo achieves 22% and 20% lower MAE and RMSE, respectively, compared to three-shot DAVE, and outperforms three-shot PSECO by 38% and 46% on the FSCD147 test set. These results highlight the robustness of GeCo to the number of exemplars, demonstrating its ability to handle inputs with lowered visual diversity.
Zero-shot counting and detection. Table reports the results of the zero-shot GeCo compared with best pero-shot variants of the density-based counters, LOCA (4) CounTR [14), RepRPN- 23 RCC [TO] and with the zero-shot variant of the best detection- counter DAVE [20]. GeCo outperforms DAVE (20] by significant margin of 14% MAE and 6% RMSE on the test set. Furthermore, outperforms
density- bbssd methods and sets new state-of-the-ard result on FSC (24) benchmark, by outperforming the top-performer Coun [14 by impressive 6% MAE on the test set. Since the zero-shot variant of the recent detection- based counter PSECO [35] does not exist, we include its prompt-based variant for complete evaluation (i.c., target object class is specified by a text prompt). Even in this setup, the zero-shot GeCo outperforms the prompt- based PSECO by 20% MAE 16% RMSE, and 2% AP50 demonstrating great robustness to different counting and detection scenarios.
Mutliclass images. To further verify the robustness of the proposed method, we validate it on subset FSCD147, that contain images with multiple object classes (FSCD147mal) 20). Results in Table indicate that most state-of-the-art methods non-discriminatively count all objects in an image due prototype over-generalization GeCo outperforms all single-stage density-, and detection- based counters on multiclass images by at least 60%/67% in MAE/RMSE, This further verifies the

---

Table 3: One-shot density-based methods (top) and detection-basee methods (bottom) on the
FSCD147 19
Validation set
Test set
Method
MAE (U)
RMSE(() AP())
APSI(1)
MAE())
RMSB(1)
AP(T)
AP50(1)
GMN 17
29.66
89.81
26.52
124.57
CFOCNet
WACVZI
27.82
71.99
28.
123.96
FamNet
24
26.55
77.01
26.76
110.95
BMNNtt+
26
17.89
61.12
16.89
96.65
CounTR
14
13.15
49.72
12.06
90.01
LOCA
4
11.36
38,04
12.53
75.32
PSECO
B5
18.3103
80.73(3
31.47
58.53
14.86(3
118.6433
41,632)
70.87
DAVEr-ba CVRR24
10.98
43.26
18.00.3
52.37()
11.54
86.62
$5,27(3 19.463
GeCo (oun))
9.97
37.85
32.82
61.31
8.10.)
60.16
43.11
74.31
Table 4: sero-shot density-based methods (top part), and detection- methods (bottom part) the FSCD147 19]
The symbol * denotes methods that also use text prompts as input.
Validation set
Test set
Method
(L)
RMSE() AP(T)
APS0(1)
MAE())
RMSE()
AP()) APS0(t)
RepRPN-C 23 ACCV22
29.24
98.11
26.66
129.11
RCC 10 xXxx2
17.49
58.81
17.12
104.5
CounTR
aMver
17.40
70.33
14.12
108.01
LOCA ICCV23
17.43
16.22
103.96
PSECO RS CVTRM
23.90C3
100.33(3)
16.583
129.77
41.14
69.03(2)
DAVE 20 CVPR2
15.71
60.34
16.31
46.87
15,51
116.54
18.553
$0.08
GeCo (cars)
14.81
64.95(2)
31.04
58,30
13.30(D
108.72
41.27
70.09
robustness of the proposed architecture, which benefits from the hard-negative mining in the proposed loss function, leads to more discriminative prototype construction and false positive reduction.
4.2 Ablation study
Dense object detection loss. To analyze the contribution of the new dense detection loss from
Section 3.4] trained GeCo using the standard loss (20) 33 that forms the ground truth objectness
score by placing unit Gaussians on object centers this variant is denoted by GeCoGans:- Table 6
shows that this leads
substantial drop in total count estimation (38% RMSE, and 34% MAE)
as well as in object detection (6% AP, and 3% AP50). Qualitative results are provided in Figure As observed in columns 3 and the classical to predict object locations from the object centers, which are not necessarily optimal for bounding box prediction. contrast, the proposed dense detection loss enables the network to learn optimal point prediction, which more accurately aggregates information of the object pose. Columns 1 and indicate that the new loss leads superior detection of objects composed of blob-like structures avoiding false detections on individual object parts. Furthermore, the hard-negative mining integrated the new loss design leads better discriminative power of the detections and subsequent reduction of false positives (column 4).
Sait-Guussin--ba based loss 20 35 forces the network
Table 5: Performance on FSCD147 19 test split, and its multiclass subset FSCD mu]
FSCD147
FSCD147 mul
Method
MAE(L)
RMSE(L)
MAE({)
RMSE(L)
C-DETR 19
ECCV22
16.79
123.56
23.09
30.09
PSECO 35
CVPR24
13.05
112.86
25.73
44,95
LOCA IUVVI3
10.79(
56.97
21.28
43.67
Coun TRunTR
4
BMVC22
11.95
91.23
14.56
27.41
DAVE 20
10.45
74.51
3.09
5.28
GeCo lours)
7.91(D
54.28()
5,88
9.17

---

Table 6: Ablation study on the FSCD147
19
validation split.
Counting
Detection
Method
MAE(()
RMSE(1)
AP(1)
AP50(1)
GeCo
9.52
43.00
33.51
62.51
GeCoGauss
12.79
59.33
31.43
60.73
GeCono
10.04
47.11
33.08
62.50
GeCop
9.97
46.93
32.56
61.19
GeCoRef
10.26
43.33
24.63
61.57
GeCoq
10.32
45.14
33.01
61.68
GeCoDETR
11.45
52.46
32.24
61.60
(2]
(3)
(4
(5)
GeCo
GeCocuu
Figure 4: Response maps (in yellow), and locations for bounding box predictions (red dots) when using the proposed (first row) and the standard 20 4 35 (second row) training loss.
Architecture. To evaluate the impact concatenating the SAM-HQ [1] features the query unpacking process in the DQD module (Section 3.2), remove these features in GeCogy Table 6 shows counting performance drop 5% MAE and 10% RMSE. To validate the importance of modeling exemplar shapes, i.c., width and height, with prototypes pS, we omit them in GeCo s We observe a substantial performance decrease of 5% MAE, and 9% RMSE. Finally, we remove bounding box refinement in the detection refinement module (Section 3.3), and denote the variant as GeCorer While this does not affect the global count estimation accuracy, we observe 26% decrease in AP and 2% decrease in AP50. It is worth noting, that bounding box refinement improves the accuracy of predicted bounding boxes, however does not enhance object presence detection.
To verify the importance of the DQE module (Section 3.1 replace the dense object queries Q con- struction step (2) with standard self-attention, i.c.. Q SA(P)3x This Ieads to a 8% MAE and 5% RMSE performance drop, verifying the proposed approach. To evaluate the importance of using image features as queries (2), we change the object query construction to Q2- = CA(SA(Q)
f f to follow standard DETR like approach, and denote it as GeCOpETR- We observe a 20% MAE and 22% RMSE decrease in counting performance.
Conclusion
We proposed GeCo, novel single-stage low-shot counter that integrates accurate detection, seg- mentation, and count prediction within unified architecture, and covers all low-shot scenarios with a single trained model. GeCo features remarkables dense object query formulation, and prototype generalization across the image, rather than just into a few prototypes. It employs novel loss function specifically designed for detection tasks, avoiding the biases of traditional Gaussian-based losses. The loss optimizes detection accuracy directly, leading more precise detection and counting.

---

The main limitation of the presented method is that it cannot process arbitrarily large images, due to memory constraints, since it, as all current methods, operates globally. In future work, we will explore local counting, incremental image-wide count aggregation, optimizing inference speed utilizing faster backbone
34
Extensive analysis showcases that GeCo surpasses the best detection- based counters by approximately 25% in total count MAE, achieving state-of-the-ar performance in a few-shot counting setup and demonstrating superior detection capabilities. GeCo showcases remarkable robustness to the number of provided exemplars, and sets a new state-of-the-ane art in one-shot as well as zero-shot counting.
Acknowledgements. This work was supported by Slovenian research agency program P2-0214 14 and projects J2-2506, L2-3169, Z2-4459, J2-60054, and by supercomputing network SLING (ARNES, EuroHPC Vega IZUM).
References
[1] Nicolas Carion, Francisco Massa, Gabricl Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-ene end object detection with transformers. In European conference on computer vision,
pages 213-229. Springer, Spriner,, 2020.
[3] Zhe Dai, Huansheng Song, Xuan Wang, Yong Fang, Xu Yun, Zhaoyang Zhang, and Huaiyu Li. Video-based vehicle counting framework. IEEE Access. 1144460-64470) 2019.
[4] Nikola Djukie, Alan Lukezic. Vitjan Zavrtanik, and Matej Kristan. A low-shot object counting network
with iterative prototype adaptation.
ICCV, 2023.
[5] Thorsten Falk, Dominic Mai, Robert Bensch, Ozgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zne Jäckel, Katharina Seiwald, et al. U-net: deep learning for cell counting, detection, and morphometry. Nature methods, 16(1):67-70) 2019.
[6] Qi Fan, Wei Zhuo, Shi-Keung Tang, and Yu- Wing Tai. Few-shot object detection with attention- and multi-relation detector. In CVPR, pages 4013-40223 2020.
[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for faas adaptation of deep networks. In International conference on machine learning, pages 1126-1 1135. PMLR, 2017. [8] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356-5364, 2019.
[9] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE
international conference on computer vision, pages 2961-2201000 16699, 2017.
[10] Michael Hobley and Victor Prisacariu. Learning to count anything: Reference-less class-agnostic counting with weak supervision. arXiv preprint arXiv:2205. J0203, 2022.
[11] Lei Ke, Mingqiao Ye, Martin Danelljan, Yu- Wiig Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. Advances in Neural Information Processing Systems, 36, 2024.
[12] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026. 2023.
[13] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly,
2(1 21122):33977, 1955.
[14] Chang Liu. Yujie Zhong. Andrew Zisserman. and Weidi Xie. Countr: Transformer-based generalised visual counting. In BMVC. BMVA Press, 2022.
[15] Weizhe Liu, Mathicu Salzmann, and Pascal Fua. Context-aware crowd counting, In CVPR, June 2019.
[16] Tya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
[17] Erika Lu, Weidi Xie. and Andrew Zisserman. aaasstagnostic counting. In Computer Vision-ACCV 2018: Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pages 669-684. Springer, 2019.
14th Asian Conference
[18] Zhiheng Ma, Xiaopeng Hong, and Qinnan Shangguan. Can sam count anything? an empirical study on sam counting. 2023.
[19] Thanh Nguyen, Chau Pham, Khoi Nguyen, and Minh Hoai. Few-shot object counting and detection. In ECCV, pages 348-365. Springer, 2022.
[20] Jer Pelhan, Alan Lukežič, Vitjan Zavrtanik, and Matej Kristan. Dave venert-and--eerfy paradigm for low-shot counting. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
[21] Alec Radford, Ilya Sutskever, Jong Wook Kim, Gretchen Knueger, and Sandhini Agarwal. Clip: connecting text and images. OpenAl. https://openai. com/blog/clip/. 2021.
[22] Viresh Ranjan and Minh Hoai. Vicinal counting networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pages 4221 12300, June 2022.
10

---

[23] Viresh Ranjan and Minh Hoai Nguyen. Exemplar free class agnostic counting. In Proceedings of the Asian
Conference on Computer Vision, pages 3121-3137. 2022.
[24]
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. In CVPR.
pages 3394-3403, 2021.
[25] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese.
Generalized intersection over union. CVPR, 2019.
[26] Min Shi, Hao Lu, Chen Feng, Chengxin Liu, and Zhiguo Cao. Represent, compare, and learn:
similarity-aware framework for class-agnostic counting. In CVPR, pages 9529-9538, June 2022. [27] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627-9633 2019.
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems,
30, 2017.
[29] Yang Xiao, Vincent Lepetit, and Renaud Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. IEEE TPAMI, 45(3):3090-31000 2022.
[30] Weidi Xie, J Alison Noble, and Andrew Zisserman. Microscopy cell counting and detection with fully convolutional regression networks. Computer methods in biomechanics and biomedical engineering:
Imaging Visualization, 6(3):283-292 2018.
[31] Shuo-Diao Yang, Hung-Ting Ting Su, Winston H Hsu, and Wen-Chin Chen, Class-agnostic few-shot object counting. In WACV, pages 870-878, 2021.
[32] Zhiyuan You, Kai Yang. Wenhan Luo, Xin Lu, Lei Cui, and Xinyi Le. Few-shot object counting with
similarity-awar aware feature enhancement. In WACV, pages 2315-6324, 2023.
[33] Vitjan Zavrtanik, Martin Vodopivec, and Matej Kristan. segmentation-base
approach for polyp counting in the wild. Engineering Applications of Artificial Intelligence 88:103399, 2020.
[34] Xu Zhao, Wenchao Ding, Yongqi An. Yinglong Du Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. Fast
segment anything. arXiv preprint arXiv:2306. 230.11215.. 2023.
[35] Huang Zhizhong, Dai Mingliang, Zhang Yi, Zhang Junping, and Shan Hongming. Point, segment and count: A generalized framework for object counting. In CVPR, 2024.

---

A Supplemental material
This supplementary material provides additional comparisons of GeCo with ofate-of-the-ar under non-standard experiment, and provides additional qualitative examples.
Performance analysis on a non-standard experiment. The analysis of the detection methods in Section 4 adheres the standard evaluation protocol [19] 20), where method predicts set of bounding boxes for each image. The estimated count is the total number of predicted bounding boxes, and evaluated by the MAE/RMSE measures, while the detection accuracy is evaluated by AP/AP50 measures. Both measures are computed on the same ser of output bounding boxes.
paper, the reported evaluation deviated from the standard one in an important detail. Namely different outputs were evaluated under MAE/RMSE and AP/AP50 fully evaluate the different properties of the method. AP/AP50 was computed in all output bounding boxes, while the MAE/RMSE were computed on a subset of the boxes, obtained by thresholding the response score. In Section 4 we evaluated all methods, including PSECO under the standard experiment. Nevertheless, we additionally report GeCo evaluated under the said non-standard PSECO experiment in 7
However, in the PSECO 35
Even in this setup, GeCo outperforms PSECO by 4%/4% AP/AP50, and 1%/2% AP/AP50, on validation and test set, respectively, again with a substantially lower global count errors (~50% MSE/RMSE reduction). These results shed important insight. A method producing false positives, which increase the count errors and reduce its usefulness for counting, might achieve good detection- oriented performance measures. Thus for counting performance evaluation, the MAE/RMSE should be considered primary measures, while AP/AP50 should be secondary, as they are less strict towards false positive detections.
Table 7: Few-shot saocctio-bbase counting evaluation on FSCD 19 under the non-standard
evaluation protocol (35]
Validation set
Test set
Method
MAE (L)
RMSE(L)
AP(1)
AP50(7)
MAE())
RMSE(L)
AP(1)
AP50(1)
PSECO 35
15.31 9.52
68.34
32.71
62.03()
13.05
112.86
43.53
74.64
GeCo (ours)
43.00
34.07
64.23
7.91
54.28
43.89
76.18
Performance in crowded scenes. To evaluate counting performance in crowded scenes, con- structed a subset of the FSCD147 test set by including images with at least 200 objects and a maximal average exemplar size of 30 pixels. Notably, the new subset contains 42 images, averaging 500 objects per image, thus featuring dense scenes with small objects. Three top-performing methods from Table were included in the comparison and are shown in Table g GeCo outperforms both PSECO and DAVE by significant margin, e.g., outperforming DAVE by 23% in MAE and 36% in RMSE, which demonstrates superior counting performance on small, densely populated objects.
Table 8: Few-shot counting in crowded scenes, comparing the top-three detection based counters from T
MAE
RMSE
PSECO 35
CVPR24
173.64
594.91
DAVE 20
CVPR24
81.38
383.93
GeCo (oum)
62.60
242.82
Qualitative results. Figure compares GeCo with PSECO [35. which achieves the best AP/AP50 measures among the related counters. GeCo shows robust performance, achieving high precision (see 5 1), while achieving high recall (see 3 block 2). This challenging for related methods, particularly in densely populated scenes or with small objects. Furthermore, GeCo outperforms PSECO on elongated or more complex objects (see Figure 5 3), better exploiting the exemplars.
12

---

GeCo
ERR: 2
ERR: O
ERR: 0
D
PSECO
ERR: 23
ERR:
ERR:
(2)
coo o
GeCo
ERR: 35
ERR: 0
ERR:
PSECO
401
ERR:
ERR:
GeCo
ERR:
ERR:
ERR: 2
PSECO
ERR:
ERR:
Figure 5: Comparison of few-shot counting on FSCD147. Exemplars are shown with red color and ERR indicates count eтror.
13

---

Figure 6 visualizes the segmentations produced by GeCo, in a few-shot setup, of various objects in diverse scenes. GeCo is robust noise, achieves discriminative segmentations. and performs well elongated, non-blob-like objects and in dense scenarios. Figure F compares GeCo with all state-of-the-art detection counters [20] 19 35 GeCo achieves superior counting performance, and predicts more accurate bounding boxes.
Figure 6: Segmentation quality of GeCo diverse set of scenes and object types. Exemplars denoted by red bounding boxes.
14

---

DAVE
PSECO
CDETR
GeCo (ours)
(
ERR:
EAR:
ERR:
ERR:
ERR: 3
ERR: 7
ERR:
ERR
ERR:
ERR:
ERR: 64
ERR:
ERR: 23
ERR:
ERA:
ERR:
(4)
ERR:
ERR:
ERR:
(6)
ERR
ERR:
Figure 7: Comparison of few-shot counting and detection on FSCD147. ERR indicates count error.
15

---

In Figure 8 performance of GeCo is qualitatively demonstrated on examples with high intra-class variance. Image (a) displays marbles of various colors and textures (notable visual intra-class variance), all correctly detected and still distinguished from a visually similar coin. Example (b) shows donuts with different colors of decorations, all accurately counted and detected by GeCo. Image (c) contains bottles of various sizes, shapes, and colors, each with distinct sticker. Image (d) features transparent food containers with differently colored and shaped fruits inside, successfully detected despite significant visual diversity. Examples (e) and (f) illustrate GeCo's robustness in detecting objects with high shape variance, including partially visible birds (notable object shape intra-class variance).
(8)
06
ERR:
ERR:
(c)
(d)
ERR:
ERR: 7
ERR:
ERR:
Figure 8: Few-shot detection and counting with GeCo images with high intra-class object appearance variation. Orange and red bounding boxes denote detections and exemplars, respectively. Count error is denoted by ERR.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract and introduction, we stress out the main contributions, and results of the presented method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
We discuss the main limitation of the presented method in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
We do not derive theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification: The method clearly described, making it possible to re-implement. We will make the code publically available upon acceptance.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Upon acceptance, we will make the code and models publically available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
mublic//guides//odeSuuuii wow.ssabbsssiooooioooooooooooiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https CodeeSumiiii wwwiiisssooosoiiiio
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper clearly specifies all implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars and statistical significance are not reported by state-of-the-ar methods in their respective papers. We omit the usage of error bars, it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the implementation details we clearly state the computer resources needed to train presented model.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We reviewed the Code of Ethics and found no violations in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work will not make any societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our data and models are not risk of misuse.
21

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite respective papers of publically available benchmarks and datasets used in our work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pww.::s...aaseiiiia...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
22

---

Answer:
[NA]
Justification:
We do not conduct research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Our research does not include any research with human subjects.
Guidelines:
We do conduct research with human subjects.
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "D7ldV9AxeG", "reviewer": "Reviewer_s2FV", "summary": "This paper address the issue for low-shot and zero-shot object counting, with an object detection-based approach. The proposed method heavily uses SAM framework, to provide feature embeddings and refine detection boxes. Attention-based feature aggregation and SAM-HQ are used to get the final features for objectiveness and bounding boxes. Evaluations are carried out on standard datasets with few/one/zero-shot setups, and the proposed GeCo outperforms many state-of-the-art methods.", "questions": "Does the use of SAM in multiple places bring advantages over other baselines?", "limitations": "The limitations are sufficiently discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The proposed method achieves high quality on multiple metrics, including counting accuracy and detection accuracy, and it works well for both low-shot and zero-shot setups.\n* The counting loss directly optimizes the detection tasks and greatly improves the model's performance.", "weaknesses": "* One concern is that the paper utilizes SAM as its backbone, which is trained on SA-1B dataset. This may leads to some benefits to GeCo by having seeing large amount of diverse data. More discussion is needed to address this.\n* Also related to SAM, in table 6, it shows that without detection refinement module, the $GeCo_{\\overline{Ref}}$'s performance drops heavily and goes lower than other methods in table 1. I am wondering what if we also feed other detection-based methods' output into SAM decoder. This may be a more fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**GeCo uses SAM, trained on the SA-1B dataset, as its backbone, potentially benefiting by seeing diverse data.**\n\nEmpirical evidence in the Experimental results (Section 4.1) indicates that the key performance gain does not come from the amount of training data in the SAM backbone. For example: the performance of SAM-C, which is a low-shot counting and detection extension of SAM, is significantly lower compared to GeCo (70\\%/55\\% lower MAE/AP). Another state-of-the-art low-shot counting method, PSECO, uses SAM as well as CLIP backbones (also trained on huge and diverse datasets) but performs significantly worse than GeCo. In particular, GeCo outperforms PSECO by 39\\% in MAE on FSCD147. These results strongly support our claim that the excellent performance of GeCo does not come from the SAM backbone, but the proposed architecture and the new training loss. We will emphasize these points in the camera-ready.\n\n\n>**GeCo$\\_{\\overline{\\text{Ref}}}$ performance drops without refinement module. Apply SAM refining also to sota bboxes.**\n\nNote that the refinement module is used to improve **only bounding box accuracy**, not object presence detection, and has in fact a minimal impact on the final count accuracy. This is apparent from that GeCo$\\_{\\overline{\\text{Ref}}}$ achieves a similar counting performance as GeCo (only 7% MAE and 1% RMSE performance drop). A small decrease (2%) in AP50 confirms that GeCo$\\_{\\overline{\\text{Ref}}}$ robustly detects the objects. As the reviewer correctly identified, a larger performance drop (26%) is observed in AP, but this is because the GeCo bounding boxes more accurately fit the detected objects and not because more objects are localized or fewer are missed. The detection performance of GeCo$_{\\overline{\\text{Ref}}}$ remains superior to the best-performing methods without SAM (e.g., DAVE, C-DETR), where PSECO is omitted as it uses SAM for bounding box prediction. \n\nAdditionally, as the reviewer suggested, we feed the most recent state-of-the-art counter DAVE’s predicted bounding boxes as prompts to SAM (DAVE+SAM) and observe improved detection performance by 34\\% AP. Note, however, that GeCo still outperforms DAVE+SAM by a large margin (21% in AP and 16% in AP50), which indicates the superiority of the proposed architecture and not the use of SAM. We will add this discussion in the camera-ready.\n\n\n>**Does the use of SAM in multiple places bring advantages over other baselines?**\n\nAs demonstrated in our results, GeCo outperforms all state-of-the-art which is also based on SAM (e.g., SAM-C and PSECO), as well as the recent DAVE with SAM refinement step added (DAVE+SAM) – we will add the latter into the camera-ready. The results clearly show that the SAM refinement and the backbone are beneficial, but GeCo still by far surpasses all of the aforementioned sota. We will add this discussion in the camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8mfbu0zjSo", "reviewer": "Reviewer_HCSh", "summary": "Paper tackles the task of few-shot and zero-shot class agnostic counting, and presents Geco, a unified counting framework that can detect, segment and count objects. Geco uses SAM backbone for feature extraction, and implements Dense Query Encoder (DQE) and Dense Query Decoder (DQD) to detect prototypes. For few shot setup, prototypes are obtained from the exemplars, and zero shot setup uses learned objectness prototypes. DQD outputs a set of dense detections, which are further refined by the SAM decoder, by considering DQD detections as prompts for the SAM decoder.", "questions": "I would request the authors to kindly address the points I raised in Weaknesses section.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Paper presents a unified approach for few shot and zero shot class agnostic object counting which achieves good results on standard benchmark datasets.", "weaknesses": "1. Lines 110 - 114: authors claim that existing low shot counting approaches uses a small number of prototypes or exemplars, often leading to false detections. Authors say their approach is different from these existing approaches. However, eqn 1 in Sec 3.1 appears to be conceptually similar to the existing approaches, i.e. few prototypes p are used to create keys and values in a cross attention layer, where the query is formed by the image features.\n2. Looking at the ablation study in table 6, it appears that removing SAM decoder degrades the performance of Geco significantly in terms of AP, which raises some questions about the effectiveness of the approach. How does Geco perform without SAM decoder on the test set of FSCD147 ?\n3. Related to my point 2 above and the ablation study in table 2: it would be useful to have a baseline without using Geco, by simply using the exemplars as prompts for the SAM decoder.  This baseline does not require any new training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Prototype construction is conceptually similar to related methods.**\n\nWhile Eq. 1 is indeed conceptually similar to the existing methods, there are fundamental differences in the function of the output and subsequent steps (Eq. 2).\n\nThe existing methods perform detection by correlating exemplar prototypes directly with the image features and decoding the resulting features into a density map, or applying DETR-like detection on them. In both cases the features are assumed to have desired objects “highlighted” and others “suppressed”. Since the prototypes are expected to cover a broad range of appearances, the trained networks typically over-generalizes to achieve a high recall which leads to false positive detections. \n\nConcretely, related methods (e.g., LOCA) transfer information from the image into prototypes (i.e., attention keys and values are image features, while prototypes are attention queries) to enrich the prototypes, which leads to prototype over-generalization, since all appearance diversity is packed into only three prototypes, which are ultimately correlated with the features to detect object locations.\n\nA related method, CounTR, uses image features as attention queries and prototypes as attention keys and values, which is indeed similar to GeCo. But the result of this operation are “prototype-matched” features with objects “highlighted”, which are then decoded into a density map. This is functionally fundamentally different from GeCo. Since the information of only three prototypes is transferred into each image feature, the few prototypes are thus required to cover all appearance variations of target objects in the image, often, leading to over-generalization. \n\nIn contrast, in GeCo, Eq. 1 transforms the few prototypes (Roi-pooled visual exemplars, and shapes exemplars) into a dense set of location-specific prototypes $\\mathbf{P}$, i.e., one prototype per image location (feature space resolution). These prototypes are then not decoded into density map as in related works, but are used to construct dense local object queries $\\mathbf{Q}$ through further attention mechanisms (Eq. 2). The latter are finally decoded into object detections. This process avoids over-generalization and false detections, and from this perspective differs from the existing methods. We will emphasize these points in the camera-ready. \n\n>**Performance decrease upon removing SAM decoder. Show ablation performance on FSCD147 test set as well.**\n\nThe SAM decoder (refinement module) primarily just adjusts the predicted box boundaries, thus improving the **accuracy** of box predictions, but minimally changes the overall count (i.e., object detection), as is evident from the results in Table 6. In particular, the counting performance of GeCo$\\_{\\overline{\\text{Ref}}}$ is nearly identical to the original GeCo (only 7% MAE and 1% RMSE performance drop). Similarly, the AP50, which indicates detection robustness, also remains consistent (a mere 2% drop). However, the AP of GeCo is higher (26%) since the detected boxes more accurately encompass the objects. We will add this discussion in the camera-ready.\n\nThe reviewer mentions ablating GeCo on the test set. Ablation experiments are not performed on the test set in the paper, since the test set is meant for final performance evaluation only, by the FSCD protocol to prevent overfitting. However, to answer the reviewer’s question, we have run GeCo as requested: GeCo$\\_{\\overline{\\text{Ref}}}$ achieves 8.96 MAE, 51.11 RMSE, 32.43 AP and 70.43 AP50 on test set, which still outperforms both DAVE by 21\\%AP, 11\\%AP50 and C-DETR by 30\\%AP, 28\\%AP50. Note that PSECO must be omitted from this comparison since it utilizes SAM for bounding box prediction, thus comparison with GeCo$\\_{\\overline{\\text{Ref}}}$ is neither meaningful nor fair. These results further confirm the effectiveness of GeCo.\n\n>**Include SAM as a baseline counter.**\n\nA simple baseline as the reviewer suggested is not possible, since SAM decoder segments only a single object instance for a given prompt (exemplar bounding box) and is not able to detect multiple objects across the whole image based on the prompt. However, a more advanced baseline that applies SAM with minimal modifications already exists in the literature and is included in Table 1, denoted as SAM-C [1]. This approach is outperformed by all other methods, in particular, GeCo outperforms it by 70\\% in MAE and 55\\% in AP. This result indicates that the reason for the excellent performance of GeCo is not in the SAM backbone, but rather in the proposed architecture (and the  novel loss function). \n\n[1] Ma, Z., Hong, X., & Shangguan, Q. (2023). Can SAM Count Anything? An Empirical Study on SAM Counting. arXiv [Cs.CV].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YAc9wpblpw", "reviewer": "Reviewer_QayL", "summary": "This paper introduces GeCo, a novel unified architecture for low-shot counting that integrates object detection and segmentation. GeCo addresses the limitations of current state-of-the-art methods by generalizing object prototypes across diverse appearances and introducing a new counting loss that directly optimizes the detection task. The architecture achieves significant improvements in detection accuracy and count estimation over previous methods and sets a new benchmark in low-shot counting.", "questions": "1. Why was the SAM framework specifically chosen over other backbone networks? How does it compare in terms of generalization and performance with other pre-trained models like ResNet or EfficientNet?\n2. What specific design elements make GeCo particularly suitable for low-shot learning? Are there any modifications or components that specifically address the challenges of low-shot scenarios?\n3. How does GeCo perform on datasets with significant intra-class variability? Can it generalize well to objects with a high degree of appearance variation within the same category?", "limitations": "1. The method's performance still heavily depends on the quality and representativeness of the provided exemplars. Poor quality or unrepresentative exemplars could negatively impact the detection and counting accuracy.\n2. While the dense object query formulation improves generalization, the method's ability to handle extreme variability in object appearances within the same category needs further validation.\n3. The effectiveness of GeCo in scenarios involving small objects or highly dense regions (e.g., crowd counting, face detection in crowds) is not thoroughly evaluated. These scenarios often present unique challenges that may require additional considerations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. GeCo combines object detection and segmentation in a single framework, enhancing efficiency and performance.\n2. The novel dense object query formulation improves the generalization of object prototypes across diverse appearances, reducing false positives.\n3. The proposed counting loss directly optimizes detection tasks, leading to more accurate object counts and better handling of annotation noise.\n4. GeCo surpasses existing few-shot detection-based counters by approximately 25% in total count MAE and shows superior detection accuracy. The method demonstrates substantial robustness to the number of exemplars, performing well even in one-shot and zero-shot scenarios.", "weaknesses": "1. While SAM's pre-training provides strong generalization with other potential backbone networks, it is unclear if similar results could be achieved with other networks like ResNet or EfficientNet. Thus the complexity and computational requirements of the proposed GeCo architecture might limit its scalability and real-time applicability, especially in resource-constrained environments.\n2. Although the method is claimed to be low-shot friendly, the paper does not provide explicit architectural components or techniques specifically designed for low-shot scenarios.\n3. The performance of GeCo on highly heterogeneous datasets, where object appearances vary significantly within the same category, is not adequately demonstrated.\n4. The effectiveness of this method in scenarios involving small objects or high-density areas, such as crowd density estimation or face detection in dense crowds, has not been well explored as this type of dataset has large-scale variations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Why was SAM backbone chosen over other backbones?**\n\nWe primarily wanted to use SAM as a box refiner, due to its accurate mask prediction, which can easily be converted to a bounding box. For computational efficiency and to keep the framework unified, we then decided to also use the SAM backbone for feature extraction in GeCo. This makes GeCo simply another head on the SAM backbone, thus adding another capability to the SAM Swiss-knife.\nHowever since the proposed GeCo detection pipeline is general, any backbone network could be used instead of SAM in the image encoding stage. We do not expect significant performance drop due to a different backbone, since the networks such as Resnet have been shown to produce features with excellent detection capabilities. \n\nNote, however, that, to optimize the inference speed and memory consumption of GeCO, SAM could be replaced with the recently proposed Fast Segment anything [1] that speeds up the original SAM inference 50x and features only a marginal segmentation accuracy loss. This will be the subject of future work and we will add this discussion into the final paper. \n\n[1] Zhao, X., et. al (2023). Fast Segment Anything. arXiv [Cs.CV]. \n\n>**Why is GeCo particularly suitable for low-shot scenarios?**\n\nThe proposed prototype construction addresses a specific low-shot prototype learning issue in counters. The current state-of-the-art low-shot counters predominantly construct a few prototypes from the annotated exemplars. Since the few prototypes are expected to cover a broad range of object appearances, the trained networks typically learn to over-generalize which leads to false positive detections. In contrast, GeCo non-parametrically (i.e., exploits all feature locations) constructs dense location-specific prototypes, thus avoiding the need for only few prototypes to cover the entire appearance diversity of the present objects. This ensures a high recall at high precision (i.e., minimal false positive detection) as verified in our experiments. Furthermore, the non-parametric formulation enables excellent performance in highly dense and crowded regions, where classical DETR-like counters, which can handle only a limited number of queries, fail.\n\nAnother low-shot counting modification is the new detection-optimized loss, which avoids the standard losses that force the network to predict a unit Gaussian on the object center and are prone to center annotation noise issues and ad-hoc setting of the kernel width hyperparameter. The new loss avoids these issues completely by training the network to predict response maps on which objects are reliably detected by nms. Results show significant performance boosts. \n\n>**How does GeCo perform on high intra-class variability?**\n\nWe report in the paper the GeCo performance on the FSCD-LVIS dataset, which is derived from a detection dataset and in which the objects feature a high intra-class variability. The results are reported in Table 2 of the paper. Note that  GeCo outperforms all state-of-the-art by a significant margin, implying superior capability in handling the visual diversity. \nFor qualitative analysis, we show examples of objects with significant visual intra-class variability in Figure 1 (see attached rebuttal PDF) which will be added to the supplementary material in the final version of the paper.\n\n>**Performance depends on the quality of exemplars.**\n\nWe agree that exemplar annotation quality plays an important role, and that all low-shot counting algorithms rely on the accuracy and representativeness of the exemplars. Since only a few exemplars (~3) are required, it is safe to assume that in practice the user will pay attention to their annotation. \n\nNevertheless, we did include the experiment where the number of exemplars varied from 3 to 1 (Section 4.1, Table 1 and 2). Performance of GeCo drops by only 2\\%/11\\% of MAE/RMSE and 1\\%/1\\% AP/AP50 on the test split between three-shot and one-shot counting, which is almost a negligible performance drop, indicating a degree of robustness to exemplar selection.\n\nThe results are even more impressive when comparing the **one-shot GeCo** to the **three-shot sota** methods. In particular, the one-shot GeCo outperforms the three-shot DAVE by 22\\% and 20\\%, and three-shot PSECO by 38\\% and 46\\% in terms of MAE and RMSE on the test set of FSCD147, respectively. This is a strong indicator that GeCo does not require finely selected exemplars that cover a broad range of appearances, as might be the case with sota. We attribute this to dense prototype and query construction process in GeCo. We will more clearly expose this property in the final version of the paper and we plan to further explore this interesting aspect that the reviewer pointed out in our future work.\n\n>**What is performance with small objects or highly dense regions (eg. crowd counting)?**\n\n\nTo address the reviewer’s concern, but still remain in the object-agnostic low-shot counting and detection setup, we created a subset of the FSCD147 test set, where only images with at least 200 objects with an average exemplar size of  30 pixels are considered. This subset consists of 42 images (~500 objects per image in average), making it a solid representative of dense scenes with small objects. We evaluated the three top-performing methods from Table 1 (including GeCo) on this dataset. Results are shown in the table below. GeCo outperforms both PSECO and DAVE, with the latter being outperformed by 23% in MAE and a significant 36% in RMSE.  We will add this table to the supplementary material of the final paper. Note that performance comparison in dense scenes is already qualitatively verified in an example in the supplementary material, Figure 5, example (2).\n\n|       | MAE    | RMSE   |\n| ----- | ------ | ------ |\n| PSECO | 173.64 | 594.91 |\n| DAVE  | 81.38  | 383.93 |\n| GeCo  | 62.60  | 242.82 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Firstly, we sincerely thank the reviewers for their constructive feedback and hope that responses to your questions clarify the strengths and innovative aspects of GeCo. We appreciate the recognition of novelty and state-of-the-art counting and detection performance. In the following, we summarize our responses, but detailed comments are discussed with each reviewer separately. 

Reviewers pointed out that GeCo's strong performance might be due to SAM backbone, trained on the extensive SA-1B dataset. Experimentally, we demonstrated that the strong SAM-based baseline (denoted by SAM-C) achieves significantly lower performance on FSCD147 compared to GeCo, indicating that a powerful backbone (e.g., SAM) is not enough for excellent few-shot counting and detection results. Moreover, GeCo also outperforms a recent PSECO, that uses SAM and CLIP, combined trained on more training data. In the rebuttal, we also clarify that the SAM-based decoder in GeCo is used to refine detections only, i.e., to improve the bounding box accuracy, and does not significantly impact detection robustness and overall count. These results indicate that the key to GeCo’s state-of-the-art performance are the proposed architecture and loss function, rather than the backbone, amount of training data or decoder. 

In the rebuttal, we also clarify that the formulation of prototype construction in GeCo is conceptually different from existing low-shot counting and detection methods. In particular, GeCo transforms the few exemplars into a dense set of location-specific prototypes (one prototype per image location). These prototypes are then used to construct dense local object queries, which are finally decoded into object detections. This process avoids over-generalization and false detections and from this perspective differs from the existing methods. Compared to classical DETR-like counters, which can handle only a limited number of queries, the formulation with dense object queries in GeCo enables excellent performance in highly dense and crowded regions. In addition to the architecture formulation, the new loss avoids the issues related to the construction of Gaussian-based ground truth by training the network to predict response maps on which objects are reliably detected by non-maxima suppression.

Performance on objects with high intra-class variability is already demonstrated on the FSCD-LVIS dataset (Section 4.1) — GeCo outperforms state-of-the-art methods by a significant 178% and 73% in AP and AP50. In addition, we include a figure in the rebuttal PDF, to qualitatively demonstrate GeCo’s low-shot counting and detection capabilities on images with high object visual diversity. 

In the rebuttal, we also discuss the dependency on the quality of exemplars of low-shot counters. An exhaustive evaluation of this aspect goes beyond the scope of this paper, but an experiment comparing performance on three- and one-shot scenarios (Section 4.1) indicates that GeCo is much more robust to exemplar diversity and selection than other methods. We also show that GeCo significantly outperforms existing methods in dense scenes and small objects, which further highlights its effectiveness in challenging scenarios.

Meta-review: This paper is about "low-shot counting" of objects. After discussion between reviewers and authors, the paper received two ratings of  "borderline accept" and one of "weak accept". The AC recommends that the paper be accepted.
Decision: Accept (poster)