A Novel Unified Architecture for Low-Shot Counting by Detection and Segmentation

Authors: jer  pelhan
alan  lukežič
vitjan  zavrtanik
matej  kristan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:15:50

Abstract: Low-shot object counters estimate the number of objects in an image using few or no annotated exemplars. Objects are localized by matching them to prototypes, which are constructed by unsupervised image-wide object appearance aggregation.
Due to potentially diverse object appearances, the existing approaches often lead to overgeneralization and false positive detections.
Furthermore, the best-performing methods train object localization by a surrogate loss, that predicts a unit Gaussian at each object center. This loss is sensitive to annotation error, hyperparameters and does not directly optimize the detection task, leading to suboptimal counts.
We introduce GeCo, a novel low-shot counter that achieves accurate object detection, segmentation, and count estimation in a unified architecture.
GeCo robustly generalizes the prototypes across objects appearances through a novel dense object query formulation. 
In addition, a novel counting loss is proposed, that directly optimizes the detection task and avoids the issues of the standard surrogate loss. 
GeCo surpasses the leading few-shot detection-based counters by $\sim$25\% in the total count MAE, achieves superior detection accuracy and sets a new solid state-of-the-art result across all low-shot counting setups. 
The code will be available on GitHub.

Main paper: 
 introduction: 
 Low-shot object counting considers estimating the number of objects of previously unobserved category in the image, given only a few annotated exemplars (few-shot) or without any supervision (zero-shot) [22] . The current state-of-the-art methods are predominantly based on density estimation [4; 14; 32; 26; 22; 31; 7; 31] . These methods predict a density map over the image and estimate the total count by summing the density.While being remarkably robust for global count estimation, density outputs lack explainability such as object location and size, which is crucial for many practical applications [33; 30] . This recently gave rise to detection-based low-shot counters [20; 19; 35] , which predict the object bounding boxes and estimate the total count as the number of detections. Nevertheless, detection-based counting falls behind the density-based methods in total count estimation, leaving a performance gap.In detection-based counters, a dominant approach to identify locations of the objects in the image involves construction of object prototypes from few (e.g., three) annotated exemplar bounding boxes and correlating them with image features [20; 35; 19] . The exemplar construction process is trained to account for potentially large diversity of object appearances in the image, often leading to overgeneralization, which achieves a high recall, but is also prone to false positive detection. Post-hoc detection verification methods have been considered [20; 35] to address the issue, but their multi-stage formulation prevents exploiting the benefits of end-to-end training.Currently, the best detection counters [20; 35] predict object locations based on the local maxima in the correlation map. During training, the map prediction is supervised by a unit Gaussian placed on GeCo, ERR 0 GeCo, ERR 0 DAVE, ERR 9 CDETR, ERR 31Figure 1 : DAVE [20] predicts object centers (red dots) biased towards blob-like structures, leading to incorrect partial detections of ants (bottom left), while GeCo(ours) addresses this with the new loss (top left). CDETR [19] fails in densely populated regions (bottom right), while GeCo addresses this with the new dense query formulation by prototype generalization (top right). Exploiting the SAM backbone, GeCo delivers segmentations as well. Exemplars are denoted in blue.each object center. However, the resulting surrogate loss is susceptible to the center annotation noise, requires nontrivial heuristic choice of the Gaussian kernel size and in practice leads to detection preference of compact blob-like structures (see Figure 1 , column 1&2). Recently, DETR [1] inspired counter was proposed to avoid this issue [19] , however, it fails in densely populated regions even though it applies a very large number of detection queries in a regular grid (see Figure 1 , column 3&4).We address the aforementioned challenges by proposing a new single-stage low-shot counter GeCo, which is implemented as an add-on network for SAM [12] backbone. A single architecture is thus trained for both few-shot and zero-shot setup, it enables counting by detection and provides segmentation masks for each of the detected objects. Our first contribution is a dense object query formulation, which applies a non-parametric model for image-wide prototype generalization (hence GeCo) in the encoder, and decodes the queries into highly dense predictions. The formulation simultaneously enables reliable detection in densely-populated regions (Figure 1 , column 3&4) and prevents prototype over-generalization, leading to an improved detection precision at a high recall.Our second contribution is a new loss function for dense detection training that avoids the ad-hoc surrogate loss with unit Gaussians, it directly optimizes the detection task, and leads to improved detection not biased towards blob-like regions (Figure 1 , column 1&2).GeCo outperforms all detection-based counters on challenging benchmarks by 24% MAE and the density-based long-standing winner [4] by 27% MAE, while delivering superior detection accuracy.The method shows substantial robustness to the number of exemplars. In one-shot scenario, GeCo outperforms the best detection method in 5% AP50, 45% MAE and by 14% in a zero-shot scenario. GeCo is the first detection-based counter that outperforms density based counters in all measures by using the number of detections as the estimator, and thus sets a milestone in low-shot detection-based counting. 
related works: 
 Traditional counting methods focus on predefined categories like vehicles [3] , cells [5] , people [15] , and polyps, [33] requiring extensive annotated training data and lacking generalization to other categories, necessitating retraining or conceptual changes. Low-shot counting methods address this limitation by estimating counts for arbitrary categories with minimal or no annotations, enabling test-time adaptation.With the proposal of the FSC147 dataset [24] low-shot counting methods emerged, which predict global counts by summing over a predicted density maps. The first method [24] proposed an adaptation of a tracking backbone for density map regression. BMNet+ [26] tackled learning representation and similarity metric, while SAFECount [32] introduced a new feature enhancement module, improving appearance generalization. CounTR [14] utilized a vision transformer for image feature extraction and a convolutional network for encoding the exemplar features. LOCA [4] argued that exemplar shape information should be considered along with the appearance, and proposed an iterative object prototype extraction module. This led to a simplified counter architecture that remains a top-performer among density-based counters.To improve explainability of the estimated counts and estimate object locations as well, detectionbased methods emerged. The first few-shot detection-based counter [19] was an extended transformerbased object detector [2] with the ability to detect objects specified by the exemplars. Current state-of-the-art DAVE [20] proposed a two-stage detect-and-verify paradigm for low-shot counting and detection, wherein the first stage it generates object proposals with a high recall, but low precision, which is improved by a subsequent verification step. PSECO [35] proposed a three-stage approach called point-segment-and-count, which employs more involved proposal generation with better detection accuracy and also applies a verification step to improve precision. Both DAVE and PSECO are multi-stage methods that train a network for the surrogate task of predicting density maps for object centers, from which the bounding boxes are predicted. Although detection-based counters offer additional applicability, they fall behind the best density-based counters in global count estimation. The proposed detection-based counter GeCo pipeline proceeds as follows (see Figure 2 ). The image is encoded by a SAM [12] backbone into f I ∈ R h×w×d , where h = H 0 /r, w = W 0 /r and d is number of feature channels. In the few-shot setup, two kinds of prototypes (appearance and shape) are extracted from each annotated object exemplar. The appearance prototypes p A ∈ R k×d are extracted by RoI-pooling [9] features f I from the exemplar bounding boxes. Following [4] , shape prototypes p S ∈ R k×d are extracted as well, by p S i = Φ([W bi , H bi ]), where W bi and H bi are the width and height of the i-th exemplar bounding box, and Φ(•) is a small MLP network. The concatenation of p A and p S yields p ∈ R 2k×d prototypes.Note, however, that in a zero-shot setup, exemplars are not provided and the task is to count the majority-class objects in the image. In this setup, a single zero-shot prototype is constructed by attending a pretrained objectness prototype p Z to the image features, i.e., p = CA(p Z , f I , f I ), where CA(a, b, c) is cross-attention [28] followed by a skip connection, with a, b and c as attention query, key and value, respectively.The prototypes p (either from few-shot or zero-shot setup) are then generalized across the image, and dense object detection queries are constructed by the Dense query encoder (DQE, Section 3.1). These are decoded into dense detections by the Dense query decoder (DQD, Section 3.2). The final detections are extracted and refined by a post-processing step (Section 3.3). The aforementioned modules are detailed in the following sections. 
dense object query encoder (dqe): 
 To account for the variation of the object appearances in the image, the current state-of-the-art [4; 20; 35] aims at constructing a small number of prototypes (e.g., three) that compactly encode the object appearance variation in the image, often leading to overgeneralization and false detections. We deviate from this paradigm by considering image-wide prototype generalization with a non-parametric model that constructs w • h location-specific prototypes P N P ∈ R w•h×d . Let P 0 = f I be the initial dense generalized prototypes (i.e., one for each location). The final dense generalized prototypes P N P are calculated by the following iterative adaptation via cross-attentionEQUATIONwhere i ∈ {1, ..., N P }. Note that spatial encoding is not applied, to enable spatially-unbiased information flow from the prototypes p to all locations.Next, dense object queries are constructed from the generalized prototypes by the following iterationsEQUATIONwhere j ∈ {1, ..., N Q }, Q 0 = P N P , and SA(•) is a self-attention followed by a skip connection to adapt the input features to the current queries. In both cross-and self-attentions, positional encoding is applied to enable location-dependent query construction. In the remainder of the paper, the dense object queries Q N Q are denoted as Q for clarityThe dense queries Q from Section 3.1 are decoded into object detections by a dense object query decoder (DQD). Note that the spatial reduction of image by the SAM backbone may lead to encoding several small objects into the same query in Q. To address this, the object queries are first spatially unpacked into high-resolution dense object queries i.e., Q HR ∈ R H×W ×d , where H = H 0 /2, W = W 0 /2 and d is the number of feature channels. The unpacking process consists of three convolutional upsampling stages, with each stage composed of a 3 × 3 convolution, a Leaky ReLU and a 2× bilinear upsampling. To facilitate unpacking of small objects, the features after the second stage are concatenated by the SAM-HQ features [11] f HQ before feeding into the final stage.Finally, the objectness score y O ∈ R H×W ×1 is calculated by a simple transform, i.e., y O = LRelu(W O • Q HR ), where W O is a learned projection matrix and LReLU(•) is a Leaky ReLU. Each query is also decoded into the object pose by a three-layer MLP, i.e., y BB = σ(MLP(Q HR )), where σ(•) is a sigmoid function and y BB ∈ R H×W ×4 are bounding box parameters in the tlrb format [27] .The final detections are extracted from y O and y BB as follows. Bounding box parameters are read out from y BB at locations of local maxima on a thresholded y O (using a 3 × 3 nonmaxima suppression, NMS). The bounding boxes are refined by feeding them as prompts into a SAM decoder [12] on the already computed backbone features f I . The boxes are refitted to the masks by min-max operation and finally non-maxima suppression with IoU = 0.5 is applied to remove duplicate detections. This process thus yields the predicted bounding boxes B P and their corresponding masks M P .GeCotraining requires supervision on the dense objectness scores y O and the bounding box parameters y BB . Ideally, a network should learn to predict points on objects that can be reliably detected by a NMS, and also from which the bounding box parameters can be reliably predicted. We thus propose a new dense object detection loss that pursues this property.Following the detection step (Section 3.2) in the forward pass, a set of local maxima {i} i=1:NDET is identified by applying a NMS on y O and keeping all maxima higher than the median response, to ensure detection redundancy. The maxima are then labelled as true positives (TP) and false positives (FP) by applying Hungarian matching [13] between their bounding box parameters {y BB i } i=1:NDET and the ground truth bounding boxes {B GT j } j=1:NGT . To account for missed detections, centers of the non-matched ground truth bounding boxes are added to the list of local maxima and labeled as false negatives (FN). The new training loss is thus defined asEQUATIONwhere gIoU(•, •) is the generalized IoU [25] , and HUN(i) is the ground truth index matched with the i-th predicted bounding box. Note that the new loss simultaneously optimizes the bounding box prediction quality, promotes locations with better box prediction capacity that can be easily detected by a NMS, and enables automatic hard-negative mining in the objectness score via FP identification. 
experiments: 
 Implementation details.Using the SAM [12] backbone, GeCo reduces the input image by a factor r = 16, and projects the features into d = 256 channels (Section 3). In DQE (Section 3.1), N P = 3 iterations are applied in prototype generalization (1) and N Q = 2 iterations in dense object query construction (2) . Following the established test-time practice [20; 19; 26] , the input image is scaled to fit W 0 = H 0 = 1536 if the average of the exemplars widths and heights is below 25 pixels, otherwise it is downscaled to fit the average of the exemplar width and height to 80 pixels and zero-padded to W 0 = H 0 = 1024.As in [20] , the zero-shot GeCo is run twice, first to estimate the objects size and then again on the resized image.Training details. With the SAM backbone frozen, GeCo is pretrained with the classical loss [20] for initialization and is then trained for 200 epochs with the proposed dense detection loss (3) using a mini-batch size of 8, AdamW [16] optimizer, with initial learning rate set to 10 -4 , and weight decay of 10 -4 . The training is done on 2 A100s GPUs with standard scale augmentation [20; 4] and zero-padding images to 1024×1024 resolution. For the zero-shot setup, the few-shot GeCo is frozen and only the zero-shot prototype extension is trained for 10 epochs. Thus the same trained network is used in all low-shot setups.Evaluation metrics and datasets. Standard datasets are used. The FSCD147 [19] is a detectionoriented extension of the FSC147 [24] , which contains 6135 images of 147 object classes, split into 3659 training, 1286 validation, and 1190 test images. The splits are disjoint such that target object categories in test set are not observed in training. The objects are manually annotated by bounding boxes in the test set [19] , while in the train set, the bounding boxes are obtained from point estimates by SAM [35] . For each image, three exemplars are provided. The second dataset is FSCD-LVIS [19] , derived from LVIS [8] and contains 377 categories. Specifically, the unseen-split is used (3959 training and 2242 test images), which ensures that test-time object categories are not observed during training.The standard evaluation protocol [24; 26; 32] with Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) is followed to evaluate the counting accuracy. Following [19] , Average Precision (AP) and Average Precision at IoU=50 (AP50) is used on the same output to evaluate the detection accuracy.Few-shot counting and detection. GeCo is compared with state-of-the-art density-based counters (which only estimate the total count) LOCA [4] , CounTR [14] , SAFECount [32] , BMNet+ [26] , VCN [22] , CFOCNet [31] , MAML [7] , FamNet [24] and CFOCNet [31] , and with detection-based counters C-DETR [19] , SAM-C [18] , PSECO [35] , and DAVE [20] , which also provide object locations by bounding boxes. Results are summarized in Table 1 .GeCo outperforms both recent state-of-the-art detection-based counters DAVE [20] and PSECO [35] by a 24% and 39% MAE, and a remarkable 27% and 51% RMSE on the test split, setting a new stateof-the-art in detection-based counting. Notably, GeCo outperforms all single-stage density-based counters (top part of Table 1 ) by a large margin, which makes it the first detection-based counter that outperforms the longstanding total count estimation winner LOCA [4] by a remarkable 27% MAE and 4% RMSE on test split. In this respect, GeCo closes the performance gap that has been present for several years between state-of-the-art density-, and detection-based counters. In terms of detection performance, GeCo surpasses all state-of-the-art methods, including PSECO [35] which uses both, SAM [12] and CLIP [21] backbones, by 1% AP, and 2% AP50. Note that GeCo also outperforms PSECO in count prediction by a large margin (∼40%), which is crucial, as an ideal detection counter should deliver both accurate total count prediction as well as feature good object localization. In addition, GeCo also outperforms SAM-C, which is a low-shot counting and detection extension of SAM by 70%/55% MAE/AP. To demonstrate the impact of the refinement step in existing methods, we modified DAVE [20] by feeding predicted bounding boxes to SAM [12] as prompts, which results in a GeCo-like box refinement. Compared to modified DAVE, GeCo achieves 21% and 16% higher AP and AP50, respectively, indicating that the reason for the excellent performance of GeCo lies in its architecture, rather than in segmentation-based refinement.Figure 3 visualizes detections for qualitative analysis 1 . GeCo predicts bounding boxes of superior quality for elongated objects (row 1), validating the selection of bounding box prediction locations. On detecting complex, non-blob-like objects (row 2), GeCo outperforms concurrent methods, by more accurately generalizing the prototypes. In densely populated scenes (row 3), GeCo achieves higher accuracy in both count and bounding box predictions. In comparison with state-of-the-art, GeCo features better object discrimination (row 4), which can be attributed to better prototype generalization in DQE (Section 3.1) and hard negative mining in the new loss from Section 3.4.We further evaluate GeCo on FSCD-LVIS [19] . Results in Table 2 show that GeCo outperforms the best method by significant 178% and 73% in AP and AP50, respectively, and performs on-par in terms of MAE. The experiment supports the results on FSCD147. One-shot counting and detection. In the one-shot counting setup, a single exemplar is considered. Table 3 shows comparison with the recent density-and detection-based methods. GeCo outperforms all state-of-the-art single-stage density-based counters, outperforming LOCA 1-shot [4] version specifically trained for the one-shot setup, by a significant margin of 35% MAE and 20% RMSE on validation and test split, respectively. GeCo also outperforms state-of-the-art method PSECO [35] byEQUATION(2) 4% AP and 5% AP50, and by significant 45% MAE and 49% RMSE on test split. These results show that GeCo features remarkable robustness to the number of exemplars since a single network (without re-training or fine-tuning) is used in both three-and one-shot setups. In particular, the performance drops by only 2%/11% of MAE/RMSE and 1%/1% AP/AP50 on the test split between both setups. In a one-shot setting, GeCo surpasses state-of-the-art three-shot models. Specifically, one-shot GeCo achieves 22% and 20% lower MAE and RMSE, respectively, compared to three-shot DAVE, and outperforms three-shot PSECO by 38% and 46% on the FSCD147 test set. These results highlight the robustness of GeCo to the number of exemplars, demonstrating its ability to handle inputs with lowered visual diversity.EQUATIONZero-shot counting and detection. Table 4 reports the results of the zero-shot GeCo compared with best zero-shot variants of the density-based counters, LOCA [4] , CounTR [14] , RepRPN-C [23] , RCC [10] and with the zero-shot variant of the best detection-based counter DAVE [20] . GeCo outperforms DAVE [20] by a significant margin of 14% MAE and 6% RMSE on the test set. Furthermore, it outperforms all density-based methods and sets a new state-of-the-art result on FSC [24] benchmark, by outperforming the top-performer CounTR [14] by impressive 6% MAE on the test set. Since the zero-shot variant of the recent detection-based counter PSECO [35] does not exist, we include its prompt-based variant for complete evaluation (i.e., target object class is specified by a text prompt). Even in this setup, the zero-shot GeCo outperforms the prompt-based PSECO by 20% MAE 16% RMSE, and 2% AP50 demonstrating great robustness to different counting and detection scenarios.Mutliclass images. To further verify the robustness of the proposed method, we validate it on a subset of FSCD147, that contain images with multiple object classes (FSCD147 mul ) [20] . Results in Table 5 indicate that most state-of-the-art methods non-discriminatively count all objects in an image due to prototype over-generalization. GeCo outperforms all single-stage density-, and detectionbased counters on multiclass images by at least 60%/67% in MAE/RMSE. This further verifies the robustness of the proposed architecture, which benefits from the hard-negative mining in the proposed loss function, leads to more discriminative prototype construction and false positive reduction.Dense object detection loss. To analyze the contribution of the new dense detection loss from Section 3.4, we trained GeCo using the standard loss [20; 35] that forms the ground truth objectness score by placing unit Gaussians on object centers -this variant is denoted by GeCo Gauss . Table 6 shows that this leads to a substantial drop in total count estimation (38% RMSE, and 34% MAE) as well as in object detection (6% AP, and 3% AP50). Qualitative results are provided in Figure 4 . As observed in columns 3 and 5, the classical unit-Gaussian-based loss [20; 35] forces the network to predict object locations from the object centers, which are not necessarily optimal for bounding box prediction. In contrast, the proposed dense detection loss enables the network to learn optimal point prediction, which more accurately aggregates information of the object pose. Columns 1 and 2 indicate that the new loss leads to superior detection of objects composed of blob-like structures avoiding false detections on individual object parts. Furthermore, the hard-negative mining integrated in the new loss design leads to better discriminative power of the detections and subsequent reduction of false positives (column 4). Architecture. To evaluate the impact of concatenating the SAM-HQ [11] features in the query unpacking process in the DQD module (Section 3.2), we remove these features in GeCo HQ . Table 6 shows a counting performance drop 5% MAE and 10% RMSE. To validate the importance of modeling exemplar shapes, i.e., width and height, with prototypes p S , we omit them in GeCo p S . We observe a substantial performance decrease of 5% MAE, and 9% RMSE. Finally, we remove bounding box refinement in the detection refinement module (Section 3.3), and denote the variant as GeCo Ref . While this does not affect the global count estimation accuracy, we observe a 26% decrease in AP and 2% decrease in AP50. It is worth noting, that bounding box refinement improves the accuracy of predicted bounding boxes, however it does not enhance object presence detection.GeCo GeCo GaussTo verify the importance of the DQE module (Section 3.1), we replace the dense object queries Q construction step (2) with a standard self-attention, i.e., Q = SA(P) 3× . This leads to a 8% MAE and 5% RMSE performance drop, verifying the proposed approach. To evaluate the importance of using image features as queries in (2), we change the object query construction toQ j = CA(SA(Q j-1 ), f I , f I )to follow a standard DETR [1] -like approach, and denote it as GeCo DETR . We observe a 20% MAE and 22% RMSE decrease in counting performance. 
conclusion: 
 We proposed GeCo, a novel single-stage low-shot counter that integrates accurate detection, segmentation, and count prediction within a unified architecture, and covers all low-shot scenarios with a single trained model. GeCo features remarkables dense object query formulation, and prototype generalization across the image, rather than just into a few prototypes. It employs a novel loss function specifically designed for detection tasks, avoiding the biases of traditional Gaussian-based losses. The loss optimizes detection accuracy directly, leading to more precise detection and counting.The main limitation of the presented method is that it cannot process arbitrarily large images, due to memory constraints, since it, as all current methods, operates globally. In future work, we will explore local counting, incremental image-wide count aggregation, optimizing inference speed utilizing a faster backbone [34] .Extensive analysis showcases that GeCo surpasses the best detection-based counters by approximately 25% in total count MAE, achieving state-of-the-art performance in a few-shot counting setup and demonstrating superior detection capabilities. GeCo showcases remarkable robustness to the number of provided exemplars, and sets a new state-of-the-art in one-shot as well as zero-shot counting.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Upon acceptance, we will make the code and models publically available. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Our work will not make any societal impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our data and models are not a risk of misuse. 
 
 Appendix: This supplementary material provides additional comparisons of GeCo with state-of-the-art under a non-standard experiment, and provides additional qualitative examples.Performance analysis on a non-standard experiment. The analysis of the detection methods in Section 4 adheres to the standard evaluation protocol [19; 20] , where a method predicts a set of bounding boxes for each image. The estimated count is the total number of predicted bounding boxes, and evaluated by the MAE/RMSE measures, while the detection accuracy is evaluated by AP/AP50 measures. Both measures are computed on the same set of output bounding boxes.However, in the PSECO [35] paper, the reported evaluation deviated from the standard one in an important detail. Namely different outputs were evaluated under MAE/RMSE and AP/AP50 to fully evaluate the different properties of the method. AP/AP50 was computed in all output bounding boxes, while the MAE/RMSE were computed on a subset of the boxes, obtained by thresholding the response score. In Section 4, we evaluated all methods, including PSECO under the standard experiment. Nevertheless, we additionally report GeCo evaluated under the said non-standard PSECO experiment in Table 7 .Even in this setup, GeCo outperforms PSECO by 4%/4% AP/AP50, and 1%/2% AP/AP50, on validation and test set, respectively, again with a substantially lower global count errors (∼50% MSE/RMSE reduction). These results shed an important insight. A method producing false positives, which increase the count errors and reduce its usefulness for counting, might achieve good detectionoriented performance measures. Thus for counting performance evaluation, the MAE/RMSE should be considered primary measures, while AP/AP50 should be secondary, as they are less strict towards false positive detections. Performance in crowded scenes. To evaluate counting performance in crowded scenes, we constructed a subset of the FSCD147 test set by including images with at least 200 objects and a maximal average exemplar size of 30 pixels. Notably, the new subset contains 42 images, averaging 500 objects per image, thus featuring dense scenes with small objects. Three top-performing methods from Table 1 were included in the comparison and are shown in Table 8 . GeCo outperforms both PSECO and DAVE by a significant margin, e.g., outperforming DAVE by 23% in MAE and 36% in RMSE, which demonstrates superior counting performance on small, densely populated objects. Qualitative results. Figure 5 compares GeCo with PSECO [35] , which achieves the best AP/AP50 measures among the related counters. GeCo shows robust performance, achieving high precision (see Figure 5 block 1), while achieving high recall (see Figure 5 block 2). This is challenging for related methods, particularly in densely populated scenes or with small objects. Furthermore, GeCo outperforms PSECO on elongated or more complex objects (see Figure 5 block 3), better exploiting the exemplars. Figure 6 visualizes the segmentations produced by GeCo, in a few-shot setup, of various objects in diverse scenes. GeCo is robust to noise, achieves discriminative segmentations, and performs well on elongated, non-blob-like objects and in dense scenarios. Figure 7 compares GeCo with all state-of-the-art detection counters [20; 19; 35] . GeCo achieves superior counting performance, and predicts more accurate bounding boxes. • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.See supplementary material for more visualizations 


Decision: This paper is about "low-shot counting" of objects. After discussion between reviewers and authors, the paper received two ratings of  "borderline accept" and one of "weak accept". The AC recommends that the paper be accepted.

Reviews: review: paper_summary: paper_summary: This paper address the issue for low-shot and zero-shot object counting, with an object detection-based approach. The proposed method heavily uses SAM framework, to provide feature embeddings and refine detection boxes. Attention-based feature aggregation and SAM-HQ are used to get the final features for objectiveness and bounding boxes. Evaluations are carried out on standard datasets with few/one/zero-shot setups, and the proposed GeCo outperforms many state-of-the-art methods.
strength_weakness: strength_weakness: * One concern is that the paper utilizes SAM as its backbone, which is trained on SA-1B dataset. This may leads to some benefits to GeCo by having seeing large amount of diverse data. More discussion is needed to address this.
* Also related to SAM, in table 6, it shows that without detection refinement module, the $GeCo_{\overline{Ref}}$'s performance drops heavily and goes lower than other methods in table 1. I am wondering what if we also feed other detection-based methods' output into SAM decoder. This may be a more fair comparison.
questions: questions: Does the use of SAM in multiple places bring advantages over other baselines?
limitations: limitations: The limitations are sufficiently discussed in the paper.
paper_summary: paper_summary: Paper tackles the task of few-shot and zero-shot class agnostic counting, and presents Geco, a unified counting framework that can detect, segment and count objects. Geco uses SAM backbone for feature extraction, and implements Dense Query Encoder (DQE) and Dense Query Decoder (DQD) to detect prototypes. For few shot setup, prototypes are obtained from the exemplars, and zero shot setup uses learned objectness prototypes. DQD outputs a set of dense detections, which are further refined by the SAM decoder, by considering DQD detections as prompts for the SAM decoder.
strength_weakness: strength_weakness: 1. Lines 110 - 114: authors claim that existing low shot counting approaches uses a small number of prototypes or exemplars, often leading to false detections. Authors say their approach is different from these existing approaches. However, eqn 1 in Sec 3.1 appears to be conceptually similar to the existing approaches, i.e. few prototypes p are used to create keys and values in a cross attention layer, where the query is formed by the image features.
2. Looking at the ablation study in table 6, it appears that removing SAM decoder degrades the performance of Geco significantly in terms of AP, which raises some questions about the effectiveness of the approach. How does Geco perform without SAM decoder on the test set of FSCD147 ?
3. Related to my point 2 above and the ablation study in table 2: it would be useful to have a baseline without using Geco, by simply using the exemplars as prompts for the SAM decoder.  This baseline does not require any new training.
questions: questions: I would request the authors to kindly address the points I raised in Weaknesses section.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces GeCo, a novel unified architecture for low-shot counting that integrates object detection and segmentation. GeCo addresses the limitations of current state-of-the-art methods by generalizing object prototypes across diverse appearances and introducing a new counting loss that directly optimizes the detection task. The architecture achieves significant improvements in detection accuracy and count estimation over previous methods and sets a new benchmark in low-shot counting.
strength_weakness: strength_weakness: 1. While SAM's pre-training provides strong generalization with other potential backbone networks, it is unclear if similar results could be achieved with other networks like ResNet or EfficientNet. Thus the complexity and computational requirements of the proposed GeCo architecture might limit its scalability and real-time applicability, especially in resource-constrained environments.
2. Although the method is claimed to be low-shot friendly, the paper does not provide explicit architectural components or techniques specifically designed for low-shot scenarios.
3. The performance of GeCo on highly heterogeneous datasets, where object appearances vary significantly within the same category, is not adequately demonstrated.
4. The effectiveness of this method in scenarios involving small objects or high-density areas, such as crowd density estimation or face detection in dense crowds, has not been well explored as this type of dataset has large-scale variations.
questions: questions: 1. Why was the SAM framework specifically chosen over other backbone networks? How does it compare in terms of generalization and performance with other pre-trained models like ResNet or EfficientNet?
2. What specific design elements make GeCo particularly suitable for low-shot learning? Are there any modifications or components that specifically address the challenges of low-shot scenarios?
3. How does GeCo perform on datasets with significant intra-class variability? Can it generalize well to objects with a high degree of appearance variation within the same category?
limitations: limitations: 1. The method's performance still heavily depends on the quality and representativeness of the provided exemplars. Poor quality or unrepresentative exemplars could negatively impact the detection and counting accuracy.
2. While the dense object query formulation improves generalization, the method's ability to handle extreme variability in object appearances within the same category needs further validation.
3. The effectiveness of GeCo in scenarios involving small objects or highly dense regions (e.g., crowd counting, face detection in crowds) is not thoroughly evaluated. These scenarios often present unique challenges that may require additional considerations.

Comments: title: Reply
comment: Thank you. My recommendation is 6-Weak Accept
We thank the reviewer for the positive comment and we are happy to clarify the remaining point about the SAM backbone and comparison fairness. Note that recent state-of-the-art (PSECO) also uses SAM as a backbone, but GeCo outperforms it substantially, thus comparison seems fair. To further clarify, we replaced SAM backbone in GeCo with the classical ResNet. As reviewer anticipated, the performance drops a bit, but remains high, i.e., MAE (7.91-> 9.01) RMSE (54.28 -> 48.01), and this GeCo version still outperforms all state-of-the-art, including PSECO that uses the SAM backbone. We conclude that the SAM backbone is not the main contributor to good counting performance of GeCo.
thanks for the response, which addresses my concerns. I've changed the rating to Borderline accept.
Thank you for the author's response, which solved most of my problems. 

SAM may give GeCo some benefit by looking at a lot of different data, which isn't fair to other comparison methods. The author said that any backbone network can be used instead of SAM in the image encoding stage, but it is not clear whether the performance will be significantly reduced because of the different backbones.

Therefore I maintain my original rating:  6 Weak Accept