Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization
Authors: Michael Green
Matan Levy
Issar Tzachor
Dvir Samuel
Nir Darshan
Rami Ben-Ari
Venue: NeurIPS 2025 poster ()
Keywords: Image Retrieval
Instance Retrieval
Personalization
Feature Aggregation
Feature Optimization
Primary area: applications
TL;DR: We suggest a novel method and new benchmark for image retrieval with small objects

Abstract: We address the challenge of Small Object Image Retrieval (SoIR), where the goal is to retrieve images containing a specific small object, in a cluttered scene. The key challenge in this setting is constructing a single image descriptor, for scalable and efficient search, that effectively represents all objects in the image. In this paper, we first analyze the limitations of existing methods on this challenging task and then introduce new benchmarks to support SoIR evaluation. Next, we introduce \ours (\oursMI), a novel retrieval framework  which incorporates a dedicated multi-object pre-training phase. This is followed by a refinement process that leverages attention-based feature extraction with object masks, integrating them into a single unified image descriptor. Our \oursMI approach significantly outperforms existing retrieval methods and strong baselines, achieving notable improvements in both zero-shot and lightweight multi-object fine-tuning. We hope this work will pave the way and inspire further research to enhance retrieval performance for this highly practical task.

Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization
Michael Green !+ Matan Levy Issar Tzachor Dvir Samuel 1.3, Nir Darshan', ' Rami Ben-Ari
!OriginAl, Israel
The Hebrew University of Jerusalem, Israel Bar-Ilan University, Israel
Abstract
We address the challenge of Small Object Image Retrieval (SoIR), where the goal is retrieve images containing specific small object, in cluttered scene. The key challenge in this setting is constructing a single image descriptor, for scalable and efficient search, that effectively represents all objects the image. this paper, we first analyze the limitations of existing methods on this challenging task and then introduce new benchmarks to support SoIR evaluation. Next, we introduce Multi-object Attention Optimization (MaO), novel retrieval framework which incorporates dedicated multi-object pre-training phase. This is followed by refinement process that leverages attention- feature extraction with object masks, integrating them into a single unified image descriptor. Our MaO approach significantly outperforms existing retrieval methods and strong baselines, achieving notable improvements in both zero-shot and lightweight multi-object fine-tuning. We hope this work will pave the way and inspire further research to enhance retrieval performance for this highly practical task. Project page: https://pihash2kk. http://phhash2kggitubb.iee
Top-1 Result (for query A)
Query A
Retrieve
MaO
Target
(for query
Query B
MaO
Targer
here
Figure It is highly challenging to find an object instance that appears small in cluttered scene image, moreover when shuffled in a large corpus of images. Top-I retrieval results by our Multi-Object Attention Optimization (MaO) method are shown, attained with single representation per-image, allowing scalable search.
Introduction
Searching for small object within large image corpus is akin to finding needle in a haystack. The challenge arises from the object's small size, the existence of many distracting objects in the
'Equal Contribution
Corresponding author: <ramib@ orinibiiiiiiiii.cc>>>>
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

scene, and moreover, the vast number of images in the corpus. In this paper, we address the problem of Small Object Image Retrieval (SoIR), where the goal is to search in a large image corpus and identify the images containing specific (or personalized) instance of a likely small object, in cluttered scene. In other words, the task is not just about locating any needle in the haystack but about finding your specific needle. In Image Retrieval (IR), a single global descriptor offers a strong advantage in large databases, due to computational and memory benefits. They provide a fast way to perform retrieval and allow additional speed-up with approximate nearest neighbor search techniques [36, 29, 14, 37, 46, 18].
SoIR can be seen as a specialized case of Instance- mastance-Based Image Retrieval (IBIR), where the objective to retrieve images from a large corpus that contains the exact instance of a query object or scene [37, 50, 34, 35, 28J. Recent state-of-the been evaluated on datasets such as Parrss6,, ROxford5K [27], Products- 1OK [3], and GL.vv2 [48], which predominantly feature large, centrally positioned objects. However, retrieving images containing small objects in cluttered scenes remains largely unexplored. Recent findings [35] indicate that existing IBIR methods struggle in scenarios where multiple objects are present, particularly when datasets include similar objects from the same category. The INSTRE [47] dataset has been introduced as an additional benchmark for the IBIR task [11, 20, 26, 6]. While benchmarks such as RParis6K and ROxford5K contain objects occupying approximately 40% of the image area on average, INSTRE features smaller objects, covering 20 25% of the image, making retrieval more challenging. However, INSTRE images typically contain only 1-2 objects per image, limiting the complexity of multi-object interference. Recently, Li et al. [19] introduced the VoxDet dataset, originally designed for object detection. We find it particularly suitable for SoIR due to the small object sizes and the presence of many objects within each scene. Figure 1 illustrates these challenges with example images and retrieval results. For better evaluations of SolR methods, we introduce three benchmarks based on VoxDet, INSTRE and PerMiR [35], highlighting the significant performance drop of existing IBIR methods in SolR. Additionally, we propose novel retrieval approach tailored to this task.
(SoTaa) IBIR models [37, 20, 2, 36, 11] have primarily
We define small and cluttered objects in the following contexts: (a) objects occupying a small fraction of the image area (e.g., 0.5- 1)), (b) cluttered scenes with multiple objects (often 10+), including same-category instances. The key challenges in retrieving small objects are: (1) capturing the object instance characteristics in the image representation, even when the object is small (2) ensuring that all objects in a scene are well-represented in a single image descriptor.
The representation of small objects in images has recently garnered notable interest in different contexts. Recent research in multi-modal large language models (MLLMs) [51], suggests that the primary challenge in object representation within images is ensuring the quality of its representation. Studies in object-level multi-modal learning [31] has explored these challenges within tasks such as text-to-inn text-to-image semantic search [16, 17, 6] and instance localization [7]. Additionally, Samuel et al. [35] demonstrated that the presence of multiple objects within an image can cause significant confusion and performance degradation in multi-object scenarios, while Abbasi al. [1] highlighted a strong encoder bias toward larger objects, further complicating small object retrieval.
To address these challenges, we propose a novel SolR method, as illustrated in Figure 2. Our method first decomposes the image into individual objects using an open- vocabulary detector (OVD), encodes them separately, and then integrates their representations into a single global descriptor. By isolating and encoding each object separately, our approach ensures balanced representation for all objects, regardless of size, while filtering out irrelevant background elements (e.g. sky, walls). Additionally, we refine the image representation through post-training attention optimization process, aligning the image descriptor with object attention maps to improve object representation and eventually, retrieval accuracy. We refer to this method as Multi-object Attention Optimization (MaO).
We extensively evaluate our approach through comprehensive experiments, demonstrating its superior performance against existing methods and strong baselines. The results highlight our method's 's ability to effectively retrieve images containing small instances of a target object embedded in cluttered images. Sample results are shown in Fig. I.
We summarize our contributions as follows:
2

---

Stage A: Multi Object Fine -ueeelunnngg
Target Image
Query
Vision Enc
CE
U1
Vision Encoder
Boundray Boxes
U
24
Vision Enc.
Stage B: Multi-Object Attention Optimization
DICE
Forward pass
Vision
Detect & Mask
Explainability
Enc..t
LeGrad pass
Maps
Object Cover: 0.28% of the image
Matching
Vision
Enc
Global Represeenation
Trained
Frozen
DICE
Figure 2: Overview of our MaO encoding method. Stage A (Top): Contrastive fine-tuning where for each training image, objects are cropped and encoded using visual backbone. The objective is to align the average object-wise representation with a single object query. Stage B (Bottom): The global image representation is optimized by simultaneously aligning the explainability maps of object crops with their corresponding object masks, leveraging the pre-trained visual backbone from Stage A. Note that we optimize single vector with respect to-K different image crops (K objects), while backpropagating gradients only through single (unified) vector. Our global image descriptor, derived from MaO, enables efficient search for the occurrence of a specific small object, during inference.
We study and analyze the problem of Small Object Image Retrieval (SoIR), identifying key challenges and limitations in existing methods. Our study focuses on three critical factors: object size, clutter (scenes with many objects), and resolution.
We introduce new benchmarks specifically designed for SoIR, enabling more rigorous evaluation and fostering future research in this domain.
We propose MaO, novel method that leverages explainability maps to refine object repre- sentations and unify them into single image descriptor. Our approach outperforms existing methods and strong baselines.
2
Related Work
Below, review key approaches relevant to work.
Instance- Image Retrieval: CNN-based methods have played fundamental role in IBIR for many years. Region-based techniques [44, 32, 6] explore various region-matching strategies to compute image similarity. For instance, GeM [29] enhances retrieval performance by fine-tuning CNN network, introducing the Generalized-Mel MMan (GeM) pooling layer. Despite their effectiveness, these methods are computationally expensive and rely on predefined, grid-based regions, posing two major drawbacks: (1) the predefined grid is content-agnostic, making it unlikely to align with object boundaries, and (2) many grid regions capture irrelevant background information, creating clutter in the image representation. Moreover, these approaches tend to emphasize larger objects while failing to adequately represent smaller ones. In contrast, we replace rigid grid sampling with an open-vocables vocabulary object detector (OVD) that detects arbitrary objects, allowing for a more flexible and content-aware representation. Each object representation is further refined through attention based encoding, allowing the integration of multiple objects into single image descriptor.
Local feature based methods: different approach suggests represent an image with set of local features while using a similarity function to compare two local descriptor sets [14, 38, 15, 42]. Due to high computational cost, these methods are frequently used for small scale datasets or top-K re-ranking and are not suitable for large scale image retrieval. Some methods attempt to make local feature based methods more efficient e.g. [14, 37, 38). Other methods such as GSS [20] and HP [2] leverage graph networks to perform spatial verification. However, these approaches require storing

---

multiple local features for each gallery image or performing additional per-query computations, beyond simple vector similarity, leading to increased memory consumption and computational overhead. In contrast, our approach produces a single compact representation per image, ensuring both memory efficiency and fast search capability across large datasets. SuperGioball [37] similarly focuses on memory-efficient retrieval with re-ranking stage based solely on feature refinement. However, its performance drops significantly when encountering images with small objects.
While CNNs have traditionally been the dominant approach for IBIR, PDM [35] was among the first to leverage foundation models for this task. Despite its strong performance in cluttered scenes, PDM relies on large feature maps and a prompt for pairwise comparisons, making it less suitable for only an image based query and scalable global search.
Previous works have explored generating a global representation by aggre- gating region-wise descriptors. Gordo al. [9] employs region proposal network to learn which regions should be pooled to form the global descriptor, while Sun et al. [39] uses a generic object proposal identify regions and aggregating their SIFT [21] features into compact representation. Recent work [43] combines a custom domain-specific detector with a CNN backbone and a match- kernel-based aggregation [45]. In contrast these methods, which depend on custom detectors or keypoint-based features, our approach fine-tunes a visual encoder for the multi-object representation task. We further refine opject-specific representations by leveraging object masks, while employing an effective aggregation strategy to enhance global representation.
Detection-based IBIR:
Small Objects: While IBIR methods have primarily focused benchmarks where images contain a single, prominent object, few studies have specifically addressed the retrieval of small objects [20, 11, 6], mainly evaluating performance
the INSTRE dataset [47]. To better assess small object retrieval, we introduce two new challenging subsets from INSTRE, named INSTRE-XS and INSTRE-XXS. Additionally, we establish a new benchmark using the Vox dataset [19], dubbed VoxDet-SolR, which features small objects in cluttered scenes, a scenario that has not been explored in prior instance-based retrieval research.
Object-Focused Representation: a-CLIP [40] introduced a CLIP variant with an additional channel for object masks, enabling focuse- representations. As we demonstrate in this study, a-CLIP provides a more effective solution for handling small objects compared to CLIP and some previous methods. In this work, we show how existing foundation models like CLIP and DINOv2 can be leveraged for encoding an image containing small objects. We effectively utilize object masks to improve the representation of small objects and enhance retrieval of images with the target instances, even in cluttered scenes.
Recently, Bousselham et al. [5] introduced MaskInversion, a method used for referring expression classification and localized captioning. It generates an fbject-focused representation, by optimizing propose an image token to align with the image's 's explainability map. Inspired by this approach, a novel feature refinement stage in our framework. However, instead of operating on a single image, our method integrates multi-object attention from multiple image crops into a unified global representation. This refinement process is guided by our image decomposition and multi-object fine-tuning strategy, ensuring more comprehensive and scalable retrieval solution.
3 Method
Our task is to retrieve all images from a large gallery that contain the object depicted in a given query image. In this section, we present Multi-object Attention Optimization (MaO), a method that encodes both query and gallery images while tackling three key challenges: (1) the small size of target objects within images (2) clutter: the presence of multiple distracting elements in gallery images, existing in the foreground and background (3) Deriving a single image representation for each gallery image. Instead of encoding entire images as a whole, our approach first isolates individual objects, encodes them separately, and then refines their encoding by finding an optimized representation for all objects in the scene with a single descriptor. This structured encoding reduces the impact of irrelevant regions (e.g. sky, ceiling, or wall) while preserving essential object-level details. Our method consists of two stages: (A) Multi-Object fine-tuning of a visual backbone to learn representation for multiple objects within an image, and (B) post-training refinement, where the initial object representations are jointly refined and unified through representation optimization, leveraging object masks and attention-bel based cues. A schematic overview of our method is presented in Figure 2.

---

Feature Extraction: First, we extract and filter relevant image information for the task, focusing on individual objects. Let us consider an RGB image X E 23*W*x Using given annotations or an open- veee-oocabllary object detector (OVD), we divide the image into k crops, z; S X, for = 1, ....., k, each centered on a single object. To this end, we leverage the OVD for object-proposal (agnostic to the object category). Each object is then encoded using a visual backbone, producing k feature vectors: (VI,..., J E gd. This process filters irrelevant regions, improving feature focus for the SoIR task.
Multiple Object Fine Tuning (Stage A): Our objective is to preliminary fine-tune our backbone on multi-object and instance based scenario. To achieve this, we align each query image with a representation computed from its corresponding ground truth gallery image. For query image (an object-focused image), we extract its feature vector 2 using the visual encoder backbone. For the corresponding gallery image, we fuse multiple object representations by average pooling the k object-level features, yielding te (see Figure 2-Stage A). We then compute the cosine similarity between 2e and l/g. and train the model using the InfoNCE contrastive loss [24] over the batch. This process is schematically described in Figure 2-Stage A. The Multi-Object Attention Optimization stage is applied post-training, producing the final refined representation & as described below.
Multi-Object Attention Optimization (Stage B): for explainability in ViT that relies solely on attention map gradients. We leverage LeGrad [4] to optimize single token (representation) based on the image's 's attention map. However, instead of applying single binary mask single image, we refine the token by aligning with different images. Each crop's 's explainability map is optimized to align with its corresponding mask, ensuring that the final token effectively captures the object within each crop. Essentially, we optimize a single vector with respect to-k different image crops (I objects), while backpropagating gradients only through a single (unified) vector (as illustrated in 2-gae-Stage B). This process ensures that the final vector effectively represents all objects in the image, while maintaining compact yet expressive feature encoding. The attention -based representation suggests an advantage, as it effectively encodes small objects even when they are not perfectly centered in the bounding box. For the query image, the same optimization is applied, while considering a single object.
LeGrad [4], introduced a gradient-based method
For each crop, we assume the availability of an object mask, m, E R RW typically obtained using SAM [12]. Leveraging our trained vision encoder (from stage A), we first encode the image crops r; to obtain et. Using LeGrad [4], we compute an explainability (heat) map, Elr,- v2) E R W; W.xHH,
indicating the attention map associated with a given representation vector 2; To attain a single representation, we then optimize single token i, for an explainability map of all objects in the image (within individual crops). The objective of this optimized token is defined as:
iC argmax Ve
Σ IoU (E(v, u;), m,) + a Σ U/
This optimization aims to adjust the token's 's explainability map to best align with all object masks. Here, IoU denotes the Intersection over Union measure between the explainability map and object mask, while represents the normalized dot product (i.e., cosine similarity). For preliminary details on LeGrad and computing of E please refer to Appendix A. During optimization we initialize i
with the Ue obtained from stage A, (.50) Ve- Following [5]. we also incorporate a regularization term controlled by a to ensure that the optimized token remains close to its initial embedding, Ve- Note that the new representation obtained from Stage B, is based on all object embeddings derived from our stage A trained encoder. The optimization process is performed using standard gradient descent, resulting in a refined representation vector for each gallery image, denoted as 0z. Image retrieval is then conducted via conventional, vectorized similarity search in the gallery.
Our experiments show that the proposed object-based aggregation model, which extends Mask- Inversion to handle multiple objects, provides strong benefits for small-object retrieval, a challenge that is often underexplored by existing methods. An additional advantage arises from applying Stage-B refinement to the customized multi-object encoder trained in Stage-A, resulting in two tightly integrated and complementary stages.
4 Evaluation
In this section, we conduct extensive experiments across multiple datasets to assess our method's performance on the Small Object Retrieval (SoIR) task. We compare approach against several
5

---

methods, demonstrating that: (1) of-e-----he-rrr IR methods suffer a significant performance drop in the SolR task, and (2) our proposed method, MaO, substantially improves retrieval performance, particularly for small objects in the gallery and cluttered scenes. As baselines, we first evaluate conventional IR methods that encode the entire image into a single global representation using foundation visual encoders such as CLIP [30]. DINOv2 [25]. and a-CLIP [40]. Additionally, we compare against previous IBIR methods, including GSS [20]. GeM [29], CVNet [14] and SoTA SuperGlobal (SG) [37] and AMES [38], using each with its corresponding global and re-ranking component, if exists, and as prescribed, egg. in SG (with CVNet) and AMES (with DINOv2 and SG pre-compute [38]). We also fine-tune DINOv2, CLIP, and GeM on VoxDet and assess their performance on all of our benchmarks. We evaluate MaO using the widely adopted mean average precision (mAP) metric. For all models we use the public pre-trained weights and keep the original setting.
Implementation Details: We use the Adam W optimizer, initializing the learning rate at 5 x 10-5 with an exponential decay rate of 0.93 down to 1 x 10-6, We fine-tune all the transformer-basel models on the VoxDet training set using LoRA [10] adapter of rank 256. We train with batch size of 128 for epoch, across four NVIDIA-A 100 nodes. For inference, we apply OWLv2 [22] as OVD, applied in object-proposal mode, to detect any object in gallery images, considering bounding boxes with a confidence threshold above 0.2. For detected objects, we enforce a minimum bounding box size equal to the backbone's 's input dimensions by centering cropped region around the object. The refinement process is done for 80 iterations with a = 0.03 and a learning rate of 1 x 10 which takes 0.03 seconds of computation per object. For gallery images, this is performed offline. Our global feature dimension is the same as the backhone, 512D for CLIP and 768D for DINOv2.
4.1 Datasets
Creating instance-based datasets is particularly challenging. requires the same object instance to appear across multiple images in varying positions, backgrounds, and configurations. Most commonly used IBIR datasets focus on large, prominent objects, such RParisóK, ROxford5K [27], and Google Landmark Dataset (GLDv2) [48], or object-centric datasets like Products- -00 [3] where objects are typically large and centered in the image (see Table 1). We define Object Size as the ratio of the object-mask area to the total image area. Table I presents fundamental characteristics of several datasets used for IBIR task, in terms of clutter and object size.
INSTRE:
The INSTRE dataset [47] presents a more challenging setting by including 1-2 smaller objects per image. It comprises 28,543 images across 250 object classes and provides bounding box annotations. To adapt it for our evaluation, we introduce two new subsets: INSTRE-XS (Extra Small): Containing 2,428 queries and gallery of 2,065 images, with objects occupying less than 15% of the image area. INSTRE-XXS (Extra-Extra Small): A more challenging subset with 106 queries and gallery of 120 images, where objects are most 5% of the image.
The PerMiR dataset [35] focuses on multiple object instances across 16 categories (e.g. cars, people, animals, and food items). It comprises 150 queries and a gallery of 450 images, posing a greater challenge due to the presence of multiple instances per image, often including objects from the same category (but not the same instance) the query. PerMiR also provides object mask annotations, which enable object-level evaluation.
PerMiR:
VoxDet-SoIR:
Recently, Li et al. [19] introduced VoxDet, the largest instance-based dataset designed for object detection, featuring 3D voxel-rendered images. The synthetic training set contains 9.6K instances, featuring 55K scenes and 180K bounding boxes, and the real test set includes 20 unique instances, 9, 109 annotations, and 24 complex, cluttered scenes with diverse poses, lighting, and shadows. For evaluation, converted VoxDet into large-scale instance-based retrieval dataset, name it as VoxDer-Solle (VoxDet for Small Object Image Retrieval). The training set consists of distinct objects that do not overlap with those in the test set, averaging 6.34 objects per gallery image with an average object size of 2.82%. This design ensures the dataset's 'ss suitability for both multi-object and small-object scenarios. For the rest of the paper we'll lll refer to this dataset VoxDet for sake of brevity.
To summarize, we present 4 benchmark datasets for SoIR, each addressing a distinct retrieval challenge: (1) INSTRF-XS/XXS: Small objects, but with minimal clutter (2) PerMiR: Instance based retricval in cluttered scene, with less emphasis on object size. (3) Vex Both small and

---

multi-object retrieval, Table compares key properties of several benchmarks: number of Annotated (Annot.) Detected (OVD) objects and their average size ratio. Our proposed benchmarks are highlighted in bold. Notably, VoxDet exhibits the most challenging scenario introducing the highest object density and the smallest object sizes.
For examples, more details and statistics see the Ap- pendix. For more detailed analysis, we created two versions of the VoxDet and PerMiR datasets. The first version, based on ground-truti (GT) object annota- tions, separates the impact of detection performance. The second version, referred to the "in- the-wilds (VoxDetw. PerMiRw), uses object an OVD, particu- larly OWL.x2 [22], simulating more realistic, uncon- strained retrieval scenario. Since INSTRE inherently contains only a single annotated object per image, evaluate it exclusively in the "in-the-wild" set- ting. Detector performance details can be found in the Appendix.
Table 1: Benchmarks: Key Properties.
#Obj
#Obj
Obj. Size
Annot.
OVD
(in %6
Vox
5.8
14.7
PerMiR
4.7
10.4
13.3
INSTRE-XS
1.8
6.6
INSTRE-XXS
1.9
2.2
INSTRE (S1)
1.8
21.0
Products 10K
2.1
27.1
ROxford
5.9
37,6
RParis6K
4.9
41.4
Table 2: Performance comparison (in mAP). All results are for single global feature vector per image. We apply our MaO method CLIP and DINOv2 backbones, in zero-shot and fine-tuning settings. As observed, MaO significamtly improves performance across all backbones and benchmarks. Note that baselines that encodes the entire image yields identical results on VoxDet and VoxDetw and PerMiR/PerMiRRw indicates results with re-ranker.
VoxDet
PerMiR
VexDetw
PerMiRw
INSTRE-XS
INSTRE-XXS
GSS [20]
52.01
26.73
52.01
26.73
82.34
67.98
CVNet' [14]
43.77
24.72
43.77
24.72
43.46
38.72
SuperGlobal [37]
47.33
17.48
47.33
17.48
56.11
33.02
Zero
AMES' [38]
48.68
29.72
48.68
29.72
78.61
68.08
Shot
GeM [29]
51.08
25.98
51.08
25.98
74.74
53.27
CL.IP
44.52
26.98
44.52
26.98
51.97
37.49
DINOv2
51.23
40.57
51.23
40.57
38.92
31.08
a-CLIP
45.30
88.21
42.06
28.96
33.77
16.44
MaO-CLIP
65.22
89.51
52.09
35.81
89.39
71.23
MaO-DINOv2
70.20
89.86
59.60
43.46
71.28
53.13
GeM
61.45
41.20
61.45
41.20
82.61
65.58
CLIP
52.80
38.49
$2.80
38.49
72.90
62.04
Fine
DINOv2
54.33
30.47
54.33
30.47
53.39
44.08
Tune
a-CLIP
59.74
90.13
44.22
38.37
60.18
29.48
MaO-CLIP
79,86
90.86
63.18
39.10
91.29
77.46
MaO-DINOv2
83.70
90.07
68.54
40.44
90.01
75.91
4.2
Results
Table presents mAP results across several benchmarks. Among the rop-perfrrming methods, traditional instance retrieval techniques such GSS[20) and GeM[29] often outperform foundation models. In the zero-shot test and the controlled setting (two left-most columns), e.g. on Vex GSS achieves 52.01% 1 mAP against 44.52% and 51.23% for CLJP and DINOv2 respectively. This result highlights also the limitations of local feature matching used CVNet[ [4] and AMES[38], to effectively match small objects, especially optimization approach (Stage B in Figure 2) to CLIP and DINOv2, we achieve significant performance boost, reaching 65, .22 for CLIP and 70.20% for DINOv2, without any fine-tuning. MaO surpasses previous eestance-based retrieval techniques by 18 26 mAP points on VoxDet. On PerMiR, mAP boosted from range of 17.4-440.577 48.00.57% for techniques without object masks and 88.21% % on a- CLIP (which incorporates object masks) to 89.86%. Furthermore, after fine-tuning on the VoxDet (synthetic) training set, all methods demonstrate improved performance. Notably, a-CLIP exhibits substantial performance gain, attributed to its effective use of object masks. Importantly, MaO, utilizing multi-object fine-tuning strategy and mask optimization, consistently outperforms other methods across both benchmarks.
cluttered scenes. However, by applying our attention
7

---

Next, we present the results for the in-the-wild benchmarks: VoxDetw, PerMiRw. INSTRE-XS and INSTRE-XXS, where an OVD [22] employed. For global methods, the results on VoxDet and VoxDety remain identical since the entire image is processed (with no bounding boxes). MaO-CLIP and MaO-DINOv2 consistently outperform all other zero-shot methods, highlighting the robustness of our approach even in real-world scenarios. Fine-tuning further improves performance across all models, with MaO maintaining its leading position. The only exception occurs with DINOv2 on PerMiRy, where performance declines. We attribute this drop to DINOv2 over-adapting to the VoxDet domain, and increasing the domain gap, whereas PerMiR primarily contains objects such as people and vehicles. Although GeM CNN backbone shows strong improvement after full fine-ture tuning, instead of our finetuned model, MaO-DINOv2 remains the top- performing method on PerMiR w, in zero-shot. On INSTRE, all methods suffer performance drops when dealing with smaller objects, as evidenced by the difference between INSTRE-XXS and INSTRE-XS. However, MaO still achieves the best results with significant margin on both datasets.
The results emphasize the robustness of MaO in handling small and multi-object scenarios with high clutter. Despite processing an average of nearly 15 detected objects per image, MaO consistently shows improved retrieval results with small target objects. This highlights that object characteristics are better captured in our global image representation.
Figure 1 presents qualitative examples demonstrating the effectiveness of our MaO approach in retrieving images containing extremely small objects, even in cluttered scenes. Additionally, Figure compares Top-I retrieval results under challenging conditions, contrasting MaO-CLIP with fine-tuned versions of CLIP and DINOv2 as well as GeM.
Performance vs. Object Size on VoxDet
mAP vs. Resalution on VoxDet
(UP225))
85
MaC-CLIP
Average Precision (AP) [96]
70
SubllGibbll
McD-DINov2
aCLIP(FTI
60
75
CLIPIFT)
DINOv2
50
70
GeM
40
MAP
65
MuO-cuP
S48022
60-
Smaller
50-
2.00 1.75
1.50
1.25
1.00
0.75 0.50
0.25
0.00
250
500
750
1000
1250
1500
1750
Object Area Ratio (%)
Scaled Image Size
(a) Impact of Object Size
(b) Impact of Image Resolution
Figure 3: Retrieval performance analysis on our VoxDet benchmark. The results demonstrate key challenges in small object retrieval, showing the impact of object size (a) and image resolution (b) on retrieval accuracy. Performance declines as object size decreases. Higher image resolutions enhance retrieval effectiveness for MaO, whereas other alternatives marginally benefit from increased resolution. MaO consistently outperforms existing methods, showing robustness across all conditions.
Figure 5 further provides a visualization of explainability maps, comparing fine-tuned DINOv2 and MaO. While DINOv2's attention predominantly focuses on background elements, such as shelves, the attention maps of MaO are more effectively distributed across the relevant objects, highlighting the advantage of our Stage-B mask optimization, ensuring a more precise and object- focused representation. We also report in Appendix C the performance of MaO non-small objects using PODS [41] and ILIAS [13] benchmarks. Appendix presents a comparison with an MLLM method.
4.3 Controlled Analysis
Figure 3a shows the retrieval performance on VoxDet (measured AP) as a function of the object size in the gallery, quantified by the mean object-to-iman tmme area ratio. The evaluation covers objects occupying roughly 0.1% % to 2% of the total image area.
To isolate the impact of object size from detection quality, this experiment runs in controlled setting, utilizing ground-truth (GT) object bounding boxes and object masks both o-CLIP and MaO. We observe that global fine-tuning consistently improves performance across all methods, as
8

---

indicated by the contrast between dashed and solid lines of the same color. Notably, MaO achieves the most significant performance gains with both CLIP and DINOv2 backbones, outperforming other approaches, even for very small objects (down to 0.5% of the image area). MaO surpasses previous retrieval techniques, including SuperGlobal [37]. GeM [29], and fine-tuned models such as CL.P 2INOv2(FT), and GeM(FT). Particularly, our method achieves an AP of 50% for objects occupying just 0.5% the image area, demonstrating a substantial improvement over prior methods. The fine-tuning benefits of MaO are particularly pronounced compared to other techniques, reinforcing its effectiveness in handling small object retrieval in cluttered scenes.
Finally, we analyze the impact of image resolution retrieval performance. Figure 3b illustrates how performance changes images are resized to different resolutions, while the relative object size, remains constant. For global methods. retrieval performance either plateaus or declines resolution increases. While a-CLIP and DINOv2 exhibit slight improvements with higher resolution input, further increase in resolution leads to drop in performance. This trend is likely due to the limitations discussed [8], which attributes the decline to the "ovverfitting" of foundation and pre-trained models low-resolution training images. In contrast, MaO effectively leverages high-resolution images, consistently outperforming global methods. MaO further exhibits a predictable and gradual decline in performance with resolution decrease, demonstrating its robustness across different image
scales.
Query
CLIP
DINOv2
GeM
Ours
Sandhaaah Cookie
Figure 4: Examples of Top-1 retrieval results from VoxDetw (top row), INSTRE- XXXX (middle row), and perMiRw (bottom row). Red and green frames indicate incorrect and correct retrievals, respectively. In the images, red bounding boxes correspond to OVD detections, while green bounding box highlights the target object. MaO successfully retrieves the correct image, even in cases with high levels of clutter and small objects.
4.4 Runtime Analysis
In this section, we report and compare the encoding and retrieval times for various methods. Most state-of-the-art retrieval methods adopt either two-stage design with re-ranking (e.g. SuperGlobal) or rely on computationally intensive local matching (e.g. AMES), making them significantly less efficient during online search. We present runtime analysis comparing MaO against two SoTA methods of SuperGlobal (SG) and AMES. Table presents the runtime results, for both the encoding and the re-ranking stages (used in SG and AMES). All experiments were conducted on the VoxDet- SoIR benchmark, containing 1 1,581 images in gallery and 2,000 query images. The retrieval time based vectorized (global) feature ranking is approximately 0.03 ms across all methods and is included the search time.
The online search-time advantage of MaO is clear: at test time (i.e., from receiving query image to producing ranked retrieval results), MaO requires ms for query encoding (single object). In comparison: SG takes 65 ms for query encoding and 0.37 ms for re-ranking (per-query). AMES takes

---

Method
Vox
Vox
Backbone ZS DINOV2
S1.23
$1.23
Fine tuning
54.33
54.33
image optimization
Multi-object fine-tuning (Stage A)
75.04
63.14
Multi-objeet optimization (Stage B)
83,70
68.54
Table 4: Ablation study of our method on VoxDet and VoxDety (mAP).
Figure 5: Explainability maps for the full image (center) and MaO (right) with DINOv2. Warmer colors indicate higher attention.
65 ms for query encoding (same SG) but 2,286 ms for retrieval (per-query) due to the extremely heavy local matching process. Considering 2,000 queries the total search time is 0.8, 4572 and 0.06 sec for SG, AMES and MaO respectively. For offline gallery encoding, SG and AMES each require about 65 ms per image, while MaO takes 174 ms per image plus roughly 0.5 s per image for OVD mrepproesssing Notably, we did not apply any acceleration techniques, such as batch processing or optimized OVD implementations (e.g.. ONNX) for the OVD pre- pre-processing. Consequently, our method delivers substantially higher accuracy (see Tab. 2) and is significantly faster than SG and AMES at test time, albeit with a greater offline encoding cost. Encoding time can be further reduced through code-level optimizations and parallelization, as the MaO pipeline is inherently well-suited for data parallelism.
Table 3: Runtime comparison on VoxDet dataset. Encoding time corresponds to an average of ~ objects per image. Search time corresponds to the retrieval stage for 2,000 queries, including vectorized search, re-ranking, and local matching where applicable. ": without OVD.
Method SG
Encoding (ms/img)
Re-Ranking (ms/query)
Search Time (sec)
65
0.37
0.8
AMES
65
2 2286
4572
MaO
174"
0.06
4.5
Ablation Study
To further assess the contribution of each component in our method, we conduct ablation study, presented in Table 4, which reports mAP results on VoxDet and VoxDet, across different configura- tions. The baseline model is off- shel- DINOv2 with global image representation. Fine-tuming is performed with VoxDet training set. Full image optimization extends the fine mne-tuned model by naively incorporating global and single optimized attention map for all objects. This method in fact results in performance drop on VexDe Multi-object fine-tuning is MaO's stage A, and our Multi-object optimization (based on optimization for individual crops and aggregation) indicates our complete object-wise MaO method, which presents significant gains, on both benchmarks, showing the further impact for our masked optimization refinement. For more ablation studies including clutter impact please refer to the Appendix.
5 Summary and Discussion
While most prior IIR work examines large objects or scenes, the domain of retrieving small instances in cluttered environments, remains underexplored. To bridge this gap. we first establish benchmarks that enable an in-depth investigation of this challenge, highlighting the significant impact of object size and scene clutter on retrieval performance. We further propose MaO, a novel approach for SoIR that leverages foundation models such CLIP and DINOv2. MaO encodes images in two stages: multi-object fine-tuning stage, followed by post-training refinement process (multi-object attention optimization) that yields focuuee representation, ultimately generating single-image descriptor for search and retrieval.
MaO achieves strong retrieval performance, successfully identifying objects as small as 0.5% of the image area. Nonetheless, certain limitations remain. Although prior work has demonstrated the effectiveness of detector-based retrieval methods (e.g., [9, 39, 43]), our performance can be affected by imperfect detector recall, resulting in slight retrieval drops. Appendix F presents a thorough analysis of the OVD impact and discusses further limitations our approach, yet highlighting strong generalization capabilities. We hope this work will inspire further research in the challenging task of small object retrieval.
10

---

References
[1] Reza Abbasi, Ali Nazari, Aminreza Sefid, Mohammadali Banayeeanzade, Mohammad Hossein Rohban, and Mahdich Soleymani Baghshah. CLIP Under the Microscope: Fine-Grained Analysis of Multi-Object Representation. In CVPR, 2025. 2
[2] Guoyuan An, Yuchi Huo, and Sung Eui Yoon. Hypergraph propagation and community selection for objects retrieval. In NeurIPS, 2021. 2,3
[3] Yalong Bai, Yuxiang Chen, Wei Yu, Linfang Wang, and Wei Zhang. Products-10k: large-scale product recognition dataset. arXiv preprint arXiv:2008, 10545, 2020. 2,
[4] Walid Bousselham, Angie Boggust, Sofian Chaybouti, Hendrik Strobelt, and Hilde Kuehne. Legrad: An explainability method for vision transformers via feature formation sensitivity. arXiv:2404.05214, 2024. 5,
[5] Bousselham, Walid and Chaybouti, Sofian and Rupprecht, Christian and Ferrari, Vittorio and Kuehne, Hilde. Maskinversion: Localized embeddings via optimization of explainability maps. arXin:2407.20054 2024. 4,5
[6] Zhenfang Chen, Zhanghui Kuang, Wayne Zhang, and Kwan- K Wong. Learning local similarity with spatial relations for object retrieval. Proceedings of the 27th ACM international conference on Multimedia, pages 1703-1711, 1, 2019. 3.4
[7] Sivan Doveh, Nimrod Shabtay, Wei Lin, Eli Schwartz, Hilde Kuchne, Raja Giryes, Rogerio Feris, Leonid Karlinsky, James Glass, Assaf Arbelle, et al. Teaching vlms to localize specific objects from in-context examples. arXiv:2411. 1.14411,,131,,, 2024.
[8] Anthony Fuller, Danicl G Kyrollos, Yousef Yassin, and James R Green. LookHere: Vision transformers with directed attention generalize and extrapolate. NewrIPS, 2024. 9
[9] Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image scarch. In ECCV, pages 241 2-72 Springer, 2016. 4, 10
[10] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ICLR. 2022. 6
[11] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Teddy Furon, and Ondrej Chum. Efficient diffusion on region manifolds: Recovering small objects with compact CNN representations. In CVPR, pages 2077-2086. 2017. 2,4
[12] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yan Yen Lo, et al. Segment anything. In CVPR, pages 4015-4026, 2023. 5.3
[13] Giorgos Kordopatis-Zilos, Vladan Stojnić, Anna Manko, Pavel Suma, Nikolaos-Antonios Ypsilantis, Nikos Efthymiadis, Zakaria Laskar, Jifí Matas, Ondřej Chum, and Giorgos Tolias. Ilias: Instance-level image retrieval at scale. CVPR, 2025. 8, 2
[14] Scongwon Lec, Hongje Scong, Suhycon Lec, and Euntai Kim. Correlation verification for image retrieval. In CVPR, pages 5374-5384, 2022. 2, 3, 6,7
[15] Seongwon Lee, Hongje Seong, Suhyeon Lee, and Euntai Kim. Correlation verification for image retrieval. In CVPR, pages 5374-5384, 2022. 3
[16] Hila Levi, Guy Heller, Dan Levi, and Ethan Fetaya. Object-centric open-vocabulary image-retrieval with aggregated features. BMVC, 2023. 2
[17] Hila Levi, Guy Heller, and Dan Levi. For: Finetuning for object level open vocabulary image retrieval. WACV, 2025. 2
[18] Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. Chatting Makes Perfect: Chat-based Image Retrieval. In NeurIPS, pages 61437-61449, Curran Associates, Inc., 2023. 2
[19] Bowen Li, Jiashun Wang, Yaoyu Hu, Chen Wang, and Sebastian Scherer. VoxDet: Voxel learning for novel instance detection. In NeurIPS, pages 10621, 2023. 2, 4, 6
[20] Chundi Liu, Guangwei Yu. Maksims Volkovs, Cheng Chang, Himanshu Rai, Junwei Ma, and Satya Krishna Gorti. Guided similarity separation for image retrieval. NewrlPS, 32, 2019. 3, 6,

---

[21] David G Lowe. Object recognition from local scale- features. In CVPR, pages 1150-1157. Ieee. 1999. 4
[22] Matthias Minderer, Alexey A. Gritsenko, and Neil Houlsby. Scaling Peen-Vocabulary Object Detection. In NeurlPS, 2023, 6, 7, 8,
[23] Phuc Nguyen, Tuan Due Ngo, Evangelos Kalogerakis, Chuang Gan, Anh Tran, Cuong Pham, and Khoi Nguyen. Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance. In CVPR, pages 4018 4028, 2024.
[24] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding,
2018. 5
[25] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. 7MLR, 2024. 6
[26] Anton Osokin, Denis Sumin, and Vasily Lomakin. Os2d: One-stage one-shot object detection by matching anchor features. In ECCV, pages 635 652. Springer, 2020. 2
[27] Filip Radenović, Ahmet Iseen, Giorgos Tolias, Yannis Avrithis, and Ondicj Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In CVPR, pages 5706-5715, 2018. 2,
[28] Filip Radenović. Giorgos Tolias, and Ondiej Chum. Fine-tuning enn image retrieval with no human annotation. TPAMI, (17))6655-1668 2018."
[29] Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning enn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(7):1655-1668, 2018. 2, 3, 6, 7,
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., pages 8748-8763, 2021. 6
[31] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: tanguagg-gguiedd dense prediction with sontext-award prompting. In CVPR, pages 18082- 18091, 2022. 2
[32] Ali S Razavian, Josephine Sullivan, Stefan Carlsson, and Atsuto Maki. Visual instance retrieval with deep convolutional networks. ITE Transactions on Media Technology and Applications, 4(3):251-255 2016.
[33] Tianhe Ren, Shilong Liu. Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang. Yukang Chen, Feng Yan, et al. Grounded SAM: Assembling open-world models for diverse visual tasks. arXiv:2401.141: 14159, 2024. 3
[34] Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto Souza. Learning with average precision: Training image retrieval with a listwise loss. In CVPR, pages 5107-5116, 2019.
[35] Dvir Samuel, Rami Ben-Ari, Matan Levy, Nir Darshan, and Gal Chechik. Where's waldo: Diffusion features for personalized segmentation and retrieval. In NeurlPS, 2024. 2, 4,6
[36] Artsiom Sanakoyeu, Vadim Tschemezki, Uta Buchler, and Bjorn Ommer. Divide and conquer the embedding space for metric lcarning. In CVPR, pages 471-480, 2019. 2
[37] Shihao Shao, Kaifeng Chen, Arjun Karpur, Qinghua Cui, André Araujo, and Bingyi Cao. Global features are all you need for image retrieval and reranking. In CVPR, pages 1 11036-11046, 2023. 2,3, 4, 7,9
[38] Pavel Suma, Giorgos Kordopatis Zilos, Ahmet Iscen, and Giorgos Tolias. AMES: Asymmetric and eefeer-efficient similarity estimation for instance-level retrieval. In ECCV, pages 307- 325-325. Springer, 2024. 6,7
[39] Shaoyan Sun, Wengang Zhou, Qi Tian, and Houqiang Li. Scalable object retrieval with compact image representation from generic object regions. ACM Transactions on Multimedia Computing, Communications, and Applications, 12(2):1211, 2015, 4,
[40] Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want. In CVPR, pages 13019-13029, 2024. 4, 6
[41] Shobhita Sundaram, Julia Chae, Yonglong Tian, Sara Beery, and Phillip Isola. Personalized representation from personalized generation. arXiv preprint arXiv:2412.16156, 2024. 8.2
12

---

[42] Fuwen Tan, Jiangbo Yuan, and Vicente Ordonez. Instance-level image retrieval using reranking transform- In CVPR, pages 12105-12115, 2021.
[43] Marvin Teichmann, Andre Araujo, Menglong Zhu, and Jack Sim. Rettrieto-Retrievee: efficient regional aggregation for image search. In CVPR. pages 5109-5118, 2019. 10
[44] Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max-pooling of cnn activations. ICLR, 2015. 3
[45] Giorgos Tolias, Yannis Avrithis, and Hervé Jégou. Image search with selective match kemels: aggregation across single and multiple images. International Journal Computer Vision, 116:247-261, 2016. 4
[46] Issar Tzachor, Boaz Lerner, Matan Levy, Michael Green, Tal Berkovitz Shalev, Gavriel Habib, Dvir Samuel, Noam Kornuut Zailer, Or Shimshi, Nir Darshan, al. Effovpr: Effective foundation model utilization for visual place recognition. 1rXii:2445.18000 2024.
[47] Shuang Wang and Shuqiang Jiang. Instre: benchmark for instance-level object retrieval and recognition. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 113):11-1,, 2015. 2, 4,
[48] Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In CVPR, pages 2575-2584, 2020. 2. 6
[49] Bin Yan, Martin Sundermeyer, David Joseph Tan, Huchuan Lu, Federico Tombari. Towards real-time open-vocabulary video instance segmentation. In WACV, pages 1861-1871. IEEE, 2025.
[50] Min Yang, Dongliang He, Miao Fan, Baorong Shi, Xuetong Xue, Li, Errui Ding, and Jizhou Huang. DOLG: Single-stage image retrieval with deep orthogonal fusion of local and global features. Pro- ceedings of the IEEE/CVF International conference on Computer Vision, pages 11772-11781, 2021.
[51] Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip llievski. MLLMs know where to look: Training-free free perception of small visual details with multimodal LLMs. In ICLR, 2025. 2
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper contributions are based the claims introduced in the introduction and abstract, and back-up with experiments and discssuions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discuss the the method's limitations and trade-offs, in the introduc- tion, experiments and the discussion sections.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper provide full details for the methods and experiments, with dedi- cated section of implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
While all data used is public, we intend to release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss... wttpssccccpcccccccceecec.eccccct http/gudddes/oooee /Cooe/eoooiisseseeeeee ssssssooiccyyyys for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The paper provides all the details within the paper content, and additionally with code release.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: The paper follows previous abd standard evaluation setups for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the specific architecture on which our expierments were performed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : httpppp.cccppppccctttccccpccccct ctt.ciiiiiiiiiiieieeeeeee
Question:
Answer: [Yes]
Justification: To the best of our knowledge, there no special concerns regarding code of ethics in this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: this paper we introduce a general fine-tuning strategy for Neural Networks. To the best of our knowledge, there is additional/special concerns raised from this paper in particular.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper were properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in wwwrrsswtthhhe.....
package should be provided. For popular datasets, paperswi thcode. com conddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http://euurissceee /eseur/ppsccc/ooofeeenenee/////e for what should or should not be described.
20

---

Appendix
This Supplemental Material presents the following additional details concerning the experimental results and their analysis.
A. Preliminary on LeGraD and Explainability Map
B. Visualization of Explainability Maps
C. Evaluation non-small Objects and more Datasets
D. Comparison to MLLM
E. Additional Ablation Studies
E. Limitation Analysis
G. More Qualitative Results
H. Image Samples from PerMiR dataset Additional Analysis of INSTRE
Additional Analysis of VoxDet
A Preliminary on LeGraD and Explainability Map
LeGrad [4] is gradient-based feature attribution method specifically developed for Vision Trans- former (ViT) architectures. Unlike traditional explainability techniques such Grad-CAM, which are optimized for convolutional networks, LeGrad exploits the attention mechanisms inherent in ViTs to generate an explainability map that highlights the most influential image regions contributing to the model's decision.
Mathematical Formulation Consider a vision- vision-language model F that processes an input image 33xW>* and outputs an activation score E which may represent classifier's confidence level or the cosine similarity between vision and text embeddings. In ViT with L. layers, let A' represent the attention maps at layer 1. The gradient of the activation s with respect to the attention maps is computed as:
Os
V
TA
LeGraa determines the explainability score for each attention map A by applying the ReI.U function to eliminate negative contributions:
Ōs
(*)
hm ΣΣ ReLU
(3)
OA
where h denotes the number of attention heads and n represents the number of visual tokens.
The final explainability map E is obtained by averaging the scores across all layers:
1 E == I
(4)
To construct the final explainability map, the influence of patch tokens is isolated, reshaped into a 2D spatial representation, and normalized using min-max scaling:
E norm(reshape(( E)).
B Visualization of Explainability Maps
Figure shows the explainability maps before and after our multi-object optimization stage on sample from VoxDet.

---

Figure B1:
Explainability maps for optimized token. Columns from left to right. (1) The cropped object. (2) ground-truth mask (3) Explainabilityy map before optimization in MaO-CLIP (4) Explain- ability map after our multi-object optimization with MaO-CLIP. The explainability attend to the objects after optimization (warmer colors coincide the object areas). A single optimized token can capture multiple objects across different input object crops.
C Evaluation on non-small Objects and more Datasets
To further validate the generalization of our approach, conducted additional experiments new dataset, ILIAS [13], which includes a mix of small, medium, and large objects. We further add MaO also with additional backbone of SIGLIP. Note that ILIAS also includes small objects among the gallery images. The ILIAS dataset consists of 1,000 unique classes, 1,232 query images, and 4.715 positive gallery images. available in two versions: one with 5 million distractor images and another with 100 million distractors. We report results and comparisons on the version without distractors avoiding heavy computation yet showing clear differences between various approaches. In addition to the baselines, we also include results for Stage A, as defined in the main paper, in the zero-shot setting and also after applying multi-object fine-tuning our supervised setting.
Table C1 compares our method to the strongest baseline, GeM, as well as to DINOv2 and SIGLIP models whose results reported in [13]. Specifically, include DINOv2-Base and SIGLIP-Base (without linear adaptation), and DINOv2-Large and SIGLIP-B' (with linear adaptation). Note that the reported values for all baselines from [13] are mAP@ IK, which tends to be higher than the true mAP computed over the entire gallery. We follow the same performance measure to allow comparison.
Although the extensive experiments in the main paper include tests on common backbones and SoTA instance based retrieval methods, additionally present results of MaO using the also the SIGLIP backbone. We observe that incorporating our multi-object optimization (MaO) into SIGLIP leads to substantial performance gain also this strong backbone. The results indicate that method significantly outperforms all baselines in both zero-shot and fine-tuned settings with the DINOv2 backbone. Furthermore, applying MaO to SIGLIP in Zero-Shot manner achieves top performance even when compared to the fine-tuned linear adaptation model.
MaO Performance on large and general image content- retrieval: We conducted an additional evaluation on the PODS dataset [41], which has an average object-to-image area ratio of 20%. As shown in Tab. C2, MaO performs robustly these larger instances, with no degradation in performance.
D Comparison to MLLM
Although current multimodal large language models (MLLMs) such as Qwen or Ferret are not designed for large-scale image retrieval, we still provide an evaluation for comparison. We can operate such models by requiring pairwise checks between the query image and each gallery image, e.g., prompting whether the object appears in a given image, with both images passed through the full MILMM pipeline. This approach is computationally costly and unsuitable for large-scale scenarios, yet as an interesting experiment we illustrate the performance of Qwen2.5-VL on 100 queries from VoxDet subset considering gallery of 200 images.

---

Table C1: Retrieval performance (mAP) the ILIAS dataset (without distractors). Results shown for both zero-shot and fine-tuned settings. Stage A refers to our average pooling of object embeddings in the zero-shot setting and multi-object fine-tuning in the supervised setting. Indicates results with linear adaptation. B,L indicate Base and Large backbone versions respectively.
Method
mAP
GeM
30.20
Zero Shot
SIGLIP-B
20.20
DINOv2-B
14.30
DINOv2-B B (Stage A)
36.41
MaO-DINOv2-B
38,93
MaO-SIGLIP-B
62.22
GeM
34.46
Fine Tuned
SIGLIP-B'
58.70
DINOv2-L'
46.50
DINOv2-B B (Stage A)
45.29
MAO-DINOv2-B
48.85
Table C2: Results (with finetuned models) on non non-small objects, PODS dataset.
Method
mAP
CLIP
0.341
DINOv2
0.332
MaO-CLIP
0.368
MaO-DINOv2
0.393
Table DI: Comparison of Qwen2.5-VL and MaO-DINOv2 on a subset of VoxDet.
Method
Matching Type
Accuracy
Qwen2.5-VE
Pair Matching
19%
MaO-DINOv2
Single Vector Retrieval
85%
These results show that, beyond its impractical runtime, MILM like Qwen struggles with small- object retrieval, highlighting the advantages of dedicated retrieval models such as MaO.
E
Additional Ablation Study
E.1
Multi-Object Optimization
We present additional ablation studies to examine the effect of the regularization term on performance. Figure E1 presents the mAP results for MaO-CLIP and MaO-DINOv2 on VoxDetw dataset across different values of a in Eq. (1) of the main paper. We set the optimal value as 0.03 across all of our benchmarks, according to Figure El.
We further examine the impact of the number of iterations in refinement stage (Stage B). Figure E2 presents the result as function of the iteration count. We therefore fixed the number of iterations to 80 in all of benchmarks. This stage typically takes 0.03 see per-object, per-image.
Limitation Analysis
Our proposed approach relies on the performance the OVD. In this section we evaluate the used OVD of OWLv2 [22] on the final performance in different aspects. As our model uses an OVD retrieval pipeline. Two-stage designs are common in computer vision and are well-established paradigm that is widely adopted in various methods as seen e.g. in SAM [12] that uses an OVD as its first stage, Grounded- SAM [33] (combining Grounding DINO OVD). Open3DIS [23] that performs
Dependence on OVD:
first stage it has an inherent limitation any modular, detection-firs

---

MAP
0.70
0.03
0.00
0.45
0.40
MOC.FF Matioooo
0.32
0.00
0.02
0.00
0.06
0.08
010
Figure B1: The mAP performance of MaO-CLIP and MaO-DINOv2 as function of the regularization weight a examined on VexWet
MAP vs Number of Iterations
0.85
MLC-CRDU
0.64
0.63
Number Itorations
Figure E2: The mAP results as a function of the iteration count in our multi-object optimization stage for the VoxDetyy dataset.
open-vocabulary 3D instance segmentation, and TROY-VIS [49] that uses an existing OVD for video instance tracking, or in many object detectors that rely on object proposal methods. In our case, the OVD is used purely as an objectness (object proposal) stage. We further presented analysis showing that modern OVDs (e.g., OWLv2) are capable of detecting small and even unusual objects effectively.
Next, Table F1 we evaluate the Recall values in our setting. We present the results with different predicted bounding-box vs. tight ground truth overlaps VoxDet, measured by ToU threshold. Results are provided for two overlap thresholds ToU = 0.6, 0.9. We observe that relaxing the overlap constraint for True Positive significantly increases Recall, indicating that while OVD performs well in object detection, it may lack precision in tightly localizing objects. However, since our approach applies center-cropping around each predicted object, often with margin, particularly for small objects (avoiding resize), it effectively captures the object even when localization is imprecise. The subsequent mask optimization process further refines the representation, ensuring a more precise encoding even if the object is not perfectly centered within the crop. The average number of detected objects per scene further reflects the complexity the benchmarks. While INSTRE contains fewer than five detected objects per image, VoxDet and PerMiR feature more than ten, highlighting their higher level of clutter.
Table F1: Our OVD (OWLv2) detection performance. We report the average number of objects detected per image. For True-Posiitv detection is defined as IoU > 0.9.
In case of ToU = 0.6 we further relax the condition on objects contained in the predicted bbox by setting containment ratio threshold of 0.3 for true positive.
Avg. #Objects
Recall %]
(TxDe = 0.9)
14.7
63
VoxDet (ToU 0.6)
14.7
96
INSTRE-XS
3,8
87
INSTRE-XXS PerMiR
4.2
83
10.43
88

---

Novel OVD methods can operate even under extreme object size and clutter conditions (see Table F2), as we leverage this in our approach.
The impact of the OVD on MaO is limited and measurable: To further address this concern, we conducted an empirical evaluation our chosen OVD model on two GT-annotated benchmarks and reported the results in Tab. F2. The results show that even under imperfect detection (partial object detection) our method continues to outperform existing approaches.
Table F2: OVD performance across varying object-to-immge area ratios, on VoxDet-SoR.
Object-to-image area Ratio (%)
0.2
0.4
0.8
1.0
1.5
2.0
Object Recall
0.68
0.83
0.83
0.85
0.90
0.87
While there is performance degradation the extreme case of 0.2%, the results show the high
capability of owlv2 in discriminating even small objects.
Performance of the OVD in cluttered scenes In Tab. F3 we provide breakdown of the used OVD (on VoxDet-SoR) according to several clutter levels. The OVD maintains strong performance even in extreme cases where 12-24 objects are detected in the image. We further analyze the impact of clutter level on retrieval performance. To systematically evaluate performance under increasing clutter, we conduct an experiment where the number of objects in the gallery images is gradually increased. Using the GT masks, we encode positive images by combining the representation
the target object with those of randomly selected negative objects. At each step, as more objects are added, we incrementally incorporate additional object embeddings into our MaO when forming the global image representation. The results this experiment are presented in Figure F1 and confirm a moderate performance drop with gradually increased clutter (up to objects), decreasing from 0.96 mAP for a single object to approximately 0.82 mAP when six objects are present. This further highlights MaO capability in handling multi-objec representation.
mAP vs. Clutter
96
MaO(CLIP)
MaO(DINOv2)
94
92
mAP [%]
90
88
86
84
82
1
2
4
5
No. of Objects (Clutter)
Figure Fl: Impact of scene clutter on retrieval. As expected Increased scene clutter negatively affects accuracy. Table F3: Recall of the OVD across different numbers of detected objects (on VoxDet-SolR).
Objects
12
16
20
24
Recall
0.80
0.95
0.95
0.89
In summary, we believe our method effectively harnesses the strengths an OVD provides for image retrieval, while addressing and mitigating its inherent limitations.
Partial Detection, Occlusion and Heavy Clutter: example of partial detection shown in Figure F2, where the OVD detects only part of the target image. Scenes with high clutter and
5

---

ALLIE
Figure F2: Retrieval failure likely due to partial object detection. The green box is the ground truth, while red boxes are OVD detections, none covering the full object. The object occupies 1% of the image area.
numerous objects present additional challenges to model. In cases where the OVD detects more than 25 items, we often find that the target object, although detected, may under-representee and thus missed during retrieval. Occlusion and overlapping objects further increase the sensitivity for small objects, making difficult to extract distinctive features necessary for accurate retrieval.
Finally, we further noticed that our model is more biased toward objects and has shortcomings on faces, individuals, and landmarks.
G More Qualitative Results
Additional qualitative comparisons on tiny objects in INSTRE-XXS are presented in Figures G1 and G2. We provide examples comparing approach with strong fine-tuned versions of DINOv2 and CLIP, as well as the state-of-the-art instance retrieval method of SuperGlobal [37]. While DINOv2 and CLIP each retrieve only one correct image in Figure GI in all other results in Figures G1 and G2 they fail to retrieve the correct images within the top-5 ranks. contrast, our approach consistently retrieves the correct images from the gallery.
In Figure G3 we further present comparative results on VoxDetw.
H Image Sampled from PerMiR Dataset
Figure Hl presents examples of multiple samples from the PerMiR dataset, the query (at top) and associated gallery images (at the bottom). Since PerMiR was constructed from videos, some cases can be easy and some extremely hard.
I Additional Analysis of INSTRE
To further highlight the differences between INSTRE-X: -S, INSTRE-XES -XXS, and the standard INSTRE- SI and INSTRE-M benchmarks, we display the largest objects from the INSTRE-SI, INSTRE-M. INSTRE-XS, and INSTRE-XXS subsets Figure 11, demosntrating the significant difference the object size, between the benchmarks. Additionally, we present the distribution of object sizes for each benchmark in Figure 12, illustrating the importance of our subset selection for small object tasks. Note that INSTRE-XS and INSTRE-XXO impose upper size limits of 15% and 5%, respectively.

---

Rank
Ours
DINOv2
CLIP
SuperGiobal
Query
Figure GI: Qualitative results on INSTRE-XS and INSTRE-XXS, comparing retrieval performance across MaO vs. fine-tuned DINOv2, CLIP, and thrte-rrr SuperGlobal. Green and red borders indicate correct and incorrect results respectively. The query input image is shown at the bottom. Zoomed object images are overlaid retrieval image for better visibility. Except for one correct image retrieved by DINOv2 and one by CLIP, all methods fail to retrieve the correct small target images within the top-5 ranks. In contrast, MaO method successfully retrieves the true target images in all top-5 results. The relative object sizes for the correct top-5 gallery images retrieved by our method are 1. 1.24%, 1.93%, 1.21 %, 2.1 %, and 1.01 %, respectively.
J Additional Analysis of VoxDet
We provide additional insights into the train and test datasets of VoxDet-SoIR. The figures illustrate samples of five instances, where the query images displayed at the top, and their corresponding gallery images are shown at the bottom. Figure J1 presents examples from the training set, while Figure J2 showcases samples from the test set. While the training dataset includes small objects within complex scenes containing multiple objects and significant occlusions, the test dataset features even smaller objects in realistic, cluttered environments with fewer occlusions. Figures J3 and J4 illustrate the distribution of the number of objects per gallery image in the training and test datasets, respectively. Similarly, Figure J5 depicts the relative sizes of objects within each gallery image for the training and test datasets, respectively. Although the test dataset exhibits less variability in the number of objects per image, the objects themselves appear even smaller compared those in the training dataset.

---

Rank
Ours
DINOv2
CLIP
SuperGlobal
Query
Figure G2: Qualitative results on INSTRE-XS and INSTRE-XXS, comparing retrieval performance across MaO vs. fine-tuned DINOv2, CLIP, and state-of-the-art SuperGlobal. Green and red borders indicate correct and incorrect results respectively. The query input image is shown at the bottom. Zoomed object images are overlaid retrieval image for better visibility. All other methods fail to retrieve the correct small target images within the top-5 ranks. In contrast, MaO successfully retrieves the true target images all top-5 results. The relative object sizes for the correct top-5 gallery images retrieved by method are 2.36%, 1.03%, 2.59%, 1.82%, and 2.08%, respectively.

---

Query
Ours
SuperGlobal
PerMiR
VoxDet-Wu W
Figure G3: Results using various methods on VoxDetw and PerMiRw, showing Rank- results. Red and green frames indicate incorrect and correct retrievals, respectively. Red boxes in the image denote OVD detections. Notice high level of clutter caused by frequent OVD detections in gallery images. Zoom-in objects are depicted at the edge to enhance visibility, The target object size for the first three lines PerMiR w are 20% 42%, and 32% of the image area, respectively, while in VoxDetw, target objects cover 0.85%, O. 0.17, and 0.51 % of the image area.
Sample images from PerMiR
Label:
Query Labot
Label:
Figure HI: Example samples from the PerMiR dataset. The top row in each subfigure represents the query images, while the bottom row shows the corresponding gallery images containing the retrieved instances. The bounding boxes indicate the objects in the query images. Gallery images typically contain multiple objects and often from the same catergory, several cars, food items or dogs.

---

ANGRY BIRDS
(a)
(b)
Figure II: Largest Objects in the INSTRE Datasets: Examples of the largest objects are presented from the following subsets: (a) INSTRE-SI, (b) INSTRE-S2, (c) INSTRE-XS, and (d) INSTRE-XXS, with their respective relative sizes being 99.26%, 90.7%, 15%, 4.99%
Object Area Ratio Distribution
INSTRE-S1 INSTRE-S2 INSTRE-XS
INSTRE XXS
Frequency (log scale)
10?
101
10a
40
100
Object Area Ratio (%)
Figure 12: Distribution of Object-Image area ratio across the four INSTRE subsets: INSTRE-S I1 INSTRE-S2 and our selected subsets INSTRE-L and INSTRE-T. Notably, INSTRE-Xi XS and INSTRE- XXS are limited to small object size, allowing a better performance assessment on this specific task.
10

---

Voxdet Train Dataset Samples
Query Label: 7354
Gurry Label: 7314
Labei: 2112
Quiry Label: 2330
Label: 3852
Figure Example samples from the VoxDet training dataset. The top row displays query images, and the bottom row presents gallery images containing the corresponding instances.
Voxdet Samples
Figure J2: Example samples from the VoxDet test dataset. Similar to the training the top row contains query images, while the bottom row includes gallery images with matching instances.
VoxDet Number of Objects per Image (Train)
10000
Mean: 6.35
Median: 6.00
3.17
8000
6000
2000
2
3
10
11
12
14
Objects
Figure J3: Distribution of the number of objects per gallery image in the VoxDet training dataset. The dataset exhibits range of object counts per image, with a mean of 6.35 objects, a median of 6, and standard deviation of 3.17.

---

VoxDet - Number of Objects per Image (Test)
500
300
200
100
Number Objects
Figure J4: Distribution of the number of objects per gallery image in the VoxDet test dataset. The number of objects varies similarly to the training set but may reflect differences in scene complexity.
VoxDet Relative Object Size Distribution (Train)
400000
Median: 0.016
250000
200000
150000
50000
1.0
Relative Object Size
VoxDet Relative Object Size Distribution (Test)
Mean: 0.011 Median: 0.006
0.024
8000
10
Relative Object Size
Figure J5: Top: Relative object sizes Object to image area ratio, within each gallery image in the VoxDet training dataset. The figure illustrates the proportion of the object size relative the image dimensions. Bottom: relative object sizes within each gallery image in the VoxDet test dataset.
12


Reviews:
[{"review_id": "7QEJjfiK3Z", "reviewer": "Reviewer_23tf", "summary": "This paper introduces Multi-object Attention Optimization (MaO), a novel framework for Small Object Image Retrieval (SoIR) in cluttered scenes. MaO proposes a two-stage approach: multi-object fine-tuning followed by attention-based refinement using explainability maps and object masks to create a unified image descriptor. The authors also introduce new benchmarks, featuring smaller objects and higher clutter. Experiments demonstrate MaO's superior performance on these benchmarks compared to existing methods and foundation models.", "questions": "[Q1] Given that the paper primarily focuses on the effectiveness of MaO for small objects, could the authors provide an explicit analysis of MaO's performance on large objects or its applicability and effectiveness for general image content-based retrieval beyond object-centric queries?\n\n[Q2] could the authors elaborate on the potential impact of significant failures in object detection or localization by the OVD on MaO's \"in-the-wild\" retrieval performance? Particularly in highly cluttered scenes where such failures might be more prevalent.\n\n[Q3]  Could the authors provide a more in-depth explanation of how explainability maps are precisely computed and optimized within LeGrad, specifically addressing the mechanism when multiple objects contribute to a single global token?\n\n[Q4] Could the authors provide a more detailed comparison highlighting MaO's specific advantages over  $\\alpha$-CLIP and other mask-based methods in handling multi-instance, cluttered scenes, beyond just the mAP difference?\n\n[Q5] Given that Stage B optimization takes 0.03 seconds per-object, could the authors provide a more detailed explanation of how this contributes to the total image encoding time, and clarify the total online inference time for a single query, including OVD detection?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "[S1] MaO offers an innovative solution to the challenging and practical problem of retrieving small objects in cluttered images, a task where traditional methods often fall short.\n\n[S2] MaO consistently outperforms state-of-the-art instance retrieval methods and foundation models across various settings, showing robust improvements for small object retrieval.\n\n[S3] The framework's ability to integrate with different vision backbones and perform well in both zero-shot and fine-tuned scenarios highlights its adaptability and potential for broad applicability.\n\n\n\n[W1] While highly effective for small objects, the paper's explicit analysis of MaO's performance on large objects or general image content-based retrieval is limited.\n\n[W2] MaO's \"in-the-wild\" performance depends on the accuracy of the OVD. While the method's refinement step mitigates some OVD imperfections, significant failures in object detection or localization by the OVD could impact retrieval performance, particularly in highly cluttered scenes.\n\n[W3] Although LeGrad is a key component, the main paper could benefit from a more in-depth explanation of how explainability maps are precisely computed and optimized in the context of multiple objects contributing to a single global token.", "was_revised": true, "final_justification": "Thank you to the reviewers for their time and effort. I choose keep my rating.", "rebuttal": "Thank you for your valuable feedback and positive assessment, recognizing our approach as an **“innovative solution, highlights adaptability and potential for broad applicability”** and noting it **“consistently outperforms SoTA.”** \n\n## General Concerns:  \n### **“The MaO pipeline is critically dependent on an upstream OVD.**  \n  \nThis raised concern is an inherent limitation of any modular, detection-first retrieval pipeline, and we appreciate the opportunity to address it in more depth.  \n\n\n1. Dependence on an OVD is a standard practice across several domains.  \nOur design follows a well-established paradigm that is widely adopted in various methods e.g., Grounded-SAM (combining Grounding DINO OVD), Open3DIS (CVPR 2024) that performs open-vocabulary 3D instance segmentation, and TROY-VIS (WACV 2025) that uses an existing OVD for video instance tracking.  \nThus, the use of an OVD as a building block is not unique to our method, but consistent with many best practices in the field. Novel OVD methods can operate even under extreme object size and clutter conditions (see table below), and we leverage this in our approach.   \n\n\n2. No need for object classification, just object proposals  \nWe use OVD in the “objectness” mode (L207-208) where the model is used as an object proposal and needs to binary discriminate an image area as object or background.\n  \n3. In practice, the impact is limited and measurable.  \nTo further address this concern (that was discussed in Section 5), we conducted an empirical evaluation of our used OVD model on two GT-annotated benchmarks, and reported in our  Suppl. Material (SM) in Section F (as referred in L350 in the main paper). The results show that even under imperfect detection (partial object detection) our method continues to outperform existing approaches.  \nIn summary, we believe our method effectively harnesses the strengths an OVD suggests for image retrieval, while addressing and mitigating its inherent limitations.  \n \n| Object-to-image area Ratio (%) | 0.2 | 0.4 | 0.8 | 1.0 | 1.5 | 2.0 | \n|-|-|-|-|-|-|-| \n| Object Recall               \t| 0.68 | 0.83 | 0.83 | 0.85 | 0.90 | 0.87 | \n \nWhile there is a performance degradation in the extreme case of 0.2% the results show the high capability of owlv2 in discriminating even small objects.  \n  \n### **Discussion on failure cases**  \nThank you. Beyond our discussion on limitations in Section 5 and Sections F and H in the Suppl. Material (see pointer to this section in L335-336, including OVD and clutter), we further noticed that our model is more biased toward objects and has shortcomings on faces, individuals, and landmarks. We will extend and add visual examples (omitted here due to formatting constraints) in the final version.  \n\n## Individual Concerns:  \n### **[W1,Q1] Mao Performance on large and general image content based retrieval** \nThank you for this suggestion. In the SM (Section C), we report results on ILIAS dataset, which includes a mix of small, medium, and large objects. To further address this concern, we conducted an additional evaluation on PODS dataset [1], which has an average object-to-image area-ratio of 20%. As shown in the table below, MaO performs robustly on these larger instances, with no degradation in performance.  \nResults (with finetuned models) on non-small objects, PODS Dataset: \n| Method| mAP|\n|-|-| \n| CLIP| 0.341 | \n| DINOv2| 0.332 | \n| MaO-CLIP| 0.368 | \n| MaO-DINOv2| 0.393 | \n \n[1] Sundaram et al,, Personalized representation from personalized \nGeneration, https://arxiv.org/abs/2412.16156 \n \n### **[W2]: Performance of the OVD in cluttered scenes** \n \nPlease see our answer on OVD limitations in the general response. \nMore specifically, we provide a breakdown of the used OVD (on VoxDet) according to several clutter levels. The OVD maintains strong performance even in extreme cases where 12-24 objects are detected in the image. \n | # Objects | 12   | 16   | 20   | 24   |\n|-|-|-|-|-|\n| Recall    | 0.80 | 0.95 | 0.95 | 0.89 |\n\n\n### **[Q3,W3] More in-depth explanation on optimization** \nThank you for the suggestion. We have provided a preliminary explanation of LeGRAD and the explainability map in Section A of SM. Essentially, we optimize a single vector with respect to K different image crops (K objects), while backpropagating gradients only through this single (unified) vector (as illustrated in Fig.2). This process ensures that the final vector effectively represents all objects present We will expand this discussion in the final version, either in the main paper or the Supplement. \n \n \n### **[Q4] MaO specific advantage over $\\alpha$-CLIP**  \nTo highlight the specific advantages of MaO over $\\alpha$-Clip, which also employs mask-based representations, we conducted a focused experiment. We selected 200 multi-object images from the VoxDet dataset and computed their representations using both $\\alpha$-CLIP (with masks over multiple objects) and MaO-CLIP. We measured the difference in distance between each object’s representation and two reference sets: (i) its corresponding source image containing the object, and (ii) a set of negative images that do not contain the object (higher values indicate better performance). In the MaO-CLIP space, this measure was on average twice as large as in $\\alpha$-CLIP, indicating that MaO yields more distinct and better-separated embeddings. \nThis implies superior aggregation and encoding, in our MaO approach as opposed to $\\alpha$-CLIP, which was primarily designed for single, often large, objects. \n \n### **[Q5] Total image encoding and inference time** \nPlease note that we provide a runtime analysis in Section D of the Suppl Material. \nIn response to the reviewer’s concern, we distinguish between two stages of computation: \n(1) Offline encoding time, which includes OVD detection and feature extraction for the gallery images, \n(2) Online query encoding and global search time, which covers real-time inference from a query image. \nAt inference time, defined as the duration from receiving a query image to producing the ranked retrieval results, the process takes 30 ms and 0.03 ms for vectorized search and ranking over 1,581 gallery images (VoxDet). We employ FAISS for efficient vector-based retrieval. \nThe OVD component, used only during gallery preprocessing (offline), adds approximately 0.5 seconds per image. This step does not affect online latency and can be significantly reduced with batching or optimized implementations such as ONNX, though this was beyond the scope of our current work. It is worth noting that the entire encoding pipeline, including OVD, is highly parallelizable due to its data-parallel nature."}, {"review_id": "OuRBdGVgfo", "reviewer": "Reviewer_Ey8L", "summary": "This paper presents a two-stage framework for small object image retrieval. The motivation is to retrieve images containing a specific, often tiny, object within cluttered scenes using a single global descriptor. In first stage, it learns a representation for multiple objects in the image by cropping and encoding objects separately and aggregating them via average pooling to align with object queries. The second stage is a post-training phase where learnt representations are enhanced via multi-object attention mechanism. Author introduces new benchmarks focused on small-object retrievals  by deriving new datasets (Instre-XS). Experiments on the  variety of datasets show improved performance over the counterpart methods", "questions": "I have listed all my concerns in the \"Strength and Weaknesses section\". Please refer to \"Weaknesses\" for the raised concerns and questions.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1)\tThe paper focuses on a rather unexplored area in the image retrieval domain, retrieval of very small objects (down to 0.5% of image area) in cluttered  scenes.\n\n2)\tTwo-stage framework , multi-object fine-tuning and multi-object attention mechanism, provides a systematic approach to address the problem.\n\n3)\tThe release of new benchmarks dedicated for small object retrieval, which can be used for future studies.\n\n4)\tExtensive experiments showing the improved performance over the counterpart methods\n\nWeaknesses\n\n1)\tThe performance heavily depends on the  open vocabulary detector. It is not clear how the model would handle rare or out of domain objects.\n\n2)\tIn stage 1, it is unclear how objects are retrieved. How do you prompt the model, and how do you ensure small and rare objects, i.e. the object of interests, are correctly cropped and encoded.\n\n3)\tIn stage 1, it seems all type of objects are cropped and encoded. The small objects or rare objects are generally underrepresented. How can proposed approach avoid the bias towards frequent/large objects? \n\n4) In second stage, why not fine-tune vision encoder to learn better representations for smaller objects?\n\n5)\tHow does the proposed method compares with more advanced approaches such as Qwen (or Ferret) models that are capable of referring expression and grounding, and expected to generalize over rare/small objects. \n\n6)\tNo information on computational complexity of proposed method compared to counterpart methods. \n\n7)\tWhat are the failure cases. A deeper analysis of failure cases would enable the readers to understand the limitations.", "was_revised": true, "final_justification": "I would like to thank the authors for their responses. I have read reviews and responses. I have increased my rating to borderline accept. I also agree with the other reviewers on the amount of novelty part. Additional novel contributions on reducing the reliance on OVD and improving computational efficiency would make this paper stronger.", "rebuttal": "Thank you for your valuable review and for recognizing that our work demonstrates **“improved performance”**, and **“provides a systematic approach”** with **“extensive experiments”** addressing an **“unexplored area”**, as well as acknowledging the contribution of a “benchmark that can be used for future studies.” \n\n## General Concerns:  \n### **“The MaO pipeline is critically dependent on an upstream OVD.**  \n  \nThis raised concern is an inherent limitation of any modular, detection-first retrieval pipeline, and we appreciate the opportunity to address it in more depth.  \n\n\n1. Dependence on an OVD is a standard practice across several domains.  \nOur design follows a well-established paradigm that is widely adopted in various methods e.g., Grounded-SAM (combining Grounding DINO OVD), Open3DIS (CVPR 2024) that performs open-vocabulary 3D instance segmentation, and TROY-VIS (WACV 2025) that uses an existing OVD for video instance tracking.  \nThus, the use of an OVD as a building block is not unique to our method, but consistent with many best practices in the field. Novel OVD methods can operate even under extreme object size and clutter conditions (see table below), and we leverage this in our approach.   \n\n\n2. No need for object classification, just object proposals  \nWe use OVD in the “objectness” mode (L207-208) where the model is used as an object proposal and needs to binary discriminate an image area as object or background.\n  \n3. In practice, the impact is limited and measurable.  \nTo further address this concern (that was discussed in Section 5), we conducted an empirical evaluation of our used OVD model on two GT-annotated benchmarks, and reported in our  Suppl. Material (SM) in Section F (as referred in L350 in the main paper). The results show that even under imperfect detection (partial object detection) our method continues to outperform existing approaches.  \nIn summary, we believe our method effectively harnesses the strengths an OVD suggests for image retrieval, while addressing and mitigating its inherent limitations.  \n \n| Object-to-image area Ratio (%) | 0.2 | 0.4 | 0.8 | 1.0 | 1.5 | 2.0 | \n|-|-|-|-|-|-|-| \n| Object Recall               \t| 0.68 | 0.83 | 0.83 | 0.85 | 0.90 | 0.87 | \n \nWhile there is a performance degradation in the extreme case of 0.2% the results show the high capability of owlv2 in discriminating even small objects.   \n  \n### **Discussion on failure cases**  \nThank you. Beyond our discussion on limitations in Section 5 and Sections F and H in the Suppl. Material (see pointer to this section in L335-336, including OVD and clutter), we further noticed that our model is more biased toward objects and has shortcomings on faces, individuals, and landmarks. We will extend and add visual examples (omitted here due to formatting constraints) in the final version.\n\n\n## Individual Concerns:\n### **[W1] OVD on rare objects** \nThank you for the comment. Please see our general response to this concern.  \nWe would like to clarify that in our retrieval task, the goal is not to classify the object’s category, but rather to obtain a discriminative representation suitable for similarity-based retrieval. This focus on representation, rather than recognition, helps mitigate challenges associated with rare or unseen objects. The OVD is used as object proposal (objectness) \nIn fact, VoxDet dataset includes a wide range of fine-grained, long-tail, and open-vocabulary categories covering workshop and household hand tools, including cordless drill or Sextant (Nautical angle measuring instrument). Illustrative examples of such rare or hard-to-name categories, used by the benchmark, are shown in the top row of Figure 4. \n \n### **[W2] In stage 1, ... How do you prompt the model …. Ensuring small objects are correctly cropped ..** \nDuring Stage-A training, we divide the dataset into query images and gallery images. Importantly, our method is entirely image-based, it does not use any text prompts in the task. \nSince our training data includes ground-truth bounding box annotations, we crop both the query and gallery images using these provided boxes (see L150–151). During gallery image encoding, we rely on an OVD to automatically generate bounding boxes for general objects in the gallery for each image (see L208).   \nIn gallery encoding stage (where there is no local annotation), we utilize OWLv2 as our OVD, to detect all objects in each image that then pass through MaO pipeline for encoding and aggregation to a single representation. For query images, which contain only a single object, the image is encoded and refined directly without the need for aggregation (see L173). We will rephrase to make this workflow clearer in the final version. \n\n\n### **[W3] In stage 1, ... Small/Rare objects under-represented** \nThank you for raising this important point. Our MaO approach specifically addresses the under-representation of small or rare objects, which is a known limitation of standard global representation methods. MaO improves their treatment in two key ways: \nObject cropping enhances effective “resolution”: By cropping individual objects, small objects are effectively assigned more input tokens in the ViT encoder (due to the fixed patching grid), leading to finer and more detailed representations. This advantage arises naturally from our detection-based design. \nBalanced aggregation: During feature pooling, all objects, regardless of size, contribute more uniformly weighted in the final representation.  \nThis mitigates the typical dominance of large objects and ensures that smaller ones (regardless of their rareness) are not overshadowed (see L59–61). \n \n### **[W4] Why not fine-tune the vision encoder** \nThank you for the suggestion. Our approach prioritizes obtaining a strong initial representation via average pooling, followed by refinement through object-mask-guided optimization, which is particularly beneficial for significance of small object representation. \nIn our current design, Stage-B refines the global vector \\hat{v}_c, based on a fixed initialization (with v_c) from Stage-A (as explained in L146). Allowing v_c​ (and the encoder) to be updated during this process would require a more complex optimization scheme, potentially involving alternate-optimization or EMA-based strategies, which would significantly increase computational overhead and complexity (maybe convergence issues) during training. We believe it represents an interesting direction for future work. \n \n \n### **[W5] Comparing with …. Qwen (or Ferret)** \nThank you for the suggestion. Current multimodal large language models (MLLMs) such as Qwen or Ferret are not well-suited for large-scale image retrieval. Their inference process typically relies on pairwise comparisons between the query image and each gallery image (e.g., prompting whether the object exists in a given image), which requires passing both images through the full MLLM pipeline. This makes the process computationally expensive and impractical for large-scale applications. \nHowever, in response to the reviewer’s comment, we also conducted a test using Qwen2.5-VL over 100 queries on a subset of the VoxDet dataset with a gallery of 200 images. \n \n| Method     \t| Matching Type | Accuracy | \n|-|-|-| \n| Qwen2.5-VL \t| Pair Matching  | 19%  \t| \n| MAO-DINOv2 \t| Single Vector Retrieval | 85%  \t| \n \nThese results show that, beyond its impractical runtime, an MLLM like Qwen struggles with small-object retrieval, highlighting the advantages of dedicated retrieval models like MaO. \n \n### **[W6] Computational complexity of the proposed method** \nWe provide a detailed runtime analysis in Section D of the Supplementary Material, reporting both encoding and search times. Our method is compared against two representative SoTA methods: SuperGlobal (SG) for global matching, and AMES, which serves as an efficient local matching approach. This comparison highlights the efficiency and scalability of our method relative to these alternatives. \nAt test time (i.e., from receiving a query image to producing ranked retrieval results), MaO requires 30 ms for query encoding (single object) and 0.03 ms for vectorized search (global matching) over 1,581 gallery images (VoxDet). In comparison: \nSG takes 65 ms for query encoding and 0.37 ms for search with global matching and re-ranking. \nAMES takes 65 ms for encoding and 2,286 ms retrieval due to the heavy local matching process. \nFor offline encoding, both SG and AMES require 65 ms per image, while MaO takes 174 ms per image, in addition to ~0.5 seconds per image for the OVD step. However, this OVD cost can be significantly reduced through batching or optimized implementations (e.g., using ONNX), although such improvements were beyond the scope of this work. Notably, the entire MaO encoding pipeline, including the OVD step, is highly parallelizable due to its data-parallel nature, offering further opportunities for encoding speed-ups. \n \n### **[W7] Failure cases** \nPlease see our general response"}, {"review_id": "TH2V0kd5CO", "reviewer": "Reviewer_1h7Q", "summary": "This paper tackles the problem of Small Object Image Retrieval (SoIR), which involves retrieving images containing specific small objects within cluttered scenes. Existing methods struggle to represent multiple small objects effectively using a single image descriptor. To address this, the authors propose Multi-object Attention Optimization (MaO)—a new retrieval framework that includes multi-object pre-training and attention-based feature refinement using object masks. MaO produces a unified image descriptor that improves retrieval accuracy, especially in zero-shot and lightweight fine-tuning settings.", "questions": "Please refer to the weakness section.", "limitations": "These items are discussed in the paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n•\tOne contribution of this paper is the introduction of new benchmarks (subsets of INSTRE and a retrieval version of VoxDet) that specifically target the weaknesses of current methods concerning small and cluttered objects. \n\n•\tThe paper presents experiments that demonstrate the superiority of MaO over a wide range of baselines, including both classic image retrieval methods and modern foundation models\n\nWeaknesses\n\n•\tThe MaO pipeline is critically dependent on the performance of an upstream open-vocabulary object detector (OVD) to first identify objects in the gallery images. The paper acknowledges that the method's performance can be affected by imperfect detector recall. This is a significant limitation, as the method would likely fail if the target object is missed by the detector. A more thorough analysis of this failure mode and its impact on performance would strengthen the paper.\n\n•\tThe Multi-Object Attention Optimization (Stage B) is an iterative process that must be performed for every image in the gallery to generate its final descriptor. The paper notes this is an offline process, but at 0.03 seconds per object and with 15 objects per image on average for VoxDet, the overhead is substantial (0.5 seconds per image). This is significantly more expensive than the single forward pass required by most baselines and may pose scalability challenges for very large galleries.\n\n•\tOverall, the novelty of the proposed method appears to be incremental. The approach of using an object detector to first isolate regions of interest is an established pattern in image retrieval. More significantly, the core technical mechanism in Stage B, which involves refining a representation by optimizing its attention-based explainability map against an object mask, is heavily inspired by recent work like LeGrad and MaskInversion (regularization term), as the authors acknowledge.\n\n•\tDuring the multi-object fine-tuning (Stage A), the representation for a gallery image is created by simple average pooling of all its detected object features. In a cluttered scene with many distractor objects, this naive averaging could dilute the features of the actual target instance. The paper would benefit from a justification for this choice or an ablation study comparing it to other aggregation mechanisms (e.g., attention-based pooling).\n\n•\tThe paper's core premise is the superiority of its single-vector representation for scalability. To better contextualize this, it would be highly informative to include a comparison against a multi-vector baseline (e.g., where each object crop is a separate vector in the index). This would help quantify the accuracy-efficiency trade-off and demonstrate how close MaO's optimized single vector comes to the performance of a less efficient but potentially more powerful representation.", "was_revised": true, "final_justification": "The rebuttal addressed some of my concerns with clarification and additional ablation, such as other attention aggregation schemes. However, I am still not convinced by the incremental novelty. I would like to keep my rating.", "rebuttal": "Thank you for your valuable review and recognizing **“the superiority of MaO”**, and the contribution of **“new benchmarks”**.\n\n## General Concerns:  \n### **“The MaO pipeline is critically dependent on an upstream OVD.**  \n  \nThis raised concern is an inherent limitation of any modular, detection-first retrieval pipeline, and we appreciate the opportunity to address it in more depth.  \n\n\n1. Dependence on an OVD is a standard practice across several domains.  \nOur design follows a well-established paradigm that is widely adopted in various methods e.g., Grounded-SAM (combining Grounding DINO OVD), Open3DIS (CVPR 2024) that performs open-vocabulary 3D instance segmentation, and TROY-VIS (WACV 2025) that uses an existing OVD for video instance tracking.  \nThus, the use of an OVD as a building block is not unique to our method, but consistent with many best practices in the field. Novel OVD methods can operate even under extreme object size and clutter conditions (see table below), and we leverage this in our approach.   \n\n\n2. No need for object classification, just object proposals  \nWe use OVD in the “objectness” mode (L207-208) where the model is used as an object proposal and needs to binary discriminate an image area as object or background.\n  \n3. In practice, the impact is limited and measurable.  \nTo further address this concern (that was discussed in Section 5), we conducted an empirical evaluation of our used OVD model on two GT-annotated benchmarks, and reported in our  Suppl. Material (SM) in Section F (as referred in L350 in the main paper). The results show that even under imperfect detection (partial object detection) our method continues to outperform existing approaches.  \nIn summary, we believe our method effectively harnesses the strengths an OVD suggests for image retrieval, while addressing and mitigating its inherent limitations.  \n \n| Object-to-image area ratio (%) | 0.2 | 0.4 | 0.8 | 1.0 | 1.5 | 2.0 | \n|-|-|-|-|-|-|-| \n| Object Recall               \t| 0.68 | 0.83 | 0.83 | 0.85 | 0.90 | 0.87 | \n \nWhile there is a performance degradation in the extreme case of 0.2% the results show the high capability of owlv2 in discriminating even small objects.   \n  \n### **Discussion on failure cases**  \nThank you. Beyond our discussion on limitations in Section 5 and Sections F and H in the Suppl. Material (see pointer to this section in L335-336, including OVD and clutter), we further noticed that our model is more biased toward objects and has shortcomings on faces, individuals, and landmarks. We will extend and add visual examples (omitted here due to formatting constraints) in the final version. \n\n## Individual Concerns:\n \n### **[W2]  Runtime of MaO iterative process**\nThank you for this comment. Most state-of-the-art retrieval methods adopt either a two-stage design with re-ranking (e.g., SuperGlobal) or rely on computationally intensive local matching (e.g., AMES), making them significantly less efficient during online search. \nWe present a detailed runtime analysis in Section D of the Supplementary Material, comparing our method against SuperGlobal (SG) and AMES. \nFor offline encoding, both SG and AMES require 65 ms per image, while MaO requires 174 ms per image, plus an additional 0.5 seconds per image for the OVD step. However, this OVD overhead can be substantially reduced through batch processing or optimized implementations such as ONNX. It’s worth noting that the entire MaO encoding pipeline is highly data-parallelizable, offering further potential for acceleration. \nThe online test-time advantage of MaO is clear: it requires only 30.03 ms per query, compared to 80.4 ms for SG and 2,286 ms for AMES (evaluated on VoxDet with 1,581 gallery images). While there is an encoding-time trade-off, we believe it can be effectively addressed through code-level optimization and parallelization. \n\n### **[W3] The approach of using an object detector to first isolate regions of interest …** \nWhile the use of object detectors to isolate regions of interest is not new, our contribution lies in how this component is integrated into our new complete retrieval framework, effectively demonstrating its strength in retrieving small objects within cluttered scenes. \nOur contribution stems from the insight that, for retrieving small objects in cluttered scenes, isolating individual objects significantly improves their representation, leading to more balanced and effective aggregation. \n \n### **[W3] … the core technical mechanism in Stage B …** \nOur new aggregation model leveraging Mask-Inversion in fact allows handling multiple objects in the image.  This addresses a critical gap in prior Mask Inversion methods, which neither target retrieval nor support multiple-object encoding. Our experiments demonstrate that this component is especially beneficial for small object retrieval, an area often under-explored by existing techniques. In our solution, we optimize a single vector against K object-level representations, backpropagating gradients only through the aggregated vector (see Fig.2). \nAn additional novelty, we believe, is the application of Stage-B refinement on a customized multi-object encoder trained in Stage-A, making the two stages tightly integrated and complementary. \n \nAs detailed in our contributions, our work presents several distinct and non-trivial components across methodology, benchmarking, and analysis. \n \n### **[W4] Comparison to Other Aggregation Mechanisms (e.g., Attention-based Pooling)** \nThank you for the valuable suggestion. While our use of average pooling is intentionally simple and non-parametric, to enable zero-shot applicability (as shown in the paper), it ensures broad and simple compatibility with different backbones, and highlights the strength of our MaO approach even with minimal aggregation complexity. \nThat said, we agree that more advanced aggregation methods may further enhance performance. In response to the reviewer’s suggestion, we evaluated our Stage-A model using a single-layer attention-based pooling mechanism. The results show a performance improvement, indicating that our framework is flexible and can effectively incorporate alternative aggregation strategies. \n\n**map on VoxDet:**\n| Aggregation Method      | mAP (%) |\n|-|-|\n| Average Pooling         | 75.04   |\n| Attention-based Pooling | 78.17   |\n\n\nWe appreciate the reviewer’s input. Exploring more advanced pooling techniques is a promising direction for future work. \n \n### **[W5] Multivector baseline ….** \nThank you. This multi-vector baseline demonstrates a slight performance gain over our MaO single-vector representation, but comes at a substantial cost. For VoxDet, a nearly 6× increase in gallery size and memory usage during search (please see discussion in L22-24). While this strategy may be useful as a potential re-ranking stage in future work, it would introduce additional search latency. \n\nResults on VoxDet with DINO (Fine-Tuned)\n| Method                \t| Avg. Vectors/Image | mAP (%) | \n|-|-|-| \n| DINO (Multi-Vector)   \t| 5.76           \t| 85.40   | \n| MaO-DINO (Single-Vector)  | 1.00           \t| 83.70   |"}]

Author rebuttal:

Meta-review: This paper received initially 1 Borderline Accept and 2 Borderline Reject scores. While reviewers praise the interesting/important problem in a relatively unexplored area, new benchmarks and extensive experiments, there were concerns around limited novelty and the technique’s dependence on an open-vocabulary detector. The rebuttal was regarded as helpful by all reviewers, clarified most of the concerns, leading one reviewer to upgrade their rating from Borderline Reject to Borderline Accept. All ratings in the end are borderline, but leaning positive. The main remaining concern is the somewhat limited novelty of the paper. However, the authors discuss this in the rebuttal, and the AC considers it in the end as satisfactory, especially taking into account that this is an area not very actively explored and this work can help push forward this important problem. After careful consideration, the AC is supportive of accepting the paper into NeurIPS, given all its positive aspects and the extensive additional information provided in the rebuttal. We require that the clarifications and all relevant information provided in the rebuttal be incorporated in the camera-ready version.
Decision: Accept (poster)