Title: Find your Needle: Small Object Image Retrieval via Multi-Object Attention Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Retrieval

Instance Retrieval

Personalization

Feature Aggregation

Feature Optimization
Abstract: We address the challenge of Small Object Image Retrieval (SoIR), where the goal is to retrieve images containing a specific small object, in a cluttered scene. The key challenge in this setting is constructing a single image descriptor, for scalable and efficient search, that effectively represents all objects in the image. In this paper, we first analyze the limitations of existing methods on this challenging task and then introduce new benchmarks to support SoIR evaluation. Next, we introduce \ours (\oursMI), a novel retrieval framework  which incorporates a dedicated multi-object pre-training phase. This is followed by a refinement process that leverages attention-based feature extraction with object masks, integrating them into a single unified image descriptor. Our \oursMI approach significantly outperforms existing retrieval methods and strong baselines, achieving notable improvements in both zero-shot and lightweight multi-object fine-tuning. We hope this work will pave the way and inspire further research to enhance retrieval performance for this highly practical task.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of Small Object Image Retrieval (SoIR), which involves retrieving images containing specific small objects within cluttered scenes. Existing methods struggle to represent multiple small objects effectively using a single image descriptor. To address this, the authors propose Multi-object Attention Optimization (MaO)—a new retrieval framework that includes multi-object pre-training and attention-based feature refinement using object masks. MaO produces a unified image descriptor that improves retrieval accuracy, especially in zero-shot and lightweight fine-tuning settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n•\tOne contribution of this paper is the introduction of new benchmarks (subsets of INSTRE and a retrieval version of VoxDet) that specifically target the weaknesses of current methods concerning small and cluttered objects. \n\n•\tThe paper presents experiments that demonstrate the superiority of MaO over a wide range of baselines, including both classic image retrieval methods and modern foundation models\n\nWeaknesses\n\n•\tThe MaO pipeline is critically dependent on the performance of an upstream open-vocabulary object detector (OVD) to first identify objects in the gallery images. The paper acknowledges that the method's performance can be affected by imperfect detector recall. This is a significant limitation, as the method would likely fail if the target object is missed by the detector. A more thorough analysis of this failure mode and its impact on performance would strengthen the paper.\n\n•\tThe Multi-Object Attention Optimization (Stage B) is an iterative process that must be performed for every image in the gallery to generate its final descriptor. The paper notes this is an offline process, but at 0.03 seconds per object and with 15 objects per image on average for VoxDet, the overhead is substantial (0.5 seconds per image). This is significantly more expensive than the single forward pass required by most baselines and may pose scalability challenges for very large galleries.\n\n•\tOverall, the novelty of the proposed method appears to be incremental. The approach of using an object detector to first isolate regions of interest is an established pattern in image retrieval. More significantly, the core technical mechanism in Stage B, which involves refining a representation by optimizing its attention-based explainability map against an object mask, is heavily inspired by recent work like LeGrad and MaskInversion (regularization term), as the authors acknowledge.\n\n•\tDuring the multi-object fine-tuning (Stage A), the representation for a gallery image is created by simple average pooling of all its detected object features. In a cluttered scene with many distractor objects, this naive averaging could dilute the features of the actual target instance. The paper would benefit from a justification for this choice or an ablation study comparing it to other aggregation mechanisms (e.g., attention-based pooling).\n\n•\tThe paper's core premise is the superiority of its single-vector representation for scalability. To better contextualize this, it would be highly informative to include a comparison against a multi-vector baseline (e.g., where each object crop is a separate vector in the index). This would help quantify the accuracy-efficiency trade-off and demonstrate how close MaO's optimized single vector comes to the performance of a less efficient but potentially more powerful representation.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nThese items are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a two-stage framework for small object image retrieval. The motivation is to retrieve images containing a specific, often tiny, object within cluttered scenes using a single global descriptor. In first stage, it learns a representation for multiple objects in the image by cropping and encoding objects separately and aggregating them via average pooling to align with object queries. The second stage is a post-training phase where learnt representations are enhanced via multi-object attention mechanism. Author introduces new benchmarks focused on small-object retrievals  by deriving new datasets (Instre-XS). Experiments on the  variety of datasets show improved performance over the counterpart methods\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1)\tThe paper focuses on a rather unexplored area in the image retrieval domain, retrieval of very small objects (down to 0.5% of image area) in cluttered  scenes.\n\n2)\tTwo-stage framework , multi-object fine-tuning and multi-object attention mechanism, provides a systematic approach to address the problem.\n\n3)\tThe release of new benchmarks dedicated for small object retrieval, which can be used for future studies.\n\n4)\tExtensive experiments showing the improved performance over the counterpart methods\n\nWeaknesses\n\n1)\tThe performance heavily depends on the  open vocabulary detector. It is not clear how the model would handle rare or out of domain objects.\n\n2)\tIn stage 1, it is unclear how objects are retrieved. How do you prompt the model, and how do you ensure small and rare objects, i.e. the object of interests, are correctly cropped and encoded.\n\n3)\tIn stage 1, it seems all type of objects are cropped and encoded. The small objects or rare objects are generally underrepresented. How can proposed approach avoid the bias towards frequent/large objects? \n\n4) In second stage, why not fine-tune vision encoder to learn better representations for smaller objects?\n\n5)\tHow does the proposed method compares with more advanced approaches such as Qwen (or Ferret) models that are capable of referring expression and grounding, and expected to generalize over rare/small objects. \n\n6)\tNo information on computational complexity of proposed method compared to counterpart methods. \n\n7)\tWhat are the failure cases. A deeper analysis of failure cases would enable the readers to understand the limitations.\n\nQUESTIONS:\nI have listed all my concerns in the \"Strength and Weaknesses section\". Please refer to \"Weaknesses\" for the raised concerns and questions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Multi-object Attention Optimization (MaO), a novel framework for Small Object Image Retrieval (SoIR) in cluttered scenes. MaO proposes a two-stage approach: multi-object fine-tuning followed by attention-based refinement using explainability maps and object masks to create a unified image descriptor. The authors also introduce new benchmarks, featuring smaller objects and higher clutter. Experiments demonstrate MaO's superior performance on these benchmarks compared to existing methods and foundation models.\n\nSTRENGTHS AND WEAKNESSES:\n[S1] MaO offers an innovative solution to the challenging and practical problem of retrieving small objects in cluttered images, a task where traditional methods often fall short.\n\n[S2] MaO consistently outperforms state-of-the-art instance retrieval methods and foundation models across various settings, showing robust improvements for small object retrieval.\n\n[S3] The framework's ability to integrate with different vision backbones and perform well in both zero-shot and fine-tuned scenarios highlights its adaptability and potential for broad applicability.\n\n\n\n[W1] While highly effective for small objects, the paper's explicit analysis of MaO's performance on large objects or general image content-based retrieval is limited.\n\n[W2] MaO's \"in-the-wild\" performance depends on the accuracy of the OVD. While the method's refinement step mitigates some OVD imperfections, significant failures in object detection or localization by the OVD could impact retrieval performance, particularly in highly cluttered scenes.\n\n[W3] Although LeGrad is a key component, the main paper could benefit from a more in-depth explanation of how explainability maps are precisely computed and optimized in the context of multiple objects contributing to a single global token.\n\nQUESTIONS:\n[Q1] Given that the paper primarily focuses on the effectiveness of MaO for small objects, could the authors provide an explicit analysis of MaO's performance on large objects or its applicability and effectiveness for general image content-based retrieval beyond object-centric queries?\n\n[Q2] could the authors elaborate on the potential impact of significant failures in object detection or localization by the OVD on MaO's \"in-the-wild\" retrieval performance? Particularly in highly cluttered scenes where such failures might be more prevalent.\n\n[Q3]  Could the authors provide a more in-depth explanation of how explainability maps are precisely computed and optimized within LeGrad, specifically addressing the mechanism when multiple objects contribute to a single global token?\n\n[Q4] Could the authors provide a more detailed comparison highlighting MaO's specific advantages over  $\\alpha$-CLIP and other mask-based methods in handling multi-instance, cluttered scenes, beyond just the mAP difference?\n\n[Q5] Given that Stage B optimization takes 0.03 seconds per-object, could the authors provide a more detailed explanation of how this contributes to the total image encoding time, and clarify the total online inference time for a single query, including OVD detection?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-17"}