Title: REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders
Venue: NeurIPS 2025
Accepted: yes
Keywords: image representation learning

region-based representations
Abstract: We introduce the Region Encoder Network (REN), a fast and effective model for generating region-based image representations using point prompts. Recent methods combine class-agnostic segmenters (e.g., SAM) with patch-based image encoders (e.g., DINO) to produce compact and effective region representations, but they suffer from high computational cost due to the segmentation step. REN bypasses this bottleneck using a lightweight module that directly generates region tokens, enabling 60x faster token generation with 35x less memory, while also improving token quality. It uses a few cross-attention blocks that take point prompts as queries and features from a patch-based image encoder as keys and values to produce region tokens that correspond to the prompted objects. We train REN with three popular encoders—DINO, DINOv2, and OpenCLIP—and show that it can be extended to other encoders without dedicated training. We evaluate REN on semantic segmentation and retrieval tasks, where it consistently outperforms the original encoders in both performance and compactness, and matches or exceeds SAM-based region methods while being significantly faster. Notably, REN achieves state-of-the-art results on the challenging Ego4D VQ2D benchmark and outperforms proprietary LMMs on Visual Haystacks' single-needle challenge. The code and pretrained models are available at https://github.com/savya08/ren.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Region Encoder Network (REN), a novel approach for generating efficient region-based image representations using point prompts. Traditional methods rely on class-agnostic segmenters (e.g., SAM) combined with patch-based encoders (e.g., DINO), which incur high computational costs due to segmentation. REN bypasses this bottleneck by directly generating region tokens through lightweight cross-attention modules, achieving ​​60× faster token generation​​ and ​​35× reduced memory usage​​ while improving token quality. Trained with DINO, DINOv2, and OpenCLIP encoders, REN demonstrates superior performance on semantic segmentation, retrieval, and challenging benchmarks like Ego4D VQ2D and Visual Haystacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) Clear problem motivation: Highlights inefficiencies of patch-based methods and limitations of SAM-based region tokens.\n\n(2) Well-structured methodology: Step-by-step explanation of REN’s architecture, training objectives (contrastive loss, feature similarity), and inference pipeline (prompting + aggregation).\n\nWeaknesses:\n\n(1) Architectural similarity to existing methods: The cross-attention module resembles DETR and MaskFormer, raising questions about novelty.\n\n(2) Lacks formal proofs or guarantees for REN’s design choices (e.g., why cross-attention works for region tokenization).\n\n(3) While REN reduces SAM’s computational cost, its performance still depends on SAM-generated masks for training and certain tasks (e.g., token aggregation thresholds).\n\n(4) Limited exploration of alternatives: The ablation study (Section 4.3) focuses on prompting strategies but omits deeper comparisons with other attention variants (e.g., multi-head vs. single-head).\n\nQUESTIONS:\nPlease see the Weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the Region Encoder Network (REN), a model that generates region-based image representations via point prompts. It bypasses expensive segmentation steps, using cross-attention to transform patch features into region tokens, achieving 60× faster token generation with 35× less memory. Experiments show REN outperforms patch-based encoders and SAM-based methods in efficiency and performance across tasks like semantic segmentation and retrieval.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well written and easy to follow.\n\n2. The reduction in cost of the proposed method compared to SAM 2 is impressive.\n\n3. The experimental results fully reflect the effectiveness of the proposed method.\n\nWeaknesses:\n\n1. The essence of REN's training is to establish the correlation between points, but the association between points and regions still seems to rely on other models for assistance, which restricts its application scope and makes it impossible to truly achieve real-time performance.  \n\n2. As described in lines 108-111 of Table 1, the RPN utilizes the prior information from SAM 2. I am curious whether this makes the training constitute an implicit distillation of SAM 2, which is why the RPN has performance advantages? Please replace the priors of other models or directly discuss this point.  \n\n3. Compared with DETR and Mask2Former, the RPN method is slightly less innovative, as it merely replaces learnable queries with point prompts. In addition, I am curious whether the region representations obtained by RPN are superior to those obtained by methods such as DETR and Mask2Former?\n\n4. Can the proposed method be directly used for Fine-tuning like [1], and bring performance advantages to fine-grained tasks?\n\n\n[1] UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding. ECCV 2024.\n\nQUESTIONS:\nPlease refer to the 'weakness' part.\n\nLIMITATIONS:\nPlease refer to the 'weakness' part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo alleviate the computational burden of generating region-based representations by existing visual foundation models, this paper designs REN, a novel module that transforms patch-based image features into object-aligned region tokens using point prompts. Apart from the efficiency, REN also improves the quality of the generated region tokens under the supervision of mask generated by SAM2 while equipped with designed contrastive and feature similarity losses. \n\n   Specifically, REN uses a lightweight stack of cross-attention blocks: point prompts attend to patch features from a frozen image encoder, producing tokens for every prompt. During training, REN is optimized with a contrastive loss (grouping tokens from the same object) and a feature-similarity loss (to align tokens with the encoder’s feature space), enabling it to “implicitly segment” objects through attention. At inference, REN can be prompted with a dense grid or superpixel-based points to cover the image.\nBy analyzing runtime and memory occupation, REN behaves more efficiently than other SAM-based methods. Experiments on several downstream tasks prove that REN is capable of extending to other encoders without dedicated training while achieving better performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths of the paper include comprehensive experiments that cover both Processing time comparisons and evaluation on promising downstream tasks, as well as the detailed analysis of strong performance of the proposed method. However, weaknesses are noted in the drawing of the method diagram. Additionally, the proposed method does not consider specific scenes, which means that what kind of scenes can make the best use of the proposed method. \nStrength:\n1. REN replaces an explicit segmentation step with cross-attention from point prompts, a creative alternative to methods that average-pool patch features within segmentation masks. The use of contrastive + feature-similarity losses to train these region tokens is a clever mechanism to ensure tokens remain semantically meaningful and aligned with the backbone features.\n2. Region tokens generated through REN are very compact. The parameters and inference time, as well as the cuda memory occupation all get huge improvement according to the experiments. Besides, in the experiments on downstream tasks, REN consistently improves accuracy, showing its superiority.\n\nWeakness:\n1. Fig. 1 is the core diagram that presents the whole pipeline of the proposed method. However, this paper lacks some crucial components in Fig. 1, such as the mask generated by SAM2. Since the mask is the Supervision signal of the network, readers may feel confused because of its incomplete presentation.\n2. This paper discusses images with few objects in Intro and argues it as a reason which causes the inefficient of current patch tokens generation. Although REN shows its efficiency in such images, it lacks of analysis about images with dense objects.\n\nQUESTIONS:\n1. In Section Intro, “existing SAM-based methods for region token ...., which can remove fine-grained details and .... Current segmentation methods also .....”, how can you make sure that it’s the simple linear aggregation operation that causes the over-representation and no-representation phenomenon?\n2. Fig. 1 can be redrawn to better epitomize the implementation details. The mask generated by SAM2 and the definition process of positive and negative samples should be added.\n3. In Section 4.3, “all ablations are conducted using REN trained with DINOv2 as the image encoder.”, since DINOv2 with registers reports its segmentation improvement, have you ever tried to use DINOv2 with registers as your encoder?\n4. Some descriptions in this paper are confusing. In Abs and Intro, this paper uses “region-based image representations”, “region representations”, “region tokens” and “region-based representations”. Are there any difference among these words? Or some of the words share the same meaning?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* This paper introduces the Region Encoder Network (REN), a model that efficiently generates region-level representations from patch-based image encoders such as DINO, DINOv2, and OpenCLIP.\n* REN directly produces region tokens by leveraging point prompts as queries and applying cross-attention over patch features, achieving a 60× speedup and 35× lower memory usage compared to class-agnostic segmenter (e.g., SAM) based approaches.\n* REN is trained with a combination of contrastive loss and feature similarity loss, which enhances object-level discriminative power while preserving the generalization capability of the backbone.\n* Across various benchmarks (semantic segmentation, visual query localization, and image retrieval), REN consistently outperforms or matches patch-based and SAM-based region representations with higher efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths**\n\n* REN eliminates the primary bottleneck of region token generation by removing the segmentation step and employs a lightweight cross-attention structure, resulting in substantial improvements in speed (up to 60× faster) and memory efficiency (up to 35× lower usage) compared to SAM-based methods.\n\n\n* The method is experimentally validated to work with various patch-based image encoders, including DINO, DINOv2, and OpenCLIP, and can be extended to new encoders without requiring additional training.\n\n\n* REN demonstrates superior performance over patch-based and SAM-based region representation methods on multiple benchmarks, including semantic segmentation, visual query localization (Ego4D VQ2D), and Visual Haystacks.\n\n\n## **Weaknesses**\n\n* While REN is described as \"segmentation-free,\" the training process relies on pseudo-labels generated by SAM, implying an indirect dependency on segmentation and possible interpretation as knowledge distillation from SAM.\n* Although the authors emphasize the importance of preserving alignment with the original encoder via feature similarity loss, the ablation study is limited and lacks quantitative analysis on its impact on downstream tasks.\n* The self-supervised region token learning in REN, based on contrastive learning for object-level consistency, can be viewed as an incremental extension of prior works such as TokenCut and LOST, rather than a fundamentally novel paradigm.\n* As acknowledged by the authors, there exists inherent ambiguity regarding whether a single point prompt represents the entire object or only a part, requiring heuristic adjustment of the aggregation threshold (µ) based on the task.\n  * The point prompting strategy may lead to reduced accuracy in complex scenes containing overlapping objects, where precise region separation is more challenging.\n* As acknowledged by the authors, REN's region segmentation provides lower boundary precision compared to SAM, making it less suitable for applications requiring high-precision interactive segmentation.\n* Minor Typo: Line 303, \"limiting it’s\" → \"limiting its\".\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}