REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders
Authors: Savya Khosla
Sethuraman T V
Barnett Lee
Alex Schwing
Derek Hoiem
Venue: NeurIPS 2025 poster ()
Keywords: image representation learning
region-based representations
Primary area: deep_learning
TL;DR: REN extracts object-centric region tokens from frozen vision features using point prompts—no segmentation needed. It’s 60× faster and 35× lighter than SAM, with strong performance across tasks.

Abstract: We introduce the Region Encoder Network (REN), a fast and effective model for generating region-based image representations using point prompts. Recent methods combine class-agnostic segmenters (e.g., SAM) with patch-based image encoders (e.g., DINO) to produce compact and effective region representations, but they suffer from high computational cost due to the segmentation step. REN bypasses this bottleneck using a lightweight module that directly generates region tokens, enabling 60x faster token generation with 35x less memory, while also improving token quality. It uses a few cross-attention blocks that take point prompts as queries and features from a patch-based image encoder as keys and values to produce region tokens that correspond to the prompted objects. We train REN with three popular encoders—DINO, DINOv2, and OpenCLIP—and show that it can be extended to other encoders without dedicated training. We evaluate REN on semantic segmentation and retrieval tasks, where it consistently outperforms the original encoders in both performance and compactness, and matches or exceeds SAM-based region methods while being significantly faster. Notably, REN achieves state-of-the-art results on the challenging Ego4D VQ2D benchmark and outperforms proprietary LMMs on Visual Haystacks' single-needle challenge. The code and pretrained models are available at https://github.com/savya08/ren.

REN: Fast and Efficient Region Encodings from Patch-Based Image Encoders
Savya Khosla
Sethuraman TV
Barnett Lee
Alexander Schwing
Derek Hoiem
University of Illinois Urbana-Champaign
(savyak2,st34, b129, aschwwng (hoiem) 0i11 moois.dd
Abstract
We introduce the Region Encoder Network (REN), fast and effective model for generating region- based image representations using point prompts. Recent methods combine class-agnostic segmenters (e.g., SAM) with patch-based image encoders (e.g., DINO) to produce compact and effective region representations, but they suffer from high computational cost due to the segmentation step. REN bypasses this bottleneck using a lightweight module that directly generates region tokens, enabling 60x faster token generution with 35x less memory, while also improving token quality. uses a few pross-attention blocks that take point prompts queries and features from a patch-based image encoder as keys and values to produce region tokens that correspond the prompted objects. We train REN with three popular encoders- DINO, DINOv2, and OpenCLIP and show that it can be extended to other encoders without dedicated training. We evaluate REN on seman- tic segmentation and retrieval tasks, where it consistently outperforms the original encoders in both performance and compactness, and matches or exceeds SAM- based region methods while being significantly faster. Notably, REN achieves state-of-the-as results on the challenging Ego41) VQ2D benchmark and outper- forms proprietary LMMs Visual Haystacks' single-needle challenge. The code and pretrained models are available hattps://gitie //githubb com/ savya08/ren
1 Introduction
Do we really need hundreds of patch-based tokens to understand every image? Existing VIT encoders divide images into fixed grid of patch tokens. This approach is inefficient for two main reasons: (I) images with few objects are represented using hundreds of tokens, leading to unnecessarily high memory and compute costs for applications like video understanding: and (2) patch tokens do not align with object boundaries, making tasks like retrieval and segmentation more challenging. A more scalable and effective alternative is a region- based representation, which divides images into object-level tokens and represents simpler images using far fewer tokens.
Recent region- based representations average-pool patch features regions generated by class- agnostic segmentation methods, such as SAM. These approaches improve semantic segmentation and retrieval accuracy, but generating accurate segmentation masks for all objects in an image remains computationally expensive, with high-quality segmentation models requiring up 30x more time and memory than basch image encoders. This significantly limits the practical usability of region-based representations. Furthermore, existing SAM-based methods for region token generation rely on simple linear aggregation of patch features within object masks, which can remove fine- grained details and discard valuable context surrounding objects. Current segmentation models also tend to produce overlapping masks or miss parts of an image, resulting in over-representation of some regions and no representation for others.
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

To address these limitations, we propose the Region Encoder Network (REN). point promptable model that transforms patch-based features into region-based representations without requiring explicit segmentation. The key idea is that image features already contain sufficient information to segment objects. So instead of relying external segmentation masks, train REN to implicitly segment and pool information through attention. Specifically, as shown in Figure I REN uses a lightweight wross-attention module trained with two learning objectives cosiiesstcontttttveeeeeeee learning and feature similarity- to extract high-quality region tokens directly from patch features. Given a point prompt (query), REN applies tross-attention over patch features (keys/values) to produce token representing the object corresponding to the queried point. To generate region tokens for all objects, REN can be prompted with a grid of points followed by a simple token aggregation step to merge tokens that correspond to the same object.
We explore the design choices that make REN effective and apply multiple patch- based image Siooderr-DING [5], DINOv2 [32). and OpenCLIP [17]. Our experiments show that REN consis- tently outperforms the patch-based feature backbone semantic segmentation and object retrieval. Further, it performs comparably to SAM-based approaches, while being more than 60x faster and while using 35x less memory. Our main contributions can be summarized as follows:
We introduce REN, a point promptable model that efficiently transforms patch-based feature maps into semantically meaningful region tokens using a lightweight cross-attention module, eliminating the need for costly segmentation models. We show that while REN can be trained for any patch-basee encoder, we can also extend to other encoders without dedicated training.
We study the design choices that affect the effectiveness and efficiency of transforming patch-based features into region tokens. For instance, we investigate the impact of prompting strategies, how guide attention mechanisms effectively, how to preserve the information encoded in backbone features, and how to move beyond simple pooling operations for feature aggregation as average or max pooling-usee in prior works.
We train REN with multiple image encoders and will release the code and trained models to facilitate future research and applications.
2 Related Works
Patch-Based Image Encoders. The Vision Transformer 99] introduced the paradigm of representing
images as sequences of fixed-size patch tokens processed by Transformers [3). Building on this,
numerous patch-based image encoders have been developed to learn rich visual semantics (5] 32
14 54 21 or align vision with other modalities like text [33 17 50 41 [3). These backbones
power wide range of applications-for example, SAM 22 uses MAE [14] and L.L.VA (25]
builds on CLIP [33] However, they represent each image with hundreds of tokens, regardless of content, leading to high computational overhead. Techniques like token pruning [3 26 49 31 and
merging [28] [2 (7 30 27 aim to reduce token count, but they often trade off performance and
still rely on grid- semantically weak tokens. This motivates exploring beyond patch tokens.
Learning Objectness. The notion of objectness that certain regions in an image correspond to coherent, meaningful entities- inspired many works. Self-supervised methods like LOST [40] and TokenCut [45] aim to segment objects by clustering image patches based on feature affinity. In parallel. supervised models like DETR and MaskFormer [ 7 learn to recognize objects using cross-attention between learnable queries and patch features under dense supervision. More recently,
SAM and its variants 122 35| [55 have demonstrated that sparse prompts (e.g.. points) can effectively guide segmentation, showing that rich object-level information can emerge from simple supervision signals. These works demonstrate that patch-level features can support object-level understanding. Building on this, we propose a lightweight module that extracts generic object representations from frozen patch features without
without complex objectives or extensive supervision.
Region-Based Representations. Region- representations have long history in computer vision, where regions were traditionally defined using low-level cues such as color, texture, and intensity 122 [5] [6 29 (6). However, these early segmentation methods lacked precision. The development of powerful class-agnostic segmenters like SAM [22] has reignited interest in region- based representations (38) These methods use region masks from SAM to average-pool features from patch-based encoders, producing region tokens that outperform patch tokens on tasks like segmentation and retrieval, while significantly reducing token count per image. Follow-up works further demonstrate their effectiveness in instance localization (20), embodied navigation (I,, long-

---

Input Image
Patch-Base Based
REN
Aligned
Features
Tokens
REN Tokens
Image Encoder
K/V
Cross
Linear
Attn
Lcont
Lieat
gi(x, y)
Points
Point Prompts
(a) REN Architecture
Aligned REN Tokens
REN Tokens
View 1
REN
Leat
View 2
REN
Coont
Image
Encoder
SAM
(b) Training Objectives: Contrastive Token Learning and Feature Similarity
Figure 1: Overview of REN. Point prompts interact with patch-based features through cross-attention blocks to produce region tokens. The training objective combines two components: (1) a contrastive loss that aligns region tokens with those generated from an augmented view of the same image. and (2) feature similarity loss that aligns a linear projection of these tokens with average- average-pooled patch features obtained using SAM masks. REN eliminates the need for explicit segmentation inference time while producing efficient and semantically rich region representations. We also show thresholded attention maps for three query points inside the cross-attention block, which show that the model learns aggregate features primarily from the regions marked by the corresponding point prompts.
While these SAM-based methods have revived region-bass based representations, use remains limited due to high computational cost, coarse feature aggregation, and incomplete image coverage. In contrast, REN uses lightweight cross-attention from point prompts to patch features, enabling fast and memory-efficient region token generation. This learned aggregation captures fine- fine-grained, context-aware information and ensures full image coverage with a compact set of region tokens. As result, REN improves both efficiency and performance across tasks while adapting the token count to image complexity.
tail object search 39 and multimodal concept learning 3
REN
Figure D provides overview of the proposed method. REN generates region tokens by cross- attending features from a frozen image encoder with point prompts, where each point prompt represented using 2D sinusoidal position embedding. We use series of four cros-aatenntion blocks. The keys and values for each block are a linear projection of the image encoder features. The queries for the first block are generated using a linear projection of the point prompts. For the subsequent of point prompts and the previous block's 's output. The
blocks, the queries are generated using final block outputs region tokens for the queried points.
3.1 Training via Self-Supervised Learning and Knowledge Retention
We train REN using images from the Ego4D dataset [3], which offers diverse scenes with objects at varying scales and clutter. For each image encoder, single REN model is trained and used across

---

(a) Image pairs showing grid-based hased point prompts (left) and token aggregated points (right).
(b) Image pairs showing SLIC-based point prompts (left) and token aggregated points (right).
Figure 2: Point prompting strategies and token aggregation results. Region tokens corresponding to point prompts within the same-colored area are aggregated, and we show a representative point prompt for each region. Thus, each image can be represented with few dozen tokens instead of the hundreds required by patch- based methods. (Best viewed in color)
all tasks. We preprocess the training dataset to extract segmentation masks using SAM 22 These masks are used to guide the contrastive and feature similarity losses, described below.
Contrastive Token Learning. We use the InfoNCE loss 42] ensure region tokens are consistent across different views the same object. Specifically, we identify regions in the training images using SAM [22] and assign an ID to each region. Then, we generate two augmented views of each image using random horizontal flipping, rotation, cropping, color jitter, and sharpness adjustments. REN is used to extract region tokens from both views. These "REN tokens" corresponding to prompt points that fall inside the same region (i.e., fully contained within the same SAM AAM-generated mask) are treated as positives, while all others are considered negatives. Specifically, we compute
Σ 1 Li =d,= (i)sm
N
Leomt =
log
(I)
N
1=1
[agi] exp(sim(ri,
where denotes the 7h REN token, id; denotes the region ID of the 7th token, N is the number of region tokens in the image pair, 7 is temperature hyperparameter set 0.1, sim(-) represents the cosine similarity function, and [[:] is an indicator function.
Feature Similarity. Training only on the contrastive loss could cause the model to drift away from the original encoder's feature space. This is undesirable, as the original models are well-trained using large and carefully curated datasets, and we would like to retain their generalization capability. To this end, introduce feature similarity objective that encourages alignment between region tokens and the original encoder features. Specifically, for each region, we generate a target token t; by average pooling encoder features within the SAM mask (38), and apply cosine embedding loss against the "aligned REN token" F, obtained via linear projection the REN token ri:
N
t ||t|||||aa|||
Sreat
N
The linear projection is crucial because it allows the contrastive objective to flexibly learn more discriminative region tokens, without being overly constrained by the structure of the original feature space. At the same time, this objective ensures that the generated region representations retain information approximate the average backbone features via simple linear projection.
Overall Loss. REN is trained using the following overall loss: L AcontCcont + Areat We set A Areat 1.0.

---

3.2 Inference via Prompting and Token Aggregation
Inference is straightforward when the target object is specified by the user or the task annotation. E.g.- in object retrieval, a mask is provided for the query object, and REN can be prompted with points on the annotated object mask to get region tokens for the query. However, for tasks requiring region tokens for all objects, parts, and background. REN must be prompted with multiple point prompts across the image, followed by token aggregation to merge tokens that correspond to the same object.
Point Prompting. We present two ways of prompting REN, as illustrated in Figure 2 (1) Grid-based prompting, which uses G G grid of points as prompts; and (2) SLIC-based prompting, where we segment the image into S = G2 superpixels using SLIC [T and use the center of each superpixel as point prompt. Grid-based prompting incurs computational cost but may produce a few noisy tokens from prompts at object boundaries. SLIC-based prompting eliminates ambiguous prompts at the cost of an inexpensive grouping step. We use Fast-SLIC [2 to get SLIC-based prompts.
Token Aggregation. We aggregate tokens by first computing the pairwise cosine similarity between all REN tokens. Based on this similarity, we construct an adjacency matrix where edges are formed for pairs with a similarity greater than μ 0.975. Connected components in this graph are identified, and tokens within each component are average pooled produce a single representative token. When prompts are densely sampled (i.c., with high G x G or S), some points fall on object boundaries, producing noisy region tokens that fail group with others. These outlier tokens can be safely discarded without impacting downstream performance. In particular, when using dense point prompts (e.g., over 1000 points), we discard all groups that contain fewer than three tokens. Figure 2 shows the output after token aggregation, demonstrating that this step significantly reduces the number of tokens required to represent an image.
Variants of Region Representations. As mentioned in Section 3.1 REN produces two sets of token-level representations: (1) REN tokens, which are directly optimized using the contrastive loss (Equation I. and (2) Aligned REN tokens, which are trained to align with the original image features via the feature similarity loss (Equation (2)). Since aligned REN tokens can be derived via linear projection of the REN tokens, the latter contains all the information needed to obtain the former. As result, we use REN tokens for downstream tasks that involve further learning (e.g., semantic segmentation). For learning-fre free setups, the choice depends on the task: REN tokens are better for tasks requiring discriminative representations of the same object (e.g., visual query localization), while aligned REN tokens are preferable for tasks that leverage the properties of the underlying image encoders- for example, CLIP's s language alignment (e.g., visual haystacks) or DINOv2's category-level semantic understanding (e.g., image retrieval).
3.3 Extending Pretrained REN to Other Encoders
We can use pretrained REN- for example, REN trained for DINO (RENDINO) generate region- based representations from any target image encoder without dedicated training. Specifically, given an input image, we use SLIC-based point prompts to generate region tokens from RENDINO. and track which point prompts are grouped together during token aggregation. For each group of points, we get a region mask by taking a union of the corresponding superpixels. Figure 2bb illlutratees examples of such region masks. To extract feature representation for each region, we use the region masks to constrain the global attention the final layer of the target image encoder. Specifically, for each region mask, we identify the set of patch tokens that overlap with the region. Then, we restrict the attention of the CLS or query token (used for global aggregation) only to the patch tokens that fall within the region. This can be efficiently implemented by duplicating the CLS/query token and applying attention masking using the region masks. By doing this for each region, we obtain region tokens that align with the fcature space of the target encoder.
The REN-based approach offers two key advantages over the SAM-based method proposed by
Shlapentokh- Milapentokh-RRohhmnn al. 38 : (1) substantially lower computational cost in both time and memory, and (2) complete coverage of the image, as unlike SAM, the masks produced by REN do not overlap and leave no area unsegmented.
5

---

Table 1: Runtime comparison. With 10x fewer parameters, REN achieves over 60x speedup com- pared to the fastest SAM-based approach, as mea- sured on a single NVIDIA A40 GPU. Evaluations use either 32x32 grid prompts or 1024 SLIC- based prompts. Reported metrics exclude the patch- based image encoder (DINO ViT-B/8: 85.8M pa-
SAM -0- Baichod-SAM EERC - SAM REN
50000
GPU Menoory (MB/ing
24598.5
5000
8035.5
2335.6
3897.2
2581.6
619.5
500
228.3
rameters, 0.011 s/img).
98,8
122.5
Method
Params (M)
Time (s/img)
64
256
1024
4096
16384
SLIC
Grid
Point prompts per hatch
Figure 3: Memory comparison. We measure peak GPU memory used for processing sin- gle image across prompt batch sizes. REN can handle larger batches and it uses substantially
SAM
22
641.1
2.310
Batched-SAa -AMM
641.1
2.048
EfficientViT-SS SAM [5]]
203.3
1.790
SAM 35
224.4
1.898
REN (w/o TokAgg)
20.1
0.020
0.017
less memory-for example, 35 x less memory than the most efficient SAM-based approach for a batch of 1024 prompts (32 x 32 grid).
REN (w/ TokAgg)
20.1
0.033
0.029
4 Experiments
We compare the computational cost of REN and SAM-based methods in Section 4.
evaluate REN
on downstream tasks in Section 4.2 and present ablation studies in Section 4.3
We additionally
discuss alternative approaches explored in Section of the supplementary.
4.1 Compute Requirements
We compare the computational requirements of REN with other SAM-based methods for region token generation. This analysis is conducted single NVIDIA A40 using images from the ADE20K dataset [52]. We exclude the contributions of patch-based encoders from the evaluated metrics, as these models are substitutable and contribute equally across approaches. Additionally, we use SAM ViT-H for this analysis, smaller SAM variants perform worse than the ViT-H-based approach 38 Table T and Figure 3 present the results, which compare three key metrics:
Parameter Count. REN trained for DINO ViT-B/8 has 10.1 x fewer parameters than EfficientViT. SAM (51), the most parameter-efficient SAM-based model.
Processing Time. For a 32x32 grid of point prompts, REN generates region tokens 61.7x faster than EfficientViT-SAM. the fastest SA.M-based approach. Batched-SAM 24 was evaluated using a batch of 8 images-the largest can handle without running out of memory on an A40. For fairness, REN was also benchmarked for batch size of 8, though it can process a much larger batch of images.
Peak GPU Memory. While REN can process dense grids of point prompts simultaneously, the SAM-based methods split prompts into smaller batches and process them separately. On a single A40, SAM, Efficient ViT-SA.. and SAM cannot handle 4096 prompts at once, and Batched-SAM fails beyond this limit. In contrast, REN can process up to 16384 prompts (64x64 grid) at once, while using less memory than what the most efficient SAM-based approach requires for just 64 prompts. For 32x32 grid, REN uses 3 less GPU memory than the most efficient SAM-based method.
4.2 Downstream Tasks
We compare REN's region representations to the original patch-based representations across several tasks. REN consistently outperforms both patch-bacd based baselines and prior region- region-based methods, demonstrating its effectiveness and versatility.
Visual Query I Localization. We evaluate REN on the Ego4D VQ2D benchmark, where the task is localize the last occurrence of a query object long video. Our approach follows a stage-wiss pipeline: (1) Use REN to extract region tokens from multi-resolution crops of video frames and the visual query; (2) cosine similarity select candidate regions in the video that match the query; (3) convert the point prompt of the selected candidates into bounding boxes using SAM 2 [35] and refine the selections by cropping around them; (4) use SAM 2 to track the last candidate as the initial

---

Table 2: Visual query localization on the Ego4D VQ2D benchmark. Our method substantially outperforms existing approaches, including those specifically developed for this task. Baseline re- sults are sourced from the official leaderboard
Method
StAP
tAP
Succ.
Rec.
SiamRCNN 13
0.13
0.21
41.6
34.0
CocoFormer
[8
0.18
0.26
48.1
43.2
VQLoC [8
0.24
0.32
55.9
45.1
Figure 4: Examples of query localization.
HERO- VQL
0.28
0.37
60.7
45.3
PRVQL [0]
0.28
0.37
59.4
45.7
REN effectively localizes target objects long
RELOCATE 20
0.35
0.43
60.1
50.6
videos despite challenges like clutter, occlusions, background blending, motion blur, viewpoint changes, and brief visibility.
RENDINOV2
0.40
0.52
61.2
49.3
Input
DINOv2
REN
Table 3: Semantic segmentation using lin- ear classifier on frozee features. For ref- erence, the absolute state-of-the-art results from Chen al. [6] and Wang et al. 44] are reported the column heading. Results for methods using external segmenters are taken from Shlapentokh Shlapentokh-Rothman al. [38].
VOC2012 (89.0)
ADE20K (63.0)
Method
With External Segmenters
SAMDINOK
83.6
50.2
SAM+SLICorNo-2
86.9
52.9
Direct Encoders
DINOv2
82.1
47.7
RENDINOVE
86.5
50.9
DINO
66.4
31.8
RENDINO
71.4
35.1
Figure 5: Qualitative comparison of semantic segmentation on ADE20K. Region tokens from REN yield cleaner, less noisy predictions com- pared patch-based features from DINOv2.
OpenCLIP
71.4
39.3
RENopecCLIP
78.0
42.8
track; and (5) use REN to generate additional visual queries from the initial track, and repeat the search, refinement, and tracking steps. This pipeline follows the approach proposed by Khosla et al. (20]. They provide detailed explanation of each step. The main difference from (0]: they SAM-based pooling to extract region tokens, while we use REN, and, given a 60x speed-up realized by eliminating SAM, we process multi-resolution crops of video frames while still being faster.
The results are presented Table 2 REN significantly outperforms previous models, achieving a new of-e-oof-the-rt on this challenging task. Notably, the second-bes method, RELOCATE, also uses region-based approach that relies SAM for region token generation. The strong performance of both RELOCATE and REN highlights the benefit of region-based representations for this task. Moreover, REN outperforms RELOCATE by leveraging contrastive learning objective that enables to capture fine aneegrained details of the query object. In contrast, RELOCATE generates region tokens by averaging features within object masks, which can sometimes overlook subtle distinguishing characteristics.
Semantic Segmentation. To evaluate representation quality, we follow the standard practice of training a linear classifier on frozen features to predict semantic class labels for objects in an image. The results are presented in Table For each image encoder F., we compare encoder E processes the input image to produce low-rcsolution feature map. A linear classifier is applied to this feature map to predict class logits, which are upsampled to the original resolution for
setups: (1) F: The
7

---

Table Visual Haystacks' single-needle challenge. Pooling SigLIP features with pretrained RENDINO leads to a substantial performance gain, outperforming proprietary LMMs, open-source LMMs, and RAG-based methods cetenodddsseeillllllllllllee at larger values of N. Results for all methods except REN and SigLIP 2 are from [46]. "E" indicates context overflow, execution failure, or API error.
Method
N=1
N=2
N=3
N=5
N=10
N=20
N=50
N=100
N=500
N=1K
Detector Oracle
90.2
88.8
88.3
86.9
85.4
81.7
77.5
73.9
Gemini 1.5 Pro
88,4
82.0
78.3
76.0
71.9
68.6
62.8
57.4
E
GPT-4o
82.5
79.9
77.5
73.3
68.2
65.4
59.7
55.3
E
LongVILA
63.8
59.0
$7.7
56.7
55.6
52.0
52.0
52.0
E
Qwen2-VL
80.9
76.6
73.6
67.9
62.6
59.1
52.6
E
Phi-3
80.5
62.0
54.8
52.6
50.8
InterVVL2
88,1
80.5
72.3
63.9
58.8
55.2
E
mPLUG-OWL3
84.4
66.0
62.1
57.0
53.2
51.5
E
LLaVA-v1.5 MIRAGE
85.8
77.1
75.8
68.6
63.6
60.4
55.3
55.4
52.9
83.2
77.8
76.6
72.8
70.5
66.0
63.6
62.0
58.7
55.7
SigLIP 2
72.0
69.2
68.1
65.3
64.1
60.3
58.7
58.3
56.6
54.9
RENDIN--SSLIP
81.2
78.6
77.4
76.0
74.0
72.1
68.3
62.3
59.2
Query Image
Some Images Retrieved from the Database
Table 5: One-shot image retrieval. Baseline results taken from
Shlapentokh- Rothman al. (38].
HELL
Method
mAP
mRP@50
DINOv2
0.13
0.33
SAMPINoe 0.45
0.58
Figure 6: Example image retrieval on COCO. Given an image of a query object (left), REN retrieves database images containing the same object.
RENDINOc
0.52
0.65
per-pixel predictions; and (2) REN REN generates region tokens for each point prompt on the input image, and a linear classifier predicts the class label for each token. Predictions are mapped back the image by assigning the label to the patch/superpixel containing the corresponding prompt.
where external models are used to segment regions and linear head predicts labels for each region. Using SAM for segmentation pro- vides these methods with high boundary precision at substantial computational costs (see Section 4.1).
Talo includes results from Hhapentooh-Rothmane
Nevertheless, RENDIN outperforms SAMpINOv2 due to better coverage of image regions.
Finding Needle in a Haystack. We evaluate our approach of extending pretrained REN to other im- age encoders (Section 3.3) on the single-needle challenge from the Visual Haystacks benchmark (46) The task involves answering queries of the form: "For the image with the (anchor object). is there [target object]?" We use RENDINO segment each image into regions and employ SigLIP 2 [41 to generate text-aligned region tokens. Specifically, for each region identified by RENDINO- we extract the corresponding patch features from the final hidden state of SigLIP 2's vision encoder and pool them using its pooling head produce region token. To answer query, we first retrieve the image containing the anchor object by computing cosine similarity between region tokens of each image and the LiP text embedding of "This a photo [anchor object]." The image with the most similar region token is selected. We then check whether this image contains the target object by computing cosine similarity between its region tokens and the text embedding for "This is photo of [target object]." the highest similarity exceeds 0.05, the answer "Yes"; otherwise, "No."
Table shows the results. While SigLIP 2 underperforms relative to the baselines, using RENDINO to pool its features into region tokens boosts the absolute performance by 8.7% average. Furthermore, our approach remains robust as the number of input images (N) increases, and we outperform proprietary LMMs, open-source LMMs, and R RAG-based methods reported in [6] by a substantial margin. This underscores the effectiveness of region-bast based representations large-scale retrieval.
Image Retrieval. We evaluate REN on single-shot object- based image retrieval, where the task is to retrieve images from a database that contains specified query object. Figure illustrates example.

---

55
50.6 51.0 51,0
50.9
Table 6: Effects of prompting technique and token aggregation. SLLI--baee prompting im- proves segmentation results on ADE20K, while image retrieval COCO remains unaffected by the prompting strategy. Token aggregation reduces the average token count per image by 19.5 without compromising performance. We use a similarity threshold of μl = 0.975 for to- ken aggregation, which yields average of 72.1 tokens per image.
50
ADE20K mAP
47.7
38.2
27.2
25
....DINOV2
--REN
20
0
20
40
60
80
NIA
Average tokens per image
Figure 7: Performance vs. token count. REN matches the performance of the original image encoder, which uses 1369 tokens/image, with just 41 tokens/image, and surpasses beyond that point. Performance stabilizes at 70 to- kens/image, NTA denotes no token aggregation, i.e., 1369 tokens/image.
Token Aggregation
ADE20K (mAP)
COCO (mRP)
Prompt
Grid Grid
50.4
0.66
50.4
0.65
SLIC
50.9
0.66
SLIC
/
51.0
0.65
Table 7: Training loss ablation. A combination of contrastive loss and feature similarity loss yields the best performance. The VQ2D performance is evaluated on 500 videos from the validation set.
Contrastive Loss Feature Similarity
VOC2012
VQ2D
77.7
46.9
86.0
49.7
86.5
63.7
Following Shlapentokh-Rotiee et al. 38], we use the COCO validation set as the image database, and 50 images with corresponding object masks serve query instances for each object class, Query features are computed by averaging region tokens generated from 128 point prompts within each query mask. These compared to region features of all database images using cosine similarity, images are ranked by their highest similarity score. We report the mean Average Precision (mAP) and mean Retrieval Precision at 50 (mRP@ 50) in Table Region- methods outperform the patch- patch-based baseline, and REN further surpasses the SAM-based baseline while offering faster and more efficient region token generation.
4.3 Ablations
We discuss the design decisions that lead to effective region encoder network. All ablations conducted using REN trained with DINOv2 as the image encoder.
Effect of Token Aggregation. As described in Section 3.2 the extent of token aggregation in REN can be controlled by the aggregation threshold μr. Setting μ > 1 disables aggregation, while lower values result in more aggressive merging, reducing the token count. Figure shows semantic segmentation performance across different values of ranging from 0.875 to 0.975. The horizontal axis indicates the average number tokens per image, and the vertical axis shows mAP. REN matches the performance of the patch-based encoder using just 41 tokens and surpasses it beyond that point. With μ 0.975, REN achieves 19.5x x reduction in token count without compromising performance. Unlike patch-based encoders, which use a fixed token count (1369 tokens per image in this case), REN provides the flexibility adjust the token count based compute constraints.
Effect of Prompting Strategies. & grid-based and SLIC-based prompting strategies. The SLIC-based approach performs better for semantic segmentation because it results in more precise predictions at object boundaries, as illustrated in Figure 2 For image retrieval, both prompting strategies perform similarly. This is expected because noisy tokens at object boundaries in grid-basce prompting are either attenuated during token aggregation or, if aggregation is not used, contribute minimally the overall similarity scores between the query token and the rest of the image tokens.
Training Loss Ablation. Tubleepresentt the results of loss ablation study, showing that combining the contrastive loss and feature similarity loss yiclds the best performance.

---

5 Conclusion
In this work, we present REN, model for efficiently generating region-based representations of an image. By removing the reliance on segmentation models, REN greatly improves the efficiency of region token extraction, making them more practical for wide range of applications. Beyond efficiency, our learning- -ased approach produces region representations that not only surpass tradi- tional batch-based features but also outperform prior region-based methods. Overall, our findings demonstrate that region tokens provide a compact, content- aware, and semantically rich alternative to patch- basch representations, and REN offers practical and efficient way of generating them.
Srriguity-whetiiee to represent a whole object or just part. This is addressed by placing multiple prompts on the same object and adjusting the aggregation threshold j based on the task: a high je favors full objects, while a low f favors parts. Additionally, region segmentations from REN are less precise than those from SAM, limiting its suitability for interactive segmentation, though they are sufficient for generating region tokens.
Limitations. Using point prompts can introduce
Future Works, Future work can explore using REN's region tokens as inputs to vision- lasion-language or multimodal models, potentially replacing patch-based tokens. Further directions include fine-tuning the image encoder with task objectives and explicitly modeling region relationships to better support complex scene understanding.
Acknowledgements
This work is supported in part by ONR award N00014-23-1-2383, U.S. DARPA ECOLE Program No. #HR001 1 12390060, National Science Foundation grants #2008387, #2045586, #2106825, and NIFA award 2020-67021-32799 We used GPUs at NCSA Delta through allocation CIS240541 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants #2138259, #2138286, #21 38307, #2137603, and #2138296. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, ONR, or the U.S. Government.
References
[1] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. V. Fua, and S. Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. PAMI, 2012.
[2] H. Bao, L.. Dong, and Wei. Beit: Bert pre-training of image transformers. ICLR, 2022.
[3] A. Blume, K. D. Nguyen, Z. Wang. Y. Chen, M. Saappentokh-Roohhaan
X. Jin, J. Kim, Z. Zhu. J. Liu, K.-H. Huang. M. Sidhu, X. Zhang. V. Liu, R. Sinha, T.-L. Wu, A. Zala, E. Stengel-Eskin, D. Yin, Y. Xiao, U. Mall, Z. Yu, K.-W. Chang, C. Cobb, K. Karahalios, L. Chilton, M. Bansal, N. Peng, C. Vondrick, D. Hoiem, and H. Ji. Miracle: An online, explainable multimodal interactive concept learning system. In Proceedings of the 32nd ACM International Conference on Multimedia. ACM, 2024.
[4] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers, 2020.
[5] M. Caron, H. Touvron, I. Misra, H. J'goo, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. ICCV, 2021.
[6] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[7] B. Cheng, L. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. CVPR, 2021.
[8] B. Cheng, A. G. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. In NIPS, 2021.
[9] A. Dosovitskiy, L., Beyer, A. Kolesnikov, D. Weissenbomn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[10] B. Fan, Y. Feng, Y. Tian, Lin, Y. Huang, and Fan. Prvql: Progressive knowledge-guides refinement for robust egocentric visual query localization. ArXiv, 2025.
[11] K. Garg, S. S. Puligilla, S. Kolathaya, M. Krishna, and S. Garg. Revisit anything: Visual place recognition via image segment retrieval, 2024.
10

---

[12] S. Gould, T. Gao, and D. Koller. Region-based segmentation and object detection. In NIPS, 2009.
[13] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan. J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane. T. Do. M. Doulaty, A. Erapaali, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasic, C. González, J. Hillis, X. Huang, Y. Huang, W. Jia. W. Khoo, J. Koláī, S. Kottur, A. Kumar, F. Landini. C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao. M. Vo, Y. Wang. X. Wu, T. Yagi. Z. Zhao, Y. Zhu, P. Arbeláez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, V. Ithapu, C. V. Jawahar,
Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, M. Rehg. Y. Sato, Shi, M. Shou, A. Torralba,
L. Torresani, M. Yan, and Malik. Ego4d: Around the world 3,000 hours egocentric video. In CVPR,
2022.
[14] K. He. X. Chen, S. Xie, Y. Li, P. Doll'ar, ar, and R. B. Girshick. Masked autoencoders scalable vision learners. CVPR, 2021.
[15] D. Hoiem, A. A. Efros, and M. Hebert. Geometric context from a single image. ICCV, 2005.
[16] D. Hoiem, A. A. Efros, and M. Hebert. Recovering surface layout from an image. IJCV, 2007.
[17] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar,
H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, 2021.
[18] H. Jiang, S. K. Ramakrishnan, and K. Grauman. Stngle-stage visual query localization egocentric videos. ArXiv, 2023.
[19] E. Jones. T. Oliphant, and P. Peterson. SciPy: Open source scientific tools for Python, 2001. URL http://wwu. scipy..ppoo..... scipy.org/
[20] S. Khosla, S. T. V, A. Schwing, and D. Hoiem. Relocate: A simple training-free baseline for visual query localization using region- based representations. In CVPR, 2025.
[21] A. Kim. Fast-slic, 2019. URL httpp/////gttaab.oo///gg////a/as
[22] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitchead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. B. Girshick. Segment anything. ICCV, 2023.
[23] Z. Kong, P. Dong, X. Ma, X. Meng, W. Niu, M. Sun, B. Ren, M. Qin, H. Tang, and Y. Wang. Spvit: Enabling faster vision transformers via soft token pruning. In ECCV, 2022.
[24] B.-K. Loe.
full--sement--nythiing
2023.
URL
cot haaabbbooooBnggggnnnn...
Full Tull-Segment-aa -Segmmnt-Anythinng Anything
[25] B. Li, Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, Zhang, Li, Liu, and C. Llava-onevision: Easy visual task transfer. ArXiv, 2024.
[26] Y. Liang, C. Ge, Z. Tong, Y. Song, J. Wang, and P. Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations, 2022.
[27] Y. Liang, C. Ge, Z. Tong. Y. Song. J. Wang, and P. Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. ArXiv, 2022.
[28] J. Liu, B. Liu, H. Zhou, H. Li, and Y. Liu. Tokenmix: Rethinking image mixing for data augmentation in vision transformers, 2023.
[29] T. Malisiewicz and A. A. Efros. Improving spatial support for objects via multiple segmentations, In BMVC, 2007.
[30] D. Marin, J. R. Chang, A. Ranjan, A. K. Prabhu, M. Rastegari, and Tuzel. Token pooling vision transformers. ArXiv, 2021.
[31] L. Meng. H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.-G. Jiang, and S. N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. CVPR, 2022.
[32] M. Oquab. T. Darcet, T. Moutakanni, H. Q. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. Ell-Noby,, M. Assran, N. Ballas, W. Galuba, R. Howes, P.Y. B. Huang, S.-W. Li, I. Misra, M. G. Rabbat, V. Sharma, G. Synnaeve, H. Xu. H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision. TMLR, 2023.
[33] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICMI,, 2021.
[34] Y. Rao. W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification, 2021.
[35] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. K. Ryali, T. Ma. H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. B. Girshick, P. Doll'an ar, and C. Feichtenhofer. Sam 2: Segment anything in images and videos. ArXiv, 2024.

---

[36] B. Russell, W. Freeman, A. Efros. J. Sivic, and A. Zisserman. Using multiple segmentations to discover objects and their extent in image collections. In CVPR, 2006.
[37] S. Ryoo, Piergiovanni, A. Arnab, M. Dehghani, A. Angelova, and Research. Tokenlearner: Adaptive space-time tokenization for videos. In NIPS, 2021.
[38] M. Malapentokh-Rothhnee
A. Blume, Y. Xiao, Y. Wu, S. T. V. H. Tao, Y. Lec, W. Torres, Y.X.. Wang, and D. Hoiem. Region-based representations revisited. In CVPR, 2024.
[39] Sidhu, Chopra, A. Blume, Kim, R. Reddy, and Ji. Search and detect: Training-free long object detection via web-image retrieval, 2024.
[40] O. Siméoni, G. Puy, H. V. Vo, S. Roburin, S. Gidaris, A. Bursue, P. Pérez, R. Marlet, and J. Ponce. Localizing objects with self-supervised transformers and no labels, 2021.
[41] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, 1. Alabdulmohsin, N. Parthasarathy, T. Evans,
I., Beyer, Y. Xia, B. Mustafa, O. Hénaff, J. Harmsen, A. Steiner, and X. Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, arXiv, 2025.
[42] A. van den Oord, Li, and Vinyals. Representation learning with contrastive predictive coding. ArXiv, 2019.
[43] A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention is all you need. In NIPS, 2017.
[44] P. Wang. S. Wang. Lin, S. Bai, X. Zhou, J. Zhou, X. Wang, and C. Zhou. One-peace: Exploring one general representation model toward unlimited modalities. ArXiv, 2023.
[45] Y. Wang. X. Shen, Y. Yuan, Y. Du, M. Li, S. X. Hu, J. L.. Crowley, and D. Vaufreydaz. Tokencut: Segmenting objects in images and videos with sulf-super transformer and normalized cut, 2023.
[46] T.-H. Wu, Biamby, Quenum, R. Gupta,
Gonzalez, T. Darrell, and Chan, Visual haystacks:
vision-centric needle-in- in-a aaahaaattck benchmark. In ICLR, 2025.
[47] J. Xu, S. D. Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. CVPR, 2022.
[48] M. Xu, Y. Li, C.-Y. Fu, B. Ghanem, T. Xiang, and J.-M. Pérez- Rúa. Where is my wallet? modeling object proposal sets for egocentric visual query localization. In CVPR, 2022.
[49] H. Yin, A. Vahdat, J. Alvarez, A. Mallya, J. Kautz, and P. Molchanov. A-vit: Adaptive tokens for efficient vision transformer, 2022.
[50] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. ICCV, 2023.
[51] Z. Zhang, H. Cai, and S. Han. Efficientvit-sam: Accelerated segment anything model without performance loss. ArXiv, 2024.
[52] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2016.
[53] C. Zhou, C. C. Loy, and B. Dai. Extract free dense labels from clip. In ECCV, 2021.
[54] J. Zhou, C. Wei, H. Wang. W. Shen, C. Xie, A. L. Yuille, and T. Kong. Image bert pre-training with online tokenizer. In ICLR, 2022.
[55] X. Zou, J. Yang, H. Zhang. F. Li, L. J. Wang. L. Wang. J. Gao, and Y. J. Lee. Segment everything everywhere all once, 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We confirm that the main claims made in the abstract and introduction ac- curately reflect the paper's contributions and scope, and are supported by experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of our work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
13

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide such details in Appendix A Section 3
and Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
14

---

We use only publicly available datasets, as summarized in Table 9 Code and trained models will be released upon acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
sublic/guides
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips /nips.cc/pppi cc/ wwtpccccppplccccccccccccccccceet htapllic/uuiieeee guides CodeSubmisea cnooinicc)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix A contains all training and evaluation details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Three of the four downstream tasks training-free free, error bars reported. For the fourth, we follow prior works, which similarly omit error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етrог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide a detailed discussion the compute requirements of REN in Additional details needed for reproducing the experiments mentioned in Appendix and Section 4.2
Section 4T
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We confirmed the research conducted the paper conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
discuss broader impact of our work in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All artifacts used in this work standard academic datasets and publicly available software, and each has been properly cited. We mention the licensing information of each dataset in Table 9
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Comprehensive details are provided in the paper to ensure the proposed assets are easy to use and understand.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development in this research does involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
19

---

Supplementary Material
This supplementary material is organized as follows: Appendix A provides implementation de- tails; Appendix B discusses alternative approaches explored during the development of REN; and Appendix C addresses broader societal impacts of our work.
A Implementation Details
We implement REN using PyTorch, with all training and evaluation performed on a single NVIDIA A40 GPU. For the vision backbone, we use frozen features from three pretrained encoders- DINO ViT-B/8, DINOv2 ViT-L/14, and OpenCLIP ViT-g/14-as detailed in Table The region encoder comprises 4 cross-attention decoder layers with attention heads.
Training, Training is performed on images sampled from the Ego4D dataset [13] We use a batch size of 16 and randomly sample up to 256 point prompts per image generate region tokens. Segmentation masks are precomputed for all training images using the SAM Automatic Mask Generator [22], with 32 x2 point grid and a stability threshold of 0.9. These masks used to assign region IDs for the contrastive loss and to average-pool encoder features for the feature similarity loss. If multiple masks overlap a given point prompt, we select the mid-sized one.
To compute the contrastive loss, we create augmented views of each image using a combination of horizontal flipping, color jittering. sharpness adjustment, affine transformations (rotation and shear), cropping, and resizing. Both the contrastive and feature similarity losses are weighted equally during training. The model is optimized using Adam W with learning rate of 0.001, cosine decay schedule, 100 warmup steps, and gradient clipping with a maximum norm of 5.0.
Inference. For tasks where we do not extend pretrained REN to new image encoder, we use either grid-based or SLIC-based prompting. For SLIC-based prompting, we use Fast-SLIC [2]] with a compactness value of 256. The number of point prompts is set to match the number of patches produced by the image encoder (e.g.. 1369 for DINOv2). Token aggregation is then performed by constructing an adjacency graph using SeiPy [9). where an edge is added between two region tokens if their cosine similarity exceeds a threshold μ.. Connected components are identified via breadth- first search (also using SciPy), and groups with fewer than three region tokens are discarded. Figure reports the average number of region tokens per image for varying values of μı; we find that μ = 0.973 performs consistently well across tasks. Additional task -ssk-spec parameters are detailed in Section
For tasks involving the extension of pretrained REN to a new image encoder, we use Saaee-based prompting with 576 point prompts and a compactness of 256. Among the available models, the REN trained on DINO features performs best region segmenter, possibly due its smaller patch size. For these transfer settings, we use lower aggregation threshold of 0.8.
Table summarrize the datasets used in this work.
B Explored Alternatives
To better understand and validate the design decisions in REN, we experimented with some alternative approaches. We discuss these explored alternatives below along with their empirical outcomes and limitations.
Attention Supervision. We examine the effect of direct attention supervision during training, which would encourage each region token to aggregate information from its corresponding object.
Table 8: Image encoders used for training REN, Each encoder is listed with its architecture, input resolution, and output feature dimensionality used for region token extraction.
Image Resolution
Feature Dimension
Encoder
Architecture
DINO [5] DINOv2
VIT-B/S ViT-L/14 ViT-g/14
384x384
768
32
518x518
1024
OpenCLII
17
224x
1408
20

---

Dataset
Task(S)
License
Ego4D
Training, Visual Query Localization
MIT License
ADE20K
Semantic Segmentation
BSD-3-Claus License
VOC2012
Semantic Segmentation
CC BY 2.5 CC BY 4.0
COCO
Visual Haystacks, Image Retrieval
Table 9: Summary of datasets used in this work. All datasets used in this work are standard academic benchmarks and publicly available. Licensing information for each dataset is provided.
DDINOV2
REN
Point Prompts
Learnable Queries
0.8
1000
1369
0.7
Tokens per image
100
Feature similarity
0.6
62.4
72.1
0.5
40.8
10
20.9
0.4
10.5
0.3
0.2
0.85
0.875
0.9
0.925
0.95
0.975
500
1000
1500
2000
Aggregation threshold
Training iterations
Figure 8: Token count vs. aggregation thresh- old. The average token count per image is com- puted using the ADE20K validation set.
Figure Point prompts vs. learnable queries Using point prompts leads to a faster conver- gence of feature similarity loss.
Specifically, for each point prompt, the model was trained to focus only on the relevant object by aligning its attention map with the object's SAM mask. Following Cheng et al. [7], we used combination of binary cross-entropy and DICE loss, assigning equal weight to both components: Lann = Lace + Ldice- In our experiments, applying attention supervision only to the attention maps of the final cross-attention block proved be the most effective strategy for supervising attention maps. However. incorporating attention supervision ultimately degraded overall performance as shown in Table 10 We attribute this to the restrictive nature of direct attention supervision, which may hinder the model's ability to flexibly learn contextual representations and predict distinct representations for different point prompts within the same region.
Point Prompts vs. Learnable Queries. In the early stages of developing REN, we explored an alternative approach based on learnable queries, inspired by DETR 4 and MaskFormer []. this setup, fixed set of learnable queries was used to cross-attend patch features and produce region tokens. While this formulation yielded reasonable initial results, it introduced complex optimization problem- Sablle--rrimarlly due to the use of Hungarian matching to align the predicted set of tokens with the target regions. In contrast,
current point-based formulation offers significantly simpler and more efficient training pipeline. It eliminates the need for set matching and allows for faster convergence as shown in Figure 9 Moreover, point-based prompting offers greater flexibility and user control. For example, in tasks like Visual Query Localization and COCO image retrieval (Section 4 is often desirable to generate region tokens for only small subset of objects, such as a specific query object. The point-based approach makes this possible in straightforward manner, which difficult to achieve with fixed learnable queries.
C Broader Impact
REN's efficiency supports broader accessibility by enabling high-quality visual understanding on resource-comee connieeeccnttaiieeed devices, such as mobile platforms or real-time systems in assistive technology and wow-bandwidth settings. also contributes scientific progress by advancing foundational research in compact representation learning and enabling tasks like episodic memory retrieval. Furthermore, by eliminating the need process hundreds of patch tokens models, REN can help reduce the energy consumption and carbon footprint of large-scale video understanding systems.
large segmentation
21

---

Table 10: Evaluating attention supervision a training objective. Incorporating attention super- vision results in a performance drop for both semantic segmentation and visual query localization.
Contrastive Loss
Feature Similarity
Attention
VOC2012
VQ2D
Supervision
86.5
63.7
86.4
60.3
At the same time, the ability to efficiently localize and track objects in video streams could be misused for mass surveillance, particularly if deployed in real-time or without user consent. Additionally, incorrect predictions-suce as false positives object localization- could lead harmful outcomes in aafety-critical systems like autonomous vehicles or security monitoring. To mitigate these risks, we recommend incorporating safeguards such as transparency deployment, human-in in-the-loop oversight, and clear communication of failure modes. As foundational work, its societal impact will depend heavily downstream use.
22


Reviews:
[{"review_id": "mASdvR4VPn", "reviewer": "Reviewer_TN4E", "summary": "* This paper introduces the Region Encoder Network (REN), a model that efficiently generates region-level representations from patch-based image encoders such as DINO, DINOv2, and OpenCLIP.\n* REN directly produces region tokens by leveraging point prompts as queries and applying cross-attention over patch features, achieving a 60× speedup and 35× lower memory usage compared to class-agnostic segmenter (e.g., SAM) based approaches.\n* REN is trained with a combination of contrastive loss and feature similarity loss, which enhances object-level discriminative power while preserving the generalization capability of the backbone.\n* Across various benchmarks (semantic segmentation, visual query localization, and image retrieval), REN consistently outperforms or matches patch-based and SAM-based region representations with higher efficiency.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## **Strengths**\n\n* REN eliminates the primary bottleneck of region token generation by removing the segmentation step and employs a lightweight cross-attention structure, resulting in substantial improvements in speed (up to 60× faster) and memory efficiency (up to 35× lower usage) compared to SAM-based methods.\n\n\n* The method is experimentally validated to work with various patch-based image encoders, including DINO, DINOv2, and OpenCLIP, and can be extended to new encoders without requiring additional training.\n\n\n* REN demonstrates superior performance over patch-based and SAM-based region representation methods on multiple benchmarks, including semantic segmentation, visual query localization (Ego4D VQ2D), and Visual Haystacks.\n\n\n## **Weaknesses**\n\n* While REN is described as \"segmentation-free,\" the training process relies on pseudo-labels generated by SAM, implying an indirect dependency on segmentation and possible interpretation as knowledge distillation from SAM.\n* Although the authors emphasize the importance of preserving alignment with the original encoder via feature similarity loss, the ablation study is limited and lacks quantitative analysis on its impact on downstream tasks.\n* The self-supervised region token learning in REN, based on contrastive learning for object-level consistency, can be viewed as an incremental extension of prior works such as TokenCut and LOST, rather than a fundamentally novel paradigm.\n* As acknowledged by the authors, there exists inherent ambiguity regarding whether a single point prompt represents the entire object or only a part, requiring heuristic adjustment of the aggregation threshold (µ) based on the task.\n  * The point prompting strategy may lead to reduced accuracy in complex scenes containing overlapping objects, where precise region separation is more challenging.\n* As acknowledged by the authors, REN's region segmentation provides lower boundary precision compared to SAM, making it less suitable for applications requiring high-precision interactive segmentation.\n* Minor Typo: Line 303, \"limiting it’s\" → \"limiting its\".", "was_revised": true, "final_justification": "The authors have addressed several concerns, including clarifying the “segmentation-free” definition, showing that SAM dependency is limited to training, adding quantitative evidence for feature similarity loss, and demonstrating robustness in dense and complex scenes.   They also provided results on stronger backbones and clarified differences from DETR and Mask2Former.  \nThese efforts are appreciated, and for these reasons, I will maintain my original borderline accept rating.", "rebuttal": "Thank you for reviewing our work and providing constructive feedback. We appreciate your recognition of REN’s efficiency, broad encoder compatibility, and strong performance across multiple benchmarks. We have carefully addressed each of your comments and concerns below.\n\n\n> [Weakness 1] While REN is described as \"segmentation-free,\" the training process relies on pseudo-labels generated by SAM, implying an indirect dependency on segmentation and possible interpretation as knowledge distillation from SAM.\n\nWhen we describe REN as “segmentation-free,” we refer to its inference phase: **region tokens are generated directly from the trained REN model without performing any explicit segmentation** step, unlike prior works. While pseudo-labels from SAM are used only during training to guide learning, this does not impose a segmentation requirement at inference. For instance, all downstream tasks in Section 4.2 operate without any segmentation step, resulting in significantly lower computational time and memory usage.\n\n\n> [Weakness 2] Although the authors emphasize the importance of preserving alignment with the original encoder via feature similarity loss, the ablation study is limited and lacks quantitative analysis of its impact on downstream tasks.\n\nThe ablation study presented in **Table 7** (copied below for quick reference) **already demonstrates that** using both feature similarity and contrastive loss yields the best overall performance. To further quantify the benefit of feature alignment, we additionally report results on the **COCO retrieval task: the mRP is 0.56 when only contrastive loss is used, and it is 0.65 when feature similarity loss is also used** (higher is better). This now constitutes an ablation across all downstream tasks discussed in the paper, except for Visual Haystacks (because there REN is used purely as a segmentation model to guide feature pooling from SigLIP 2 features).\n\n| Contrastive Loss | Feature Similarity | VOC2012 | VQ2D  |\n|:----------------:|:-----------------:|:-------:|:-----:|\n| ✓                |                   |  77.7   | 46.9  |\n|                  | ✓                 |  86.0   | 49.7  |\n| ✓                | ✓                 | **86.5**| **63.7** |\n\n\n> [Weakness 3] The self-supervised region token learning in REN, based on contrastive learning for object-level consistency, can be viewed as an incremental extension of prior works such as TokenCut and LOST, rather than a fundamentally novel paradigm.\n\nWe respectfully disagree. **TokenCut and LOST are training-free methods designed specifically for object localization** in images. In contrast, **REN is a trained model that learns generalizable image representations applicable to a wide range of tasks**, including semantic segmentation, instance-level localization, image-based retrieval, and text-based retrieval. The only commonality is that both approaches leverage object-level cues present in patch features from self-supervised Transformer models (as noted in lines 81–83), but **REN’s goal, training paradigm, and capabilities differ fundamentally from TokenCut and LOST**.\n\n\n> [Weakness 4.1] As acknowledged by the authors, there exists inherent ambiguity regarding whether a single point prompt represents the entire object or only a part, requiring heuristic adjustment of the aggregation threshold (µ) based on the task.\n\nWe have acknowledged this as a limitation. However, as shown in Figure 7, REN’s performance is robust across a wide range of aggregation thresholds (µ), indicating that the model is not highly sensitive to this parameter in practice.\n\n\n> [Weakness 4.2] The point prompting strategy may lead to reduced accuracy in complex scenes containing overlapping objects, where precise region separation is more challenging.\n\nWe think the point prompting strategy is more adaptive and better suited for complex scenes, as the **number of point prompts can be increased to capture finer object details** in dense regions. This advantage is evident in our results on the challenging Ego4D VQ2D task, which involves localizing small, overlapping, and occluded objects in highly cluttered scenes (see Figure 4). **REN significantly outperforms prior methods**, including those specifically designed for this task, highlighting its effectiveness in handling overlapping objects and complex layouts. Moreover, our segmentation results on ADE20K, where REN surpasses patch-based baselines, further support this conclusion.\n\n\n> [Weakness 6] Minor Typo: Line 303, \"limiting it’s\" → \"limiting its\".\n\nThanks for pointing it out; we will fix it in the revision."}, {"review_id": "kklSWE4Nn6", "reviewer": "Reviewer_Z7WG", "summary": "To alleviate the computational burden of generating region-based representations by existing visual foundation models, this paper designs REN, a novel module that transforms patch-based image features into object-aligned region tokens using point prompts. Apart from the efficiency, REN also improves the quality of the generated region tokens under the supervision of mask generated by SAM2 while equipped with designed contrastive and feature similarity losses. \n\n   Specifically, REN uses a lightweight stack of cross-attention blocks: point prompts attend to patch features from a frozen image encoder, producing tokens for every prompt. During training, REN is optimized with a contrastive loss (grouping tokens from the same object) and a feature-similarity loss (to align tokens with the encoder’s feature space), enabling it to “implicitly segment” objects through attention. At inference, REN can be prompted with a dense grid or superpixel-based points to cover the image.\nBy analyzing runtime and memory occupation, REN behaves more efficiently than other SAM-based methods. Experiments on several downstream tasks prove that REN is capable of extending to other encoders without dedicated training while achieving better performance.", "questions": "1. In Section Intro, “existing SAM-based methods for region token ...., which can remove fine-grained details and .... Current segmentation methods also .....”, how can you make sure that it’s the simple linear aggregation operation that causes the over-representation and no-representation phenomenon?\n2. Fig. 1 can be redrawn to better epitomize the implementation details. The mask generated by SAM2 and the definition process of positive and negative samples should be added.\n3. In Section 4.3, “all ablations are conducted using REN trained with DINOv2 as the image encoder.”, since DINOv2 with registers reports its segmentation improvement, have you ever tried to use DINOv2 with registers as your encoder?\n4. Some descriptions in this paper are confusing. In Abs and Intro, this paper uses “region-based image representations”, “region representations”, “region tokens” and “region-based representations”. Are there any difference among these words? Or some of the words share the same meaning?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths of the paper include comprehensive experiments that cover both Processing time comparisons and evaluation on promising downstream tasks, as well as the detailed analysis of strong performance of the proposed method. However, weaknesses are noted in the drawing of the method diagram. Additionally, the proposed method does not consider specific scenes, which means that what kind of scenes can make the best use of the proposed method. \nStrength:\n1. REN replaces an explicit segmentation step with cross-attention from point prompts, a creative alternative to methods that average-pool patch features within segmentation masks. The use of contrastive + feature-similarity losses to train these region tokens is a clever mechanism to ensure tokens remain semantically meaningful and aligned with the backbone features.\n2. Region tokens generated through REN are very compact. The parameters and inference time, as well as the cuda memory occupation all get huge improvement according to the experiments. Besides, in the experiments on downstream tasks, REN consistently improves accuracy, showing its superiority.\n\nWeakness:\n1. Fig. 1 is the core diagram that presents the whole pipeline of the proposed method. However, this paper lacks some crucial components in Fig. 1, such as the mask generated by SAM2. Since the mask is the Supervision signal of the network, readers may feel confused because of its incomplete presentation.\n2. This paper discusses images with few objects in Intro and argues it as a reason which causes the inefficient of current patch tokens generation. Although REN shows its efficiency in such images, it lacks of analysis about images with dense objects.", "was_revised": true, "final_justification": "I think the authors have addressed my concerns. According to the contributions of the paper, I will keep my rating.", "rebuttal": "Thank you for reviewing our work and providing constructive feedback. We appreciate your recognition of REN’s creative design, efficiency, compact region tokens, and strong performance validated by comprehensive experiments. We have carefully addressed each of your comments and concerns below.\n\n> [Weakness 1 & Question 2] Fig. 1 can be redrawn to better epitomize the implementation details. The mask generated by SAM2 and the definition process of positive and negative samples should be added.\n\nThank you for the suggestion. We will update Figure 1 to include the missing training components—specifically, the SAM-generated masks, the target region tokens used for feature similarity loss, and the positive/negative samples used for contrastive learning.\n\n\n> [Weakness 2] This paper discusses images with few objects in Intro and argues it as a reason which causes the inefficient use of current patch tokens generation. Although REN shows its efficiency in such images, it lacks analysis about images with dense objects.\n\nOur argument in the intro is that the **number of tokens needed to represent an image should depend on its complexity**, which is not the case for current patch-based methods. For example, on Pascal VOC, we can represent images using fewer than 30 tokens on average while achieving performance comparable to a patch-based counterpart that uses 1,369 tokens per image. A similar analysis on ADE20K shows that we achieve comparable performance using 41 tokens per image instead of 1,369 (Figure 7). This demonstrates that **REN allocates more tokens as scene complexity increases** (30 → 41 on average), enabling efficient handling of both simple and dense scenes.\n\nFurthermore, REN effectively handles highly complex and dense images as well. This is evident from our results **on the challenging Ego4D VQ2D task**, which involves localizing small, overlapping, and occluded objects in cluttered scenes (see Figure 4). **REN significantly outperforms** all prior methods, including those specifically designed for this task.\n\n\n> [Question 1] In Section Intro, “existing SAM-based methods for region token ...., which can remove fine-grained details and .... Current segmentation methods also .....”, how can you make sure that it’s the simple linear aggregation operation that causes the over-representation and no-representation phenomenon?\n\nSimple linear aggregation of features treats all pixels equally, which can dilute important details—over-representing irrelevant regions and under-representing key discriminative areas. Additionally, restricting feature aggregation strictly to the object region can discard valuable contextual information. This effect is reflected in our results in Tables 7 and 10, summarized below:\n\n| Method                                   | Characteristic                        | VOC2012 | VQ2D  |\n|---------------------------------------|-------------------------------------|:-------:|:-----:|\n| Feature similarity only (Table 7)         | Learn to mimic linear aggregation of features            |  86.0   | 49.7  |\n| Attention supervision (Table 10)        | Restricts aggregation to object region with equal weighting |  86.4   | 60.3  |\n| Proposed (contrastive + feature similarity) | Learn discriminative feature aggregation | **86.5** | **63.7** |\n\nFurthermore, **REN consistently outperforms SAM-based methods—which rely on simple linear aggregation**—across a wide range of tasks (Section 4.2), reinforcing that learned discriminative aggregation leads to superior region representations.\n\n\n> [Question 3] In Section 4.3, “all ablations are conducted using REN trained with DINOv2 as the image encoder.”, since DINOv2 with registers reports its segmentation improvement, have you ever tried to use DINOv2 with registers as your encoder?\n\nWe conduct additional ablations using DINOv2 with registers as the image encoder, focusing on semantic segmentation on the Pascal VOC validation set. The results are consistent with the ablations reported in the paper:\n\n**Training loss ablation:** A combination of contrastive and feature similarity loss performs the best.\n\n| Contrastive Loss | Feature Similarity | VOC2012 |\n|:----------------:|:-----------------:|:---------:|\n| ✓                |                     |  76.3     | \n|                    | ✓                 |  88.9     | \n| ✓                | ✓                 | **89.4** | \n\n**Effect of prompting strategy and token aggregation:** SLIC-based prompting provides slightly better performance, and token aggregation maintains accuracy while reducing the number of tokens required to represent each image.\n\n| Prompt | Token Aggregation | VOC2012 |\n|:----------------:|:-----------------:|:-------:|\n| Grid                |                      |  89.1 | \n| Grid                | ✓                  |  89.1 | \n| SLIC               |                      |  89.3 | \n| SLIC                | ✓                 |  89.4 | \n\nAs expected, using DINOv2 with registers provides a marginal performance improvement:\n\n| Image Encoder  | mIoU |\n|----------------|:-----------------:|\n| DINOv2                              | 88.9     | \n| DINOv2 with registers        | 89.4     | \n\nThese results show that **REN’s performance scales with backbone improvements for the given task while maintaining its core performance trends and ablation conclusions**.\n\n\n> [Question 4] Some descriptions in this paper are confusing. In Abs and Intro, this paper uses “region-based image representations”, “region representations”, “region tokens” and “region-based representations”. Are there any differences among these words? Or some of the words share the same meaning?\n\nAll these phrases mean the same thing. We will clarify this better in the revision."}, {"review_id": "Qy8VRwRPel", "reviewer": "Reviewer_Zqft", "summary": "The paper introduces the Region Encoder Network (REN), a model that generates region-based image representations via point prompts. It bypasses expensive segmentation steps, using cross-attention to transform patch features into region tokens, achieving 60× faster token generation with 35× less memory. Experiments show REN outperforms patch-based encoders and SAM-based methods in efficiency and performance across tasks like semantic segmentation and retrieval.", "questions": "Please refer to the 'weakness' part.", "limitations": "Please refer to the 'weakness' part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well written and easy to follow.\n\n2. The reduction in cost of the proposed method compared to SAM 2 is impressive.\n\n3. The experimental results fully reflect the effectiveness of the proposed method.\n\nWeaknesses:\n\n1. The essence of REN's training is to establish the correlation between points, but the association between points and regions still seems to rely on other models for assistance, which restricts its application scope and makes it impossible to truly achieve real-time performance.  \n\n2. As described in lines 108-111 of Table 1, the RPN utilizes the prior information from SAM 2. I am curious whether this makes the training constitute an implicit distillation of SAM 2, which is why the RPN has performance advantages? Please replace the priors of other models or directly discuss this point.  \n\n3. Compared with DETR and Mask2Former, the RPN method is slightly less innovative, as it merely replaces learnable queries with point prompts. In addition, I am curious whether the region representations obtained by RPN are superior to those obtained by methods such as DETR and Mask2Former?\n\n4. Can the proposed method be directly used for Fine-tuning like [1], and bring performance advantages to fine-grained tasks?\n\n\n[1] UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding. ECCV 2024.", "was_revised": true, "final_justification": "Thanks for your responses, which have addressed most of my concerns. I will increase my score.", "rebuttal": "Thank you for reviewing our work and providing constructive feedback. We appreciate your recognition of the paper’s clarity, the significant cost reduction, and the strong experimental results demonstrating the method’s effectiveness. We have carefully addressed each of your comments and concerns below.\n\n\n> [Weakness 1] The essence of REN's training is to establish the correlation between points, but the association between points and regions still seems to rely on other models for assistance, which restricts its application scope and makes it impossible to truly achieve real-time performance.\n\nAs shown in Table 1, REN processes images at **more than 30 FPS**, demonstrating that it is **capable of real-time performance**. Importantly, **REN only uses external masks from SAM during training**. At inference, REN operates independently of any segmentation model—region tokens are generated directly from the backbone features and point prompts, making the method lightweight and efficient.\n\n\n> [Weakness 2] As described in lines 108-111 of Table 1, the RPN utilizes the prior information from SAM 2. I am curious whether this makes the training constitute an implicit distillation of SAM 2, which is why the RPN has performance advantages? Please replace the priors of other models or directly discuss this point.\n\nThere is a typo in Section 3: we use SAM, not SAM 2, to generate region masks for training. This choice was made to ensure a fair comparison with baselines [1, 2], which also use SAM-generated masks. We appreciate you bringing this to our attention and we will correct it in the revision. \n\n[1]  Region-Based Representations Revisited. CVPR 2024.\n\n[2]  RELOCATE: A Simple Training-Free Baseline for Visual Query Localization Using Region-Based Representations. CVPR 2025.\n\n\n> [Weakness 3.1] Compared with DETR and Mask2Former, the RPN method is slightly less innovative, as it merely replaces learnable queries with point prompts.\n\nREN is fundamentally different from DETR and Mask2Former in its goal, training paradigm, design, and usage. **DETR and Mask2Former are trained in a fully supervised manner for detection and segmentation**, respectively. In contrast, **REN is trained using self-supervised learning to produce efficient region-based image representations applicable across diverse tasks**, including semantic segmentation, instance-level localization, image retrieval, and text-to-image retrieval. REN also employs a smaller and simpler cross-attention architecture, making it more lightweight and efficient.\n\nFurthermore, as discussed in Appendix B, using **point prompts instead of learnable queries has significant implications:** \n\n(1) **Simplified training objective** – removing the need for Hungarian matching leads to faster convergence (see Figure 9); and \n\n(2) **Greater flexibility and user control** – allowing region tokens to be generated for specific subsets of objects (e.g., a query object in Visual Query Localization or COCO retrieval), which is difficult to achieve with fixed learnable queries of DETR or Mask2Former.\n\n\n> [Weakness 3.2] In addition, I am curious whether the region representations obtained by RPN are superior to those obtained by methods such as DETR and Mask2Former?\n\nUnlike REN, **DETR and Mask2Former are not designed to produce general-purpose region-based representations**, but are specialized for detection and segmentation. As a result, REN’s region representations are more broadly applicable across diverse tasks (ref. Section 4.2).\nTo further examine this, as discussed **in Appendix B, we adapted the Mask2Former architecture for region representation learning**. While it achieved performance comparable to REN, **it required significantly more complex optimization**, leading to slower training (see Figure 9). Moreover, its reliance on fixed learnable queries limits user control, making it **challenging to generate region representations for specific objects** within an image—a capability that REN’s point prompting naturally supports.\n\n\n> [Weakness 4] Can the proposed method be directly used for fine-tuning like [1], and bring performance advantages to fine-grained tasks?  \n> [1] UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding. ECCV 2024.\n\nWe think it’s possible; great suggestion for future work."}, {"review_id": "egSZpraOKT", "reviewer": "Reviewer_PgVF", "summary": "The paper introduces the Region Encoder Network (REN), a novel approach for generating efficient region-based image representations using point prompts. Traditional methods rely on class-agnostic segmenters (e.g., SAM) combined with patch-based encoders (e.g., DINO), which incur high computational costs due to segmentation. REN bypasses this bottleneck by directly generating region tokens through lightweight cross-attention modules, achieving ​​60× faster token generation​​ and ​​35× reduced memory usage​​ while improving token quality. Trained with DINO, DINOv2, and OpenCLIP encoders, REN demonstrates superior performance on semantic segmentation, retrieval, and challenging benchmarks like Ego4D VQ2D and Visual Haystacks.", "questions": "Please see the Weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) Clear problem motivation: Highlights inefficiencies of patch-based methods and limitations of SAM-based region tokens.\n\n(2) Well-structured methodology: Step-by-step explanation of REN’s architecture, training objectives (contrastive loss, feature similarity), and inference pipeline (prompting + aggregation).\n\nWeaknesses:\n\n(1) Architectural similarity to existing methods: The cross-attention module resembles DETR and MaskFormer, raising questions about novelty.\n\n(2) Lacks formal proofs or guarantees for REN’s design choices (e.g., why cross-attention works for region tokenization).\n\n(3) While REN reduces SAM’s computational cost, its performance still depends on SAM-generated masks for training and certain tasks (e.g., token aggregation thresholds).\n\n(4) Limited exploration of alternatives: The ablation study (Section 4.3) focuses on prompting strategies but omits deeper comparisons with other attention variants (e.g., multi-head vs. single-head).", "was_revised": true, "final_justification": "I think the authors have addressed my questions. I updated the rating to Borderline accept.", "rebuttal": "Thank you for reviewing our work and providing constructive feedback. We are pleased to hear that you find our motivation clear and methodology well-structured. We have carefully addressed each of your comments and concerns below.\n\n\n> [Weakness 1] Architectural similarity to existing methods: The cross-attention module resembles DETR and MaskFormer, raising questions about novelty.\n\nREN differs fundamentally from DETR and MaskFormer in its goal, training paradigm, design, and usage. **DETR and MaskFormer are trained in a fully supervised manner for detection and segmentation**, respectively, whereas **REN is trained using self-supervised learning to produce efficient region-based image representations applicable across diverse tasks**, including semantic segmentation, instance-level localization, image retrieval, and text-to-image retrieval. \n\nMoreover, REN adopts a smaller and simpler cross-attention architecture, making it lighter and more efficient. Importantly, as discussed in Appendix B, **REN replaces fixed learnable queries with point prompts**, which has two key implications:\n\n(1) **Simplified training objective** – removing the need for Hungarian matching leads to significantly faster convergence (see Figure 9); and \n\n(2) **Greater flexibility and user control** – allowing region tokens to be generated for specific subsets of objects (e.g., a query object in Visual Query Localization or COCO retrieval), which is difficult to achieve with fixed learnable queries of DETR or Mask2Former.\n\nThus, we think the fact that REN uses cross-attention does not diminish the methodological novelty of REN.\n\n\n> [Weakness 2] Lacks formal proofs or guarantees for REN’s design choices (e.g., why cross-attention works for region tokenization).\n\nWhile formal proofs are uncommon in deep learning, our design choices are validated through ablation studies and other experiments, which is an established and widely accepted practice.\n\n\n> [Weakness 3] While REN reduces SAM’s computational cost, its performance still depends on SAM-generated masks for training and certain tasks (e.g., token aggregation thresholds).\n\nREN’s performance and the quality of its region tokens depend primarily on the feature extractor and the attention weights learned during training, not directly on SAM. **SAM masks are used only to pool backbone features to generate target** region tokens during training. **REN does not rely on SAM masks during token aggregation or inference**.\n\nTo further support that REN’s performance is independent of SAM, we highlight **two key findings** from our paper:\n\n(1)  Prior works (e.g., [1]) directly use SAM masks for generating region tokens, yet REN consistently outperforms them across a wide range of tasks (Section 4.2).\n\n(2)  In an experiment where SAM masks were used to directly supervise REN’s attention weights (see Appendix B), we observed reduced performance, indicating that REN benefits from learning its own attention rather than inheriting SAM’s segmentation.\n\n[1] Region-Based Representations Revisited. CVPR 2024.\n\n\n> [Weakness 4] Limited exploration of alternatives: The ablation study (Section 4.3) focuses on prompting strategies but omits deeper comparisons with other attention variants (e.g., multi-head vs. single-head).\n\nBeyond the ablation studies in Section 4.3, we explored the following alternative designs detailed in **Appendix B**:\n\n(1) **Supervising attention weights with SAM masks:** We tested whether directly supervising attention weights using SAM masks improves performance. The result was negative—this supervision slightly degraded performance.\n\n(2) **Adapting Mask2Former for region token generation:** We experimented with Mask2Former’s architecture and training objectives. While performance was comparable, this introduced unnecessary complexity, slowed training, and reduced user control.\n\nOther design choices, such as using multi-head self-attention, are well-established, have proven to be effective in prior works, and don’t really need further validation."}]

Author rebuttal:

Meta-review: (a) This paper introduces the Region Encoder Network (REN), a novel module designed to efficiently generate high-quality, region-level image representations from patch-based encoders like DINO and OpenCLIP. The central scientific claim is that it is possible to bypass the computationally expensive explicit segmentation step (e.g., using SAM) common in prior work. REN achieves this by using point prompts as queries in a lightweight cross-attention mechanism over patch features, directly producing region tokens. The model is trained with a combination of a contrastive loss to ensure object-level consistency and a feature similarity loss to maintain alignment with the original encoder's feature space. The key findings are that REN is massively more efficient—achieving up to a 60x speedup and 35x memory reduction—while consistently outperforming or matching both patch-based and SAM-based region representation methods across a variety of downstream tasks like semantic segmentation, visual query localization, and image retrieval.

(b) The paper's most significant strength is the massive improvement in efficiency it offers for generating region tokens, directly addressing a major bottleneck in a practical and important area of computer vision. The 60x speedup and 35x memory reduction are impressive and highly impactful. The proposed REN method is creative, replacing a heavy segmentation step with a lightweight and clever cross-attention mechanism. Reviewers also praised the comprehensive experimental validation, which demonstrates REN's superiority and versatility across multiple encoders (DINO, DINOv2, OpenCLIP) and a wide range of diverse downstream tasks. The method is well-structured, clearly explained, and the results support the claims of the paper.

(c) Initial weaknesses centered on the method's novelty and its relationship to prior work. Some reviewers noted architectural similarities to DETR and Mask2Former, questioning the originality of the cross-attention module. There was also confusion about the "segmentation-free" claim, as the model relies on SAM-generated masks for its training supervision, implying an indirect dependency. Other concerns included a lack of certain ablations, questions about performance in dense scenes with overlapping objects, and the ambiguity of using single point prompts to represent entire objects which the authors have rebutted applies only to the inference stage. 

(d) The recommendation is to Accept this paper. The primary reason is its clear, substantial, and highly practical contribution: a method that drastically reduces the computational cost of generating region-based representations while simultaneously improving their quality. The 60x speedup is a significant improvement enabling efficient runtime inference. The authors' rebuttal was also effective, successfully clarifying the method's novelty by distinguishing its self-supervised, representation-focused goal from the fully-supervised detection/segmentation goals of DETR and Mask2Former. They also convincingly addressed concerns about the dependency on SAM (clarifying it's for training only) and demonstrated the method's effectiveness in complex scenes. Unfortunately, two reviewers (PgVF
And Zqft) havent been involved in the discussion phase despite repeated requests and I am basing this on the author;s rebuttal and the key contribution of the paper.

(e) The rebuttal period was elaborate from the authors, with them given detailed responses addressing the main concerns of the reviewers A key point of discussion was the method's novelty relative to architectures like DETR and Mask2Former, raised by Reviewers Zqft and PgVF. The authors clarified this by explaining that REN's goal is fundamentally different (learning general-purpose representations via self-supervision vs. end-to-end supervised detection), and that their use of point prompts instead of learnable queries simplifies training and adds flexibility. This clarification was accepted by both the reviewers. The authors also addressed the confusion around the "segmentation-free" claim from Reviewer TN4E by explaining it applies to the inference stage, which is the source of the efficiency gains. Furthermore, they provided additional quantitative evidence and new experiments on stronger backbones to address other concerns, such as the impact of their feature similarity loss and performance in dense scenes, solidifying the reviewers' confidence in the work's contributions.
Decision: Accept (poster)