TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
Authors: Yao Xiao
Qiqian Fu
Heyi Tao
Yuqun Wu
Zhen Zhu
Derek Hoiem
Venue: Accepted by TMLR ()

Abstract: Image-text models excel at image-level tasks but struggle with detailed visual understanding. While these models provide strong visual-language alignment, segmentation models like SAM2 offer precise spatial boundaries for objects. To this end, we propose TextRegion, a simple, effective, and training-free framework that combines the strengths of image-text models and SAM2 to generate powerful text-aligned region tokens. These tokens enable detailed visual understanding while preserving open-vocabulary capabilities. They can be directly applied to various downstream tasks, including open-world semantic segmentation, referring expression comprehension, and grounding. We conduct extensive evaluations and consistently achieve superior or competitive performance compared to state-of-the-art training-free methods. Additionally, our framework is compatible with many image-text models, making it highly practical and easily extensible as stronger models emerge. Code is available at: https://github.com/avaxiao/TextRegion.

Published in Transactions on Machine Learning Research (11/2025)
TextRegion: Text-Aligned Region Tokens from Frozen Image- Text Models
Yao Xiao
Qiqian Fu
Heyi Tao
Yuqun Wu
Zhen Zhu
Derek Hoiem
Siebel School of Computing and Data Science
University of Ilinois Urbana- Champaign (yaoxi1, qiqianf2, heyitao2, yuqunsu2, zhenzhu4, dhoiem)@illinoits
Reviewed on OpenReview:
Abstract
Image-text models excel at image-level tasks but struggle with detailed visual understanding: While these models provide strong visual-language alignment, segmentation models like SAM2 offer precise spatial boundaries for objects. To this end, we propose TextRegion, a simple, effective, and training-free framework that combines the strengths of image-text models and SAM2 to generate powerful text-aligned region lokens. These tokens enable detailed visual understanding while preserving open open-vocabulary capabilities. They can be directly applicd to various downstrcam tasks, including open-world semantic segmentation, referring expression comprehension, and grounding. We conduct extensive evaluations and consistently achieve superior or competitive performance compared to state-of-the-art training-free free methods. Additionally, our framework is compatible with many image-text models, making it highly practical and easily extensible as stronger models emorge. Code available at: https:/ /gittuu..comm avaxiao TeaxxiaooTexttee Region..
Open-World Segmentation
Referral Retrieval
SAM2
C:
Query: white horse
1: grass mountain
QE Detuils?
Multi-Object Grounding
Referral Retrieval
CLIP
Query: horse
Query: mountain
SigLIP2
Ans:
TextRegion!
PE
Figure Using feature maps from frozen imago-text models and segment masks from SAM2, we generate lext-aligned region tokens that be directly applied to various ream tasks.
1 Introduction
Contrastive image-text models, such as CLIP (Radford et al., 2021), enable open vocabulary retrieval and classification for images. However, these models typically operate at the global image level. What if we are interested in classifying, localizing, or retrieving specific regions given natural language text? The ability to understand precisely at the region level remains critical for many practical vision-language tasks, including open-vocabulary semantic segmentation, visual grounding, and fine-grained retrieval.
To achieve such detailed understanding, existing methods typically fall into two categories. One group of approaches, represented by Grounding DINO (Tiu et al., 2024b) and DenseCL.IP (Rao al., 2022), explicitly

---

Published in Transactions on Machine Learning Research (11/2025)
trains models using detailed annotations (e.g., bounding boxes or pixel-leeel masks). While effective, these training-hasel approaches struggle to match the broad concept coverage provided by large-scale, contrastively trained image-text models, making their lack of open-vocabulary object recognition a significant limitation for real-world applications. In contrast, methods in the second category, such as MaskCLIP (Zhou et al., 2022), explore how to use existing components of the network to directly empower image-level CLIP for object-text alignment, without applying any training. Those works show that although image-text models are trained primarily to understand images at a global level, they implicitly acquire some ability for fine- grained risual-textual matching, However, patch-level embeddings are insufficient for retrieval grounding et al., 2024), as they often lack spatial coherence and clear boundary aligmment.
tasks (Shlapentokl-Rottieee
Given that modern segmentation models such as SAM (Kirillov et al., 2023) provide precise spatial boundaries, while image-text models like CLIP (Radford et al., 2021) capture what objects are present in images, an important question arises: can we combine the strengths of these two families of foundation models to achieve detailed region-level understanding? Morcover, as most prior work on detailed understanding has concentratod CLIP, it is still uncertain whether alternative image-lext encoders such SigLIP (Zhai el. al., 2023) can effectively address this task.
In this work, we show that when supplied with accurate object masks to resolve spatial boundaries, image-text models can achieve strong zero-shot performance on detailed understanding lasks. Our TextRegion approach provides region-text alignment using the attention mechanism to aggregate patch features within each SAM segmented region, similar to how the image-level CLS token is computed for image-text aligument. We also investigate the use of multi-resoluuion features and propose a method to mitigate the effect of "global" tokens that are prevalent in large models. Together, TextRegion enables CLIP (Radford et al., 2021), SigLIP2 (Tschannen et al., 2025), and Perception Encoder (Bolya et al., 2025) models to generate region-level tokens that are comparable to text embeddings, often outperforming more complex methods that require additional models or retraining,
Our main contribution is a simple, general, effective, and training-free approach to create text- compatible region tokens, enabling powerful zero-shot region-level understanding with existing image-text. models. Our approach has several benefits:
Excellent zero-shot performance: Achieves impressive zero-shot results on tasks including open-world semantic segmentation, referring expression comprehension, and grounding.
Broad architectural compatibility: Generalizes across diverse image-text models, with practical guidelines to ensure consistent region-text alignment.
Plug-and-pla play implementation: Requires only a single attention layer modification and segmentation masks, making TextRegion immediately usable without any training.
2 Related Work
Region-Level Representation. Although patch-based representations dominate current vision research following the advent of vision transformers (Vaswani et al., 2017), region-level representations offer greater semantic richness and sparsity, often leading to superior performance in tasks such as segmentation, retrieval, et al., 2024). Recognizing their suitability for spatially detailed
and classification SShlapentokh-Rothmee tasks, recent efforts have used region tokens for vision tasks (Cheng et al., 2024; Shlapentokh-Rothmen et al., 2024; Pan et al., 2024; Lee al., 2024; Sun et al., 2024). However, existing methods typically involve task-specific training. Training-fre free region-based methods primarily targeted tasks that do not require language understanding, such as image retrieval (Korfhage et al., 2024; Sidhu et al., 2024). To the best of our knowledge, ours is the first study to explore generating text-aligned region tokens without training.
Open Vocabulcyy Segmentation. Recent advances in open-world segmentation like MaskCLIP (Zhou et al., 2022), SCLIP (Wang et al., 2025), ClearCLIP (Wang et al., 2025) and CLIPtrase (Lan et al., 2024a) use image-text models to achieve zero-shot segmentation by aligning patch emboddings with textual descriptions. ProXyCI.IP (Lan et al., 2024b) enhances this by incorporating semantic-rich features from models like
2

---

Published in Transactions on Machine Learning Research (11/2025)
DINO (Caron et al., 2021) and DINOv2 (Oquab et al., 2023). The current SoTA, Trident (Shi et al. 2024) further integrates segmentation architectures like SAM (Kirillov et al., 2023) and SAM2 (Ravi et al., 2024). However, combining all three large models makes their framework complex. Unlike these patch-level methods, our approach aggregates patch embeddings into region-lovel features, effectively transforming the dense segmentation lask inlo a sparse region classification problem.
Referring Expression Comprehension. This task involves identifying image regions described by textual queries. Recent zero-shot methods (Subramanian et al., 2022; Yao et al., 2024; Yang et al., 2023; Shtedritski et al, 2023) combine proposal boxes from MAttNet (Yu et al., 2018) with image-text models for image-caption matching. Evaluating spatial relation understanding is another important aspect of this task, leading some studies to develop methods that enhance spatial reasoning in image-text models (Subramanian et al., 2022). However, our primary goal is to assess the visual-language alignment capability of our region tokens; thus, we focus exclusively on semantic retrieval without employing techniques to recover spatial relations.
Image-Text Models. Contrastive image-texi embedding models, such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), SigLIP (Zhai et al., 2023), SigLIP2 (Tschannen et al., 2025), and Perception Encoder (Bolya et al., 2025), align visual and textual representations through dual encoders. These models generate text-aligned image embeddings, facilitating effective zero-shot performance across classification, text-to-image, and image-to-text retrieval. In this work, we investigate how their image-level embeddings can be transformed into region-level tokens, thus extending their applicability from global image understanding to precise region recognition.
3 Method
TextRegion extends image-text models to support region-level tasks like segmentation, localization, and retrieval. We first revisit. how CLIP's class token aggregates spatial information (Sec. 3.1), noting its role in summarizing patch features. To emulate this behavior for regions, we inject spatial constraints into the transformer's cross-attention, producing text-aligned region tokens (Sec. 3.2). Finally, we introduce two useful tricks to refine region tokens (Sec. 3.3).
3.1 Background and Notation
Contrastive image-text models, such as CLIP (Radford et al., 2021), learn to produce an image-level token [CLS] that is comparable to an embedding of text that captions the image.
The model divides the input image into N patches and uses a convolutional layer to map them patch embeddings, xxx... R-xxd where d is the embedding dimension. After adding po- sitional embeddings,
learnable class token XCLS E Fd is prepended to the sequence, resulting in X [XCLS; X1: ;xN] E (N++)xdd This sequence is processed through L transformer layers. At each layer / E [1, LJ, the class token XCLS progressively aggregates global context via teltion The attention projections are computed as, Q - XW,, K XWE, XWW, where W,. Wa, W,, e Raxd denote the learned projection matrices for queries, keys, and values. The attention weights are represented by a E (N++))(N++) The attention output of the class token, denoted yCLs, is computed as a weighted sum over all value vectors v, where the weights dets,i correspond to the tention scores between XCLS and each patch token x,
Softmax [Q(K) vd
YCLS Vels acls = Octss V, VCLs Rd
(1)
YCLS is passed through an MLP projection layer and added to its original feature to form the block output. The final [CLS] is trained to match with the text embedding of that assigned to the image.
In summary, the final-layer attention output YCLS weighted sum over the value vectors of all patch tokens, with attention weights aets determining each patch's contribution to the image [CLS] token.
3

---

Published in Transactions on Machine Learning Research (11/2025)
0: dog
grass
child
ball
Input Image
(b) Patch Value Prediction
(c) Region Mask
(d) Pooling
Figure 2: Patch Value and Mask-based Attention Pooling: (b) shows the segment results based on the
patch value, indicating that the patch values are aligned with visual-language semantics, but could be noisy. (c) the resized mask for a specific region, which restricts the aggregation to patches within that region. (d) demonstrates that by attending only to region-related patches, we can obtain a text-aligned region token, effectively mitigating the influence of imprecise patch values.
3.2 TextRegion Approach
Key Insight. Echoing MaskCLIP (Zhou et al., 2022), Eq. suggest that the projected values from the value projection layer in the final block are quite rich in linguistic semantics, whereas the attention weights Ocel is trained capture global information rather than local information. To enhance the region-text alignment, one key is how organize the values. To this end, we impose region-specific attention constraints, restricting the [CLS] token to attend only to patches within designated region of interest. We present the details of region-specific allention construints in Fig. 2. Basically, this involves ignoring patches that are unrelated to the target region. Since most patch values within a region share the same segmentation prediction, simply pooling these patch values to form a region token can enhance the robustness the visual tokens.
We explain how to get region tokens in the following paragraphs, with an overview shown in Fig. 3.
Mask Generation
RX
SAM2
Rogion Masks
Resize
Crop & Resize
Buratte
Pooling
zehra
Region Tokens
2200
Eacodo
Encode
free
Resize
pirailla zehra rock
Regiom Logits
Patch Values
Sclf Attn
200
Last. Attention Block
Text Encoder
Multi-reslatton Patch Encoding
Mask-baseed Attention Pooling
Prediction
We generate R soft masks using SAM2, with values ranging from 0 to 1, where each mask corresponds to distinct region in the input image. Patch Encoding: The image encoded obtain a multi-resolution feature map, which fed into the final attention block of the frozen image-text models. See Sec. 3.3 for details. Mask-based Attention Pooling: As illustrated in Fig. 2, perform pooling based the R bilinearly downsampled masks. Prediction: Using the pooled text-aligned region tokens, support both zero-shot region-sparse classification and dense prediction.
Figure 3: TextRegion Framework. Mask Generation:
4

---

Published in Transactions on Machine Learning Research (11/2025)
Mask Generation. For an imput image of size H x W, apply SAM2 (Ravi et al., 2024) to segment it into R regions. This yields a set of soft. masks M [M;; Ma] RRxHxW where each mask M, contains per-pixel logits MA,, [0, 1]. A higher value of Ma, indicates that pixel (h, w) is more likely to belong to region r. We downsample the high-resolution soft mask M, to the feature map resolution of the vision encoder using bilinear interpolation, obtaining m, E RRV. N means the number of patch lokens. Each element myy E [0. 1] represents the relevance of patch to region r.
Mask-based Attention Pooling. As shown in Fig. 2, we impose locality by modulating attention scores with segmentation masks. At the final attention layer L, the region-specific token y, selectively aggregates features by replacing the attention scores with mr.i. We downsample the SAM2 masks to compute region tokens rather than upsample the patch feature map because 1) computing region tokens at the pixel level is shoource-heayy and less robust; 2) upsampling can introduce extra noise which is problematic in the training-free setting- Additionally, we adopt soft masks instead of hard masks to better align with the attention mechanism, allowing high-confidence patches to contribute more to the aggregated tokens. The formulation for computing region lokens is:
N My U2 m. V, yr Rd
Yr
(2)
Here, i. is the attention value feature of patch i at the final layer L. This formulation suppresses irrelevant. patches (my, == 0) and aggregates features only from region-relevant ones (m,, > 0). The resulting region token yr acts region-aware analogue of the global [CLS] token, while maintaining compatibility with the original text embedding space.
Some models, such as SigLIP (Zhai et al., 2023), compute a delegate [CLS] token using a final attention pooling block. Our approach also applies to these models with slight implementation differences, which we detail in Appendix See A.3.
Prodiction. With the global image zero-shot classification ability maintained, TextRegion supports both zero-shot region classification and pixel-level prediction. For region classification, we encode candidate text labels into embeddings etext using the pretrained text encoder. Region classification logits are computed by cosine similarity between the region token yr and cach text embedding, weighted by temperature y (= 100 in CLIP):
Ctext RBxxd_ logits, 7 cos(yr, Ctext), logits, RC
(3)
where C the number of candidate labels. For pixel-level prediction of the target region, we restore spatial details by broadcasting the region logits back the original image resolution. In detail, we broadcast logits, RC to CC and then perform element-wise multiplication with the high-resolution soft SAM2
mask M, E HRW:
pred
dense Broadcast (logits,) O M,, pred E CxHXW
(4)
This operation propagates sparse region-level logits onto their corresponding masks, yielding dense pixel-level logits for each region. To obtain the final dense prediction for the entire image, we simply sum the dense logits across all regions to produce a unified dense logit map. Note that we retain the use of soft masks M, [0, 1] during prediction, as higher mask values indicate a higher likelihood that pixel belongs to the region. Accordingly, the logits of these pixels should more closely match the computed region logits.
3.3 Region Token Refinement
Reducing global token interference. As illustrated in Fig. 4, we find that the large model often successfully segments more challenging images but fails on simpler cases. We attribute this to the presence of global patches. Shao et al. (2024); Darcet et al. (2023) and Yang et al. (2025a) observe that certain tokens in vision transformers capture global image semantics rather than localized patch-level details. When aggregated via mask-basod attention, such patches bias region tokens toward global text semantics, degrading localization performance. We introduce region consistency validation to filter out global patches. For each patch within
5

---

Published in Transactions on Machine Learning Research (11/2025)
0: garden
donkey llame goot came!
1: people
2: shovel
3: trees
sky
200
0.4
0:
Doccat
1: remote
remote
-0.
2: bed
bed
(a) Image
(b) Before
(c) After
(d) Region Masks
(e) Local Similarity
Global Patches. The first shows segmentation examples for complex images. Despite the difficulty, the model produces correct results. In contrast, the second row presents easier case where the model fails to segment properly. (b) and (c) show the segmentation results before and after removing global patches, respectively. (d) presents the region masks generated by SAM2, which are used to compute the local similarity defined in Eq. (e) visualizes the local similarity of patches, where lower similarity indicates a higher likelihood of being a global patch. In this case, the model incorrectly classifies the bed as a cat due the presence of many global patches in the bod area.
Figure 4:
region (as defined by the mask M,), compute both intra-region and inter-region similarity scores:
$in
1
Σ JEP,
cos (x, Xy), Sout
cos (x,, Xt), Stocal 8jn.i
Sout ,
(5)
P
P
kEP
Here, Pr and P-.r represent the sets of patches inside and outside region respectively. A patch is identified as global if its local similarity Stocal Sm.i - Sout, < where т is a predefined threshold. Such patches are excluded from region token aggregation, ensuring that region-specific attention remains focused on locally relevant patch tokens. We apply a fixed threshold r = 0.07 across all images.
Multi-resolution patch encoding. Standard image-text models process fixed-resolution inputs, producing low-dimensional value feature maps that poorly capture small regions, often resulting in ineffective region masks (near-zero activations). Inspired by LLaVA's AnyResolution (Liu et al., 2024a) strategy, propose: (1) split the original image into non-overlapping crops, resize and encode cach to obtain localized features, and then concatenate them into high-resolution feature map Vnighi (2) concurrently, encode the full image to retain global context but low-resolution feature Vlow The final feature Vinal = Vhigh + upsample (Vow), which integrates local details with global semantics context, improving representation of both small and large regions.
4 Experiments
We show the strong region classification capabilities of TextRegion in Sec. 4.1, where our text-aligned region tokens outperform more complex methods that rely additional models retraining. In Sec. 4.2, show that TextRegion also effectively handles instance queries, i.c., referring expression comprchension. Morcover, our framework seamlessly inlegrates with other image-lext models, such as SigLIP2 and Perception Encoder, to produce meaningful text-aligned region tokens. Going beyond standard referring expression comprehension, which retrieves only a single region, we further demonstrate in Sec. 4.3 that our pipeline supports multiple object grounding. For all experiments, we use SAM2 (Ravi et al., 2024) with the Hiera-Large backbone generate region masks. Additional hyperparameters are detailed in Appendix See A.2.

---

Published in Transactions on Machine Learning Research (11/2025)
Additional Model
With background
Without background
Method
DINO
SAM
VOC21 Context60 Object VOC20 Context59 Stuff City A ADE
Avg
Train CLIP ViT-B/16
TTD (Jo al., 2024)
61.1
37.4
37.4
23.7
27.0
CIIP-DINOiser (Wysncraíska al,, 2024)
62.1
32.4
34.8
24.6
31.7
SAM-CLIP (Wang al., 2024b)
00.6
29.2
31.3
17.1
Training- Free, CLIP V
MaskCLIP (Dong et al., 20233)
43.4
23.3
20.6
74.9
26.4
16.7
24.9
11.9
30.3
SCLIP (Wang et al., 2024a)
59.1
30.4
30.5
90.4
31.2
22.4
32.2
16.1
38.2
ResCLIP (Yang et al, 2025b)
86.0
36.8
24.7
35.9
ProxyCLIP (Lan et al., 2024b)
61.3
35.3
37.5
80.3
39.1
26.5
38.1
20.2
42.3
LaVG (Keng Cho, 2024)
62.1
31.6
34.2
82.5
34.7
23.2
25.0
15.8
38.6
LPOSS+ (Stnjnić al., 2025)
34.3
79.3
38.6
26.5
37.9
Trident (Shi al., 2024)
67.1
38.6
41.1
42.2
28.3
42.9
21.9
CLIPtrase (Shao al., 2024)
57.1
32.0
44.2
82.2
36.4
24.8
TextRegion
70.9
39.1
41.1
84.4
43.2
28.7
42.8
22.8
46.6
Training-Free, OpenCLIP VIT-H/14
SCLIP (Wang et al., 2024a)
43.8
23.5
24.6
67.5
25.6
16.8
19.5
11.3
29.1
ProxyCIIP
CLIP (Lan et al., 2024b)
65.0
35.4
38.6
83.3
39.6
26.8
42.0
24.2
44.4
Trident (Shi et al., 2024)
70.8
40.1
42.2
88.7
44.3
28.6
47.6
26.7
48.6
Tex:Rogion
73.1
41.2
40.6
89.5
46.1
31.2
47.0
27.3
19.5
Table 1: Open-world Semantic Segmentation. Evaluation results (mloU) on semantic segmentation benchmarks. TextRegion consistently achieves superior or competitive performance across various benchmarks and backbone architectures. The best results each setting are bolded.
4.1 Open-world Semantic Segmentation
Dataset. We evaluate on six widely used semantic segmentation benchmarks: PASCAL VOC 2012 (Ever- ingham et al., 2015), PASCAL Context (Mottaghi et al., 2014), COCO-Stuff (Caesar et al., 2018), COCO- Object (Lin al., 2014), Cityscapes (Cordts et al., 2016), ADE20K (Zhou et al., 2019). For VOC and Context, we test the settings without (VOC20, Context59) and with (Voc21, Context60) background labels. Since the image resolutions vary across datasets, follow the design of previous works by sctting different resizing scales for each dataset. Specifically, the shorter side is resized to 672 pixels for VOC20, VOC21, COCO-Object, Context59, ADE, and Context60; 896 pixels for COCO-Stuff; and 1344 pixels for Cityscapes.
Results. Tab. shows our open-world semantic segmentation performance. The primary distinction between our method and baselines lies in the adoption of a region-level classification strategy prior to segmentation. Specifically, we first classify those region tokens, and then propagate the region logits back to the pixel-level segmentation predictions, leveraging the fact that our region tokens are inherently associated with high- resolution segmentation masks. In contrast, other methods typically conduct predictions at the patch level and subsequently rely on upscaling additional post-processing to produce high-resolution segmentation results. Benefiting from the open- voon-vocabblrry classification capacity preserved in our region tokens and the strong segmentation capabilities of SAM2, TextRegion on effectively simplifies the dense prediction task into an instance- level sparse prediction problem, which is both easier to solve and more robust. Despite being entirely training-free and simple, our approach consistently achieves superior or competitive performance.
4.2 Zero-shot Referring Expression Comprehension
Dataset. We evaluate on RelCOCO (Yu al., 2016), RelCOCO+ (Yu al., 2016) and RefCOCOg (Mao et al., 2016) datasets. All of these benchmarks come from the MS-COCO (Lin et al., 2014) dataset, paired with expressions that refer to a unique object in each image, accompanied with bounding box. The main difference is that RefCOCO+ excludes relation-based expressions (e.g. "left of", "closer", or "bigger") and RefCOCOg has longer descriptions. The test. sets of RefCOCO and ReICOCO+ are divided into "LestA" and "testB," " which contain only people and non-people instances, respectively.
Setting. Referring Expression Comprehension (ReC) focuses on identifying the most relevant object (bounding box) in image based on given natural language expression. A common zero-shot ReC setting
7

---

Published in Transactions on Machine Learning Research (11/2025)
RefCOCO
ReSCOCO+
RefCOCOg
Method
Backbone
Val
Test A
TestB
Val
Test
TestH
Val
Test
CPT (Yao et al., 2024)
VinVL (Zhang et sl, 2021)
32.3
36.1
30.3
31.9
35.2
28.8
36.7
36.5
TextRegion
SigLIP2-L/16 (Tschanneu et al. 2025)
45.0
19.8
36.0
51.3
57.1
40.5
53.8
52.6
TextRegion
PIE-Core-//14 (Bolya et al,, 2025)
47.3
53.7
37.9
52.6
59.7
42.2
52.7
50.7
Red Circle (Sitedrittkk et al,, 2023)
CLIP VIT-L/119336
38.0
15.3
32.9
13.9
51.0
37.1
47.2
47.3
FGVP (Yaug et al., 2023)
CLIP VIT-L/1403330
46.1
53.0
40.4
50.4
57.5
42.6
54.5
54.1
TextRegion
CLIP VIT-L/146336
48.7
56.4
40.8
53.6
60.8
44.3
55.8
54.6
Table 2: Zero-shot Referring Expression Comprehension. TextRegion is compatible with different imago-text models and consistently achieves strong zero-shot results across all benchmarks.
leverages a pretrained object detector to generate object proposals and then selects the most likely proposal corresponding to the query. The accuracy is evaluated by the percentage of instances where the selected proposal has an Intersection-ove--nnnon (IoU) of at. least 0.5 with the ground-truth bounding box. For this experiment, we do not incorporate any spatial relationship refinement or post-processing methods commonly used in ReC. Retrieval accuracy is computed using object proposals producod by MAttNet (Yu al., 2018). All baseline results are laken from FGVP (Yang al., 2023). For fair comparison, we evaluate TextRegion with the same set of object proposals as those used by FGVP.
For our implementation, we first generate region tokens for the input image and then compute the cosine similarity between the text-aligned region tokens and the query text embedding. The region with the highest similarity is selected as the retrieved object, and its corresponding bounding box is determined. To select the most likely candidate from the detected proposals, we calculate the IoU between our predicted region bounding box and all proposal bounding boxes, selecting the proposal with the highest IoU, In cases where none of the proposals overlap with the selected region, we directly return the region's 's bounding box as the retrieval result.
Results. Tab. 2 shows the results for zero-shot ReC. Complex baselines depend on carefully designed pipelines and prompting strategies (Yao et al., 2024; Shtedritski et al., 2023), and using bounding box candidates as prompts for SAM to obtain better object masks (Yang et al., 2023). In contrast, our method directly generates region lokens without relying on candidate bounding boxes and still achieves superior performance.
4.3 Multiple Object Grounding
Setting. In Sec. 4.2 we test the reforring expression comprehension performance of our method. But since that lask involves referring to a single larget object, we are interested in testing our method's multi-object grounding capabilities. For this test, we use the Reasoning Segmentation dataset (Tai et al., 2024) which is commonly used to assess the visual segmentation capabilities of VLMs. As some samples in the dataset contain multiple objects related to given qucry, simply returning the region with the highest similarity is not effective and can overlook relevant objects.
To assess the relevance of a region to a given query, we create a negative label to get a score for the query. Specifically, for each input query, we generate a pscudo-conten contrastonett query to compute region similarities relative to both the original and contrastive queries. Regions whose similarity to the original query exceeds their similarity to the pscudo-contrastive query are retricved as the selected objects. The pscudo-contrastive query we use for the original reasoning segmentation query is "Background, any other thing""
Since the original queries in the reasoning segmentation dataset are long and complex, construct a simplified secnario. Specifically, we use the original LLaVA115-BB (Liu et al., 2024a) to generate text answers for the original queries, and then treal these answers as interpreted queries to retrieve relevant objects. This scenario is designed to evaluate the impact of language understanding ability. Further details are provided in Appendix See A.4. We implement bascline results for Trident (Shi et al., 2024), while the results for all other basclines are sourced from LISA (Lai et al., 2024). We use CLIP ViT-L/ 141//114363p as the backbone for this experiment.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Original: In
Short query
Long query
Overall cloU
important to have nccess to fire safety equipment. What abject in the picture specifically designed store release fire extinguishing substances?
Method
gloU
cloU
gloU
cloU
gloU
Train LLa VA1.5-72
LISA (Tai et al., 2024)
47.1
48.5
49.2
48.9
48.7
48.8
mraining-bused. (No VLM used)
OVSeg (Liang et al., 2023)
18.0
15.5
28.7
22.5
26.1
20.8
X-Decoder (Zou al., 2023a)
20.4
11.6
22.2
17.5
21.7
16.3
SEEM (Zou et al., 2023b)
20.1
11.5
25.6
20.8
24.3
18.7
Grounded-SAM (Liu al., 2024b)
17.8
10.8
22.4
18.6
21.3
16.4
Training- (No VLM used)
Interpeeted: fire extinguisher
Trident (Shi et al., 2024)
19.9
12.9
24.5
23.1
23.3
20.0
eeginguisher
TexiRegion
21.6
15.6
26.4
24.1
25.2
21.6
With interpneted queries from the original LLa VA1.5-7B
OVSeg (Liang et al., 2023)
24.2
18.7
44.6
37.1
39.7
31.8
Trident (Shi al., 2024)
23.0
24.2
43.3
42.5
38.4
39.2
Reginn
28.5
30.6
47.2
45.3
42.7
42.4
Table 3: Multiple Object Grounding. The last three rows show results using interpreted queries. TextRegion demonstrates significant performance gains when given LLa LLaVA-interpreted queries, outperforming baselines by large margin.
Figure 5: Top: Query im- age; Bottom: Grounding re- sults. The title shows the original interpreted query.
Metrics. There are two metrics for this task: gloU and cloU. gloU is defined as the average of per-image Intersection-ovve--nnino (IoUs), while cloU is computed as the cumulative intersection divided by the cumulative union. Since cloU heavily biased toward large-area objects and exhibits high variability, gloU is usually preferred (Lai et al., 2024).
Results. As shown in Tab. 3, TextRegion achieves performance comparable to training-based methods. The performance improves significantly when interpreted queries are used, especially for long queries. This suggests that the main bottleneck for TextRegion in reasoning regmentation stems from the limited language understanding of the image-text models' text encoder, rather than from its object retrieval capability.
5 Ablation and Discussion
Global Patch, Multi-resolution and CLIP variants. We validate the effectiveness of the proposed global patch removal and multi-resolution patch feature strategies in Tab. Overall, these strategies benefit range of image-text models, from small to large backbones. When using both techniques, the performance is generally the best compared to using alone. An interesting observation is that the performance of SigLIP2-
COCO Stuff
AED20K
Global Multi
Perception
SigLIP2
CLIP
Perception
SigLIP2
CLIP
B/16
L/14
B/16
S0/16
B/16
H/14
B/16
L/14
B/16
S0/16
B/16
H/14
24.0
23.8
23.6
22.4
26.9
28.7
20.5
23.3
23.3
23.7
20.6
24.1
25.1
24.2
24.7
23.7
27.0
28.6
21.1
23.6
23.9
24.6
20.7
23.8
/
27.7
25.4
26.6
21.4
28.8
30.6
24.1
24.3
25.3
21.5
22.7
26.5
/
27.9
26.1
26.8
23.2
28.7
31.2
24.3
24.6
26.0
22.1
22.8
27.3
Table 4: Ablation on Global Patch Removal and Multi-Resolltion Feature for CLIP Variants. Combining global patch removal with multi-resolution features achieves the best performance in most Our default setting is marked in green.

---

Published in Transactions on Machine Learning Research (11/2025)
Feature Stuff
ADE
Layer
Stuff
ADE
Interp mIoU s/img
Mask
Stuff
ADE
Input
0.9
1.1
-5
0.2
0.4
Output
5.7
3.0
-2
18.2
11.6
Up Down
10.2
0.33
Hard
28.5
22.7
Value
28.7
22.8
-1
28.7
22.8
28.7
0.20
Soft
28.7
22.8
Table 5: Attn Feature.
Table 6: Value Layer
Table 7: Interpolate.
Table 8: Mask Value.
S0400M/16 drops when using the multi-rcsolution patch feature, whereas all other models benefit from it. A possible explanation is that large backbones, while powerful, tend to generate more global patches (Xiao et al., 2023; Darcet et al., 2023; Shao et al., 2024). When applying the multi-resolution feature alone, this effect may amplified simpler images contain less detailed information, causing more patches to capture global features. Our global patch removal strategy helps mitigate the negative effects, so when combined with multi-resolution patch features, it consistently delivers better or comparable performance than using multi-resolution alone.
Another interesting finding is that the global patch phenomenon is less severe in CLIP compared to the perception encoder and SigLIP2. As a result, using global patch removal alone for CLIP has a limited impact. However, when combined with multi-resolution features, it significantly boosts performance, particularly for CLIP ViT-H/14.
Design choice for mask-based attention pooling. We evaluate different design choices for mask-based allention pooling using CLIP ViT-B/16. Tab. 5 compares the effect of different feature used for pooling, "Input" refers to the feature map fed into the final attention block, "Output" denotes the output of that block, and "Value" means the attention values from the last attention block. Interestingly, even with a single projection, transitioning from the input to the value features significantly improves performance from almost zero to decent results.
Tab. 6 shows the effect of pooling attention value from different layers. Results show that the value features from the second-to-last layer yields substantially better performance than directly pooling the input feature map (Tab, 5) for the last attention block.
Tab. 7 shows the results of either upscaling the feature maps or downscaling the SAM2 masks when pooling region tokens. Experiments are conducted on COCO-Suff. The reported "Up" result use a 5x upscaling of the value feature maps instead of directly matching the shape of the SAM2 masks, as the latter approach can induce much larger GPU consumption, particularly when combined with the multi-resolution strategy. "s/img" means seconds required to predict single image, measured on one A100 GPU. The results show that downsampling offers both effectiveness and efficiency advantages. Tab. 8 shows the impact of using hard masks versus soft masks for aggregating region tokens and computing region logits.
Sensitivity to Mask Quality. Tab. reports the impact of different mask generators. "Ground Truth Masks" means using the annotated ground-truth masks to compute region tokens, serving as the upper bound of TextRegion. In contrast, SLIC (Achanta et al., 2012) partitions images into superpixels based on
COCO Stuff
AED20K
Mask Generator
Perception
SigLIP2
CLIP
Perception
SigLIP2
CLIP
Ground Truth masks
34.7
33.7
36.5
32.7
35.4
31.0
SLIC (Achanta et al., 2012)
21.7
19.9
21.3
18.4
18.2
16.1
SAM (Kirillov et al., 2023)
27.0
26.1
27.9
24.3
25.8
22.1
SAM2 (Ravi et al., 2024)
27.9
26.8
28.7
24.3
26.0
22.8
Table Ablation on the effect of mask generators. "Ground Truth masks" refer using the annotated masks to extract region tokens, which subsequently used for prediction. All experiments are conducted with image-text model based on the ViT-B/16 backbone. The default configuration is highlighted in green.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Input Image
Ground Truth
GT Masks
SLIC
SAM
SAM2
VOC21
COCO Context60
Figure 6: Visualization comparison of different mask generators across datasets. "GT masks" refer to using the annotated masks to extract region tokens.
color and spatial cues, resulting in lower performance than SAM and SAM2. These results demonstrate that TextRegion is compatible with various segmentation methods, enabling object-level predictions for flexible, user-specified mask inputs. We also provide visual comparisons of those mask generators in Fig.
6
Conclusion
TextRegion is simple, effective, and training-free approach for obtaining text-aligned region tokens by com- bining image-text models with SAM2. The image-text model provides visual-language alignment capabilities, while SAM2 supplies detailed object spatial boundaries. Our region tokens show strong zero-shot performance open-world semantic segmentation, referring expression comprehension, and grounding. Our method is also highly ficxible, supporting casy intogration with various imago-text modcls, all which perform well in zero-shot settings. Moreover, TextRegion imposes no constraints on the masks, users can manually define custom masks to generate region tokens for any objects of interest, beyond those provided by SAM2.
The quality of the region tokens depends the accuracy of the region masks. Poor masks will produce low-quality region tokens. Additionally, the effectiveness of region tokens is limited by the original visual feature of the image-text model. If these feature lack spatial awareness or advanced language understanding, the region tokens will also be unable to capture such information.
Limitations:
Acknowledgement: This work is supported in part by ONR N00014-23-1-221 and DARPA HR0011-23-9 0060. This work used NVIDIA GPUs at NCSA Delta through allocation CIS240059 and CIS250059 from the Advanced Cyberinfrastruchure Coordination Ecosystem: Services Support (ACCESS) program, which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296. The views and conclusions expressed are those of the authors, and not necessarily representative of the US Government its agencies.
References
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared state-of-the-ar superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2274-22)) 2282, 2012. doi: 10.1100/TPAMM.2012.2222
11

---

Published in Transactions on Machine Learning Research (11/2025)
Daniel Bolya, Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not. at the output of the network. arXiv preprint arXiv:2504.13111, 2025.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1209-1218, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on compuler vision, pp. 9650 9660, 2021.
An-Chich Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language models. arXiv preprint S5t::2400.015..00 2024.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint ar xrXx::2309.6688: 2023.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskelip: Masked self-distillation advances contrastive language-inage pretraining. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10995-11005, 2023.
Mark Exeringham, SM Ali Eslami, Luc Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-136, 2015.
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Sanghyun Jo, Soohyun Ryu, Sungyub Kim, Eunho Yang, and Kyungsu Kim. Tid: Text-lag self-distillation enhancing image-text alignment in clip to alleviate single tag bias. In European Conference Computer Vision, pp. 341-357. Springer, 2024.
Dahyun Kang and Minsu Cho. In defense of lazy visual grounding for open-vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 143 164. Springer, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023.
Nikolaus Korfhage, Markus Mühling, and Bornd Freisleben. Scarch anything: sogmentation-baset similarity search via region prompts. Mullimedia Tools and Applications, pp. 1 1-26, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision 2024.
and Pattern Recognition, pp.
Mengcheng Lan, Chaofeng Chen, Viping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision- languaee inference. In European Conference on Computer Vision, pp. 143 160. Springer, 2024a.
Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. In Europeon Conference on Computer Vision, pp. 70-88. Springer, 2024b.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Jungboom Lee, Sanghyuk Chun, and Sangdoo Yun. Toward interactive regional understanding in vision-large language models. orXiv preprint ar arXiv:2403. 18260, 2024.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 7061-7070, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Asion-FCCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved basclines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 19296-26300 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 48-55. Springer, 2024b.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the TEFE conference computer vision and pottern recognition, pp. 11 20, 2016.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the TEEE conference on computer vision and pottern recognition, pp. 891- 109-898,45 2014.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Xiv preprinl or Xiv::2304 071193 2023.
Ting Pan, Lulu Tang, Xinlong Wang, and Shiguang Shan. Tokenize anything via prompting. In Europeon Conference on Computer Vision, pp.
Springer, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PmLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 18082-18091, 2022.
Nikhila Ravi, Valentin Gabeur, Yuan- Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyn Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, el al. Sam 2: Segment anything in images and videos. arXiv preprint Xrx: 00714, 2024.
Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training- free open vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 139-156. Springer, 2024.
Yuheng Shi, Minjing Dong, and Chang Xu. Harnessing vision foundation models for high-performance,, training-fres free open vocabulary segmentation. arXiv preprint arXin:2411.09219 2024.
Michal Shlapentokh-Rotten Rothman, Ansel Blume, Yao Xiao, Yuqun Wu, Sethuraman TV, Heyi Tao, Jae Yong Lee, Wilfredo Torres, Yu-Xiong Wang, and Derck Hoiem. Region-based representations revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 17107-17116, June 2024.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 11987-119911 2023.
Mankeerat Sidhu, Hetarth Chopra, Ansel Bhume, Jeonghwan Kim, Revanth Gangi Reddy, and Heng Ji. Search and detect: Training-free long tail object detection via web-image retrieval. arXiv preprint 10X::2240.18733, 2024.
Vladan Stojnié, Yannis Kalantidis, Jiří Matas, and Giorgos Tolias. Lposs: Label propagation over patches and pixels for open-vocabulary semantic segmentation. In Proceedings of the Compuler Vision and Pullern Recognition Conference, pp. 1794-9808) 2025.
Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. arXiv preprint ar Xrx::22004.00 05991, 2022.
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjum Xiong, Dahua Lin, and Jiaqi Wang. Alpha-clip: A clip model focusing on wherever you want. In Proceedings of the IEEE/CVF conference on compuler vision and poltern recognition, pp. 13019 13029, 2024.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, arviv preprint Xiv: 14786, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Feng Wang, Jieru Mci, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In European Conference on Computer Vision, pp.
Springer, 2024a.
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. In European Conference on Computer Vision, pp. 315-332. Springer, 2025.
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. Sam-clip: Merging vision foundation models towards semantic and spatial understanding. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pollern Recognition, pp. 3635 3647, 2024b.
Monika Wysoczaúska, Oriane Siméoni, Michaël Ramamonjisoa, Andrei Bursue, Tomasz Trzciáski, and Patrick Pérez. Clip-dinoiser: Teaching clip few dino tricks for open-vocabulary semantic segmentation. In European Conference on Computer Vision, pp. 320-337 00-3377.60 Springer, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. Xrxiv preprint arXin: 1 17453, 2023.
Lingfeng Yang, Yueze Wang, Xiang Li, Xinlong Wang, and Jian Yang. Fine-grained visual prompting. 2023.
Advances in Neural Information Processing Systems,
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingvao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792 19802, 2025a.
Yuhang Yang, Jinhong Deng, Wen Li, and Lixin Duan. Resclip: Residual attention for training-free dense vision-language inference. In Proceedings of the Computer Vision and Pallern Recognition Conference, pp- 29968-20 29978, 2025b.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. AT Open, 5:30-38, 08, 2024.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision -annn--CCVVV- 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 11 14, pp. 69 85. Springer, 2016.
Licheng Yu, The Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L. Berg. Mattnet: Modular attention network for referring expression comprchension. In Proceedings of the IEEE conference compuler vision and pollern recognition, pp. 1307-137 2018.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986 2023.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognilion, pp. 5579 5588, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127: 302-321, 2019.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference Computer Vision, pp. 696 712. Springer, 2022.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15116-15127, 2023a.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36:19769 19782, 2023b.
15

---

Published in Transactions on Machine Learning Research (11/2025)
A Appendix
A.1 Summary of contents
Sec. A.2 details the hyperparameters used our experiments. In Sec. A.3, we describe how generate text-aligned region tokens using the frozen SigLIP2 (Tschannen et al., 2025) and Perception Encoder (Bolya et al., 2025). Sec. A.4 explains the process of generating interpreted querics for Reasoning Segmentation. Finally, Sec. A.5 includes further visual examples for open-world segmentation and referring expressions, well as illustrative failure cases.
A.2 Hyperparameter
For all experiments, filter global patches using threshold of 0.07. The crop size is uniformly set to 336 for all CLIP models (ViT-B/16 through ViT-H/14), while SigLIP2 and Perception Encoder use their respective default input resolutions. Region masks are generated with SAM2 (Ravi et al., 2024) Hiera-Large, using the following configuration: pred-iou-thresh set to 0.6, stability-score 0.6, box-mme-thresh to 0.9, and points-per-side Cityscapes dataset, we increase points-per-side to 36 to due to its high resolution and the abundance of small objects. To mitigate the impact of duplicated overlapping masks, also merge masks with an overlap IoU greater than 0.8.
to 16. In the semantic segmentation experiments on the
A.3 CLIP Variants
Unlike CLIP (Radford et al., 2021), which retains the [CLS] token across multiple layers, SigLIP2 (Tschannen et al., 2025) and Perception Encoder (Bolya et al., 2025) introduce learnable query token, denoted as [q], to aggregate global image information within the final attention pooling block. Let X denote the input to this last attention pooling block:
CLS = Attn
XW
XW
(6)
key
value
Note that the core idea of TextRegion is to restrict the query's attention to region-relevant patches. To simplify the implementation, instcad of explicitly computing the value and then do pooling, we reformulate region-aware pooling by directly masking out irrelevant patches during the attention compulation. Given the SAM2 down-sampled region mask m, R.N. the text-aligned region token CLSS, is computed as:
CLS, Alin (q, XW XW.), attention mask (- inf (m, == 0))
X denotes the average of all patch features, to reduce dependence on the original attention scores and better approximate our mask-based attention pooling in CLIP, where mask values within region are typically close to one. Experiments show that using the original XW as the key also works well. Although we have not conducted a comprehensive comparison between the two approaches, visualizations suggest the averaging method performs slightly better. Given its alignment with our CLIP implementation which avoids relying on original attention scores we adopt it as the default for both SigL.IP2 and the Perception Encoder.
Recall that m, is obtained by bi-linearly downsampling M,, where the original M, E [-32, but are clamped to [0, before use. As result, each downsampled value m,, [0, 1] represents the likelihood that patch i belongs to the region r. When m,, = 0, patch i is considered irrelevant to the region and should not be attended to by the region query. To enforce this constraint, we apply an atlention mask that excludes patches with my == 0 from participating in the attention compulation.
Another modification for SigLIP2 (Tschannen et al., 2025) and the Perception Encoder (Bolya et al., 2025) is that we change the final value feature to Vinal Vnigh + 0.5 upsample (Vow) for the multirresolution strategy, they more sensitive to the influence of global patches.
16

---

Published in Transactions on Machine Learning Research (11/2025)
A.4 Interpreted Queries for Multi-Object Grounding Reasoning Segment
To convert the original long and complex queries in the reasoning segmentation dataset into interpretable queries compatible with the CLIP text encoder, we use VA (Liu et al., 2024a) for query preprocessing Specifically, we input the paired image and query into LLa and obtain concise answer using the prompt: "Please summarize the answer in five words or fewer to define the object." The answer is then used as the interpreted query to obtain our multi-object grounding results. See Fig. 7 and Fig. 8 for more examples of interpreted queries.
This interpreted setup allows us evaluate visual grounding performance independently of advanced language understanding capabilities. Since the reasoning grounding performance depends both the interpreted queries and the object grounding capability of TextRegion, also present case in Fig. 8 (example where LLaVA produces an incorrect interpreted query.
The template used to construct the contrastive query for the interpreted new query is: "Background, anylhing but finterpreted query)". the main paper, we also evaluate the grounding performance on the original complex queries. We do not apply this template ("Background, anylhing but {original query] ") to get the contrastive queries because the original query are too long, and adding them the contrastive query would further burden the text encoder. Instead, we simply use 'Background, any other thing" the contrastive query for these cases.
(a) Original: Generally speaking, dogs do not have horns on their heads, only a pair of ears. What part of the dog's head in this picture looks strange?
antlers
(b) Original: Driving at night can be very dangerous due to poor visibility, which can lead to accidents. What part of the car needs to be turned on when driving at night?
headlight
Figure Examples for Interpreted Queries (Part 1): The title means the original query from the reasoning segmentation dataset, while the legend presents the interpreted query generated by LLaVA 1.5. The left column displays the query image. The middle column shows the ground truth, with green highlighting the target grounding area, and red indicates areas that can be ignored. The right column presents the grounding rosult.
17

---

Published in Transactions on Machine Learning Research (11/2025)
(c) Original: In a rural landscape, what objects in the picture could provide shade and shelter for animals or humans?
1: trees
(d) Original: a car with a color that is closer to lipstick color
ferrari
(e) Original: To keep bread fresh and protected, it is often placed in a protective covering. What item in the picture is commonly used for this purpose?
1: plastic bag
f Original: the shadow of the red car
1: car
Figure 8: Examples for Interpreted Queries (Part 2): The middle column shows the ground truth: red highlights areas that can be ignored, and green indicates the target grounding area. Example (l) shows a failure case where LLa. provides an incorrect answer to the original query.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Input Image
GT
TextRegion
Figure Visualization for open-world segmentation. "GT" means the ground truth, while "TextReeiinn " shows our predicted results.
person holding
dish in top right corner
child sitting on womans lap
psn boy
left guy
umbrella
GT
TextRegion
Figure 10: Visualization for referring expressions. The sentences above are the query expressions. The red bounding box in the "GT" row indicates the referred object.
A.5 Visualization
Fig. 9 and Fig. 10 presents additional visualizations for open- worldworld segmentation and referring expression tasks. From the visual results, we observe that while SAM2 can segment objects with relatively accurate boundaries, it struggles with objects exhibiting complex contours. For instance, in the train example of Fig. 9, TextRegion produces slight pixel-level inaccuracies along the intricate edges.
For referring expressions, the example of "persn holding umbrella" shows that the text encoder may sometimes misinterpret the expression, leading the model to predict related object rather than the specific target described.
19


Reviews:
[{"review_id": "bi50f0leR5", "reviewer": "Reviewer_h3J4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "TextRegion is a training-free plugin that makes frozen vision-language models work at the region level. It uses SAM2 to cut an image into soft masks, then replaces the model’s last global pooling with mask-weighted attention so each segment gets its own text-aligned \"region token.\" Two practical tweaks, multi-resolution features and filtering out overly global patches, keep small objects sharp and localization crisp. This reframes dense prediction as classifying a handful of segments and painting scores back to pixels, delivering strong zero-shot results on open-vocabulary segmentation, referring expressions, and multi-object grounding across CLIP, SigLIP2, and Perception Encoder backbones.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The paper supports its claims with clear experiments on several standard datasets for segmentation, referring expressions, and multi object grounding. It compares to strong baselines using the same backbones and often wins or matches them. Ablation tests show why each part of the method helps. The method is described in enough detail to reproduce, and the authors state limits and scope so the evidence matches the claims.", "audience_yn": "Yes", "audience_explanation": "This work offers a simple, training free way to get region level understanding from frozen image text models using segmentation, with strong zero shot results on common benchmarks. Many TMLR readers in vision and multimodal learning will care because the method is practical, general across backbones, and useful as a baseline and tool for open vocabulary segmentation and referring expressions.", "requested_changes": "Research questions and hypotheses: Add 2–4 clear research questions, each with a testable hypothesis and planned metric. Rewrite the introduction to present them, then map each question to datasets, metrics, and ablations in an experimental design section. Tie the conclusion back to these questions.\n\nMethod clarity: Explain how masks are made and used. Say if masks can overlap, whether weights are soft or hard, how you downsample to the patch grid, how you handle boundaries and tiny regions, the typical number of regions per image, how the region token is formed in attention, and how region scores are sent back to pixels. Include brief pseudocode or a step diagram.\n\nReproducibility and reporting: Release configs and code. List label sets and prompts for each dataset, all thresholds and constants, image sizes and tiling choices, data splits and evaluation rules, and random seeds. Include a simple inference script and a table of all hyperparameters.\n\nSensitivity to mask quality: Compare SAM2 with another segmenter and with ground truth masks as an upper bound. Add tests with eroded, dilated, and noisy masks. Report performance versus region IoU.\n\nReferring expressions clarity and scope: Explain how proposals or boxes are obtained. State whether spatial relation phrases are in scope. Provide a small analysis split for spatial phrases and report accuracy by phrase type. Note any query simplification or other heuristics used.", "broader_impact_concerns": "Because the method reuses large pretrained vision and language models and segmentation masks without new training, it inherits their biases and errors. Region level outputs can attach labels to people or personal objects and may be used for surveillance, profiling, or searching images for sensitive content without consent. Open vocabulary queries can also infer sensitive attributes, and mistakes could lead to unfair outcomes. A short Broader Impact section should acknowledge these risks and include simple guardrails such as filtered vocabularies, clear use policies, human review in sensitive settings, and a basic bias and privacy audit.", "additional_comments": "The paper is engaging and the core idea is neat. After you add clear research questions and hypotheses, consider a short method at a glance box with 5 to 7 numbered steps so readers can grasp the pipeline quickly.\n\nKeep notation consistent across sections. Define the image, the set of regions, the mask for each region, and the region token once, and reuse the same symbols everywhere.\n\nClarify background handling for segmentation. Report results with and without background, state how background scores are computed, and note any calibration.\n\nState how you resolve ties or conflicts when a region matches several labels. Give the tie break rule and the confidence threshold.\nFor referring expressions, spell out how proposals or boxes are produced, how many you keep per image, and any dedup or NMS you apply.\n\nProvide a short glossary of key terms like region token, mask weighting, and global patch removal to help new readers.", "extra_scores": {}, "extra_text": {}}, {"review_id": "np8NVR2fvo", "reviewer": "Reviewer_dphS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method that, at its core, performs grounding (detailed) captioning. The method is further applied to various tasks, including open-world semantic segmentation, referring expression comprehension, and grounding.\n\nThe solution is based on a CLIP backbone combined with SAM segmentation. The tokens are then matched with the regions extracted by SAM via CLIP’s attention module. Finally, these region tokens are refined using two additional methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The method description is clear, allowing the technical claims to be isolated and verified. The paper presents an extensive set of experiments (Tables 1, 2, and 3), which I have cross-checked against the cited references, confirming the reported results.\n\nOverall, the paper may be criticized for its limited novelty, as it essentially combines known methods using relatively simple connections. However, since these “simple connections” have not appeared in previous work, one can argue that they are not as obvious as they might seem. The results themselves are strong.\n\nAnother aspect that could improve the paper’s impact is publishing the code. While the method is explained reasonably well, certain details can only be fully understood by examining the implementation.", "audience_yn": "Yes", "audience_explanation": "The method presents noticeable improved results in three areas that represent themes of interest for the machine learning community.", "requested_changes": "Change requested: Show more visual examples of in the supplementary material for each direction investigated/tested. Add failures (or not so good results). \n\nI have not identified minor issues (text, figure, table problems) that need adjustment. Some things may be explained differently (in my view better such as the method, making figure 2 more informative), but this is a subjective perspective. I have understood the key elements, thus the paper is fine.", "broader_impact_concerns": "In my view, the method does not raise additional concerns on the ethical implications.", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}, {"review_id": "vyug0P31UE", "reviewer": "Reviewer_g48z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present a framework that leverages image-text models to generate region tokens for enhanced visual understanding in referring expression comprehension tasks. The proposed method substitutes the attention weights in the final layer with region-specific tokens derived from SAM2 to direct the text encoder.\n\nStrengths: \n* Training-free framework that can be readily adapted to enhance pre-trained image-text models.\n* Superior performance compared to text embedding models that require additional training.\n\nWeaknesses:\n* Heavy reliance on region tokens from SAM2. Although the authors evaluate different image-text models (CLIP, SigLIP2, OpenCLIP etc.), they do not investigate the framework's robustness when using region tokens from alternative segmentation models, such as the original SAM or other less capable variants.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors compare their framework against both training-free and trained methods. \n\nInsufficient empirical support: While the reviewer acknowledges that certain patches likely indeed contain global information, the authors make an unsubstantiated claim by asserting that tokens lacking patch-level local information must necessarily represent global information, without providing adequate empirical evidence to support this binary classification.", "audience_yn": "Yes", "audience_explanation": "Text-image models are of interest to a portion of the TMLR audience", "requested_changes": "The authors claim certain patches contain 'global' information but provide no evidence beyond showing that patches with low local similarity (below τ = 0.07) are classified as 'global.' They should demonstrate that these patches actually encode global semantic content rather than simply being noisy or representing object boundaries.", "broader_impact_concerns": "N/A", "additional_comments": "The use of imagery containing weapons in Appendix A.4 may be inappropriate for an academic paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)