MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image Segmentation
Authors: Minhyun Lee
Seungho Lee
Song Park
Dongyoon Han
Byeongho Heo
Hyunjung Shim
Venue: Accepted by TMLR ()

Abstract: Referring Image Segmentation (RIS) is an advanced vision-language task that involves identifying and segmenting objects within an image as described by free-form text descriptions. While previous studies focused on aligning visual and language features, exploring training techniques, such as data augmentation, remains underexplored. In this work, we explore effective data augmentation for RIS and propose a novel training framework called Masked Referring Image Segmentation (MaskRIS). We observe that the conventional image augmentations fall short of RIS, leading to performance degradation, while simple random masking significantly enhances the performance of RIS. MaskRIS uses both image and text masking, followed by Distortion-aware Contextual Learning (DCL) to fully exploit the benefits of the masking strategy. This approach can improve the model's robustness to occlusions, incomplete information, and various linguistic complexities, resulting in a significant performance improvement. Experiments demonstrate that MaskRIS can easily be applied to various RIS models, outperforming existing methods in both fully supervised and weakly supervised settings. Finally, MaskRIS achieves new state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets. Code is available at https://github.com/naver-ai/maskris.

Published in Transactions on Machine Learning Research (11/2025)
MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image Segmentation
Minhyun Lee
mbb315.led@samasng.comm
AT Center, Somsung Electromics
Seungho Lee"
sh622.lee@samssnn.comm
AI Center, Somsung Electronics
Song Park
song.pazk@@gmail.com
Dongyoon Han
NAVER Al Lab
Byeongho Heo
Wh.heo@@naaercorrpccom
NAVER Al Lab
Hyunjung Shim
kaleshim Wkwist. ac.
Korea Advanced Institute of Science 8 Technology (KAIST)
Reviewed on OpenReview: https: openreview. net/ forum? id= EtX4madHHoe
Abstract
Referring Image Segmentation (RTS) an advanced vision-language task that involves iden- tifying and segmenting objects within an image as described by freo-form text descriptions. While previous studies focused aligning visual and language features, exploring train- ing techniques, such as data augmentation, remains underexplored. In this work, we plore effective data augmentation for RIS and propose a novel training framework called Masked Referring Image Segmentation (MaskRIS). We observe that the conventional image augmentations fall short RIS, leading to performance degradation, while simple ran- dom masking significantly enhances the performance of RIS. MaskRIS uses both image and text masking, followed by Distortion-aware Contextual Learming (DCL) fully ex- ploit the benefits of the masking strategy. This approach can improve the model's robust- ness to occlusions, incomplete information, and various linguistic complexities, resulting in significant performance improvement. Experiments demonstrate that MaskRIS can easily be applied to various RIS models, outperforming existing methods in both fully su- pervised and weakly supervised settings. Finally, MaskRIS achieves new state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets. Code available at https://github. htwpgihtwb..c..aaeeeeeee... ammm/naver-aimmakriis..
1 Introduction
Referring Image Segmentation (RIS) (Hu et al., 2016) involves precisely delineating objects within an image based on text descriptions. Unlike semantic and instance segmentation, which are constrained to pre-defined classes, RIS offers unique flexibility by segmenting objects specified by free-form expressions. This capability has led to extensive applications in diverse domains, such as languago-driven human-robot interaction (Shah al., 2023) and advanced image editing (Chen al., 2018; Patashnik et. al., 2021). One of the main
"Equal contribution. Work done during internship at NAVER Al Lab. *Corresponding authors.

---

Published in Transactions on Machine Learning Research (11/2025)
Target class: umbrella
The target objects identical after DA
Image
GT
Random Crop
Horizontal Flip
Color Distortion
(a) Semantic Segmentation
Original GT
Text description
"lady under red umbrella on left."
Human answer
Image
GT
Random Crop
Horizontal Flip
Color Distortion
The referred object is changed disappears after DA
(b) Referring Image Segmentation
Figure 1: Conventional data augmentations (DA) in semantic segmentation are incompatible with referring image segmentation. Random crop and horizontal flip could change the referred object (e.g., "lady under the red umbrella left") to another one, and color distortion could make the described object disappear.
challenges in RIS is effectively bridging the modality gap between visual and language features. Recent research (Liu et al., 2023c; Yang et al., 2022; Wang et al., 2022) has focused on developing sophisticated architectures for cross-modal alignment.
While significant progress has been made in aligning different modalities, effective training techniques for RIS remain underexplored. The trend toward developing larger and more complex models requires challenging training settings like data augmentation. Previous studies (Cubuk et al., 2018; 2020; Touvron et al., 2021) have demonstrated the crucial role of data augmentation in model training. However, designing data augmen- lalion for RIS is challenging: standard augmentatiin used in semantic segmentation is often incompatible with RIS, as some textual descriptions directly conflict with augmentation.
As illustrated Figure 1, spatial augmentation, such as random crop and horizontal flip, alters the mean- ing of "left", causing the larget. mask to become incorrect afler augmentation. Similarly, color distortion also conflicts with color-specific descriptions like "red". Due the incompatibilities, naïvely applying aug- mentations is not beneficial
empirical analysis reveals that conventional data augmentation degrades RIS (Liu et al., 2023c) performance (Figure 3(a)). This finding explains why previous RIS studies (Kim et al., 2022; Wang et al., 2022; Yang et al., 2022; Liu et al., 2023c; Wu et al., 2023; Zhang et al., 2022) have taken a cautious approach to augmentation, predominantly relying on simple resizing while avoiding complex techniques that could severely distort the data.
In this paper, we explore an effective data augmentation framework improve RIS model training. We propose a straightforward yet robust baseline, coined Masked Referring Image Segmentation (MaskRIS), which consists of two key components: (1) input masking data augmentation technique for RIS and (2) Distortion-awarr Contextual Learning (DCL) designed to maximize the benefits of input masking. In RIS tasks, it is crucial to preserve essential spatial information (c.g., relative positions, ordering) and key attributo details (e-g., color) for accurately understanding and segmenting objects based on referring expressions. Unlike conventional data augmentation, which often distorts these critical aspects, input masking minimizes such distortions by preserving spatial and attribute information while significantly expanding data. diversity. Our approach extends input masking both images and text to strengthen the model's capability handle visual and linguistic complexity. Masking parts of the text encourages the model to infer missing
2

---

Published in Transactions on Machine Learning Research (11/2025)
Global Feature guided Masking
X rich diversity X architecture-agnostic
"a eat lying seat of the sccoter"
VLM
Augmented data
high comeeiaionn rogion
CM-MaskSD
Holistic Context Masking
rich diversity architecture-agnosii
"a cat lying the seat of the scooter"
Augmented data
(b) MaskRIS
Comparison with CM-MaskSD (Wang et al., 2024). (a) CM-MaskSD employs global feature guided masking via vision-language model (VLM), such CLIP (Radford et al., 2021), which requires pre-aligned vision language representation. It focuses high-correlation regions but lacks diversity and architecture-agnosti properties. (b) MaskRIS adopts holistic context masking, yielding richer diversity and improved architecture independence.
Figure 2:
or ambiguous details, improving its understanding of diverse referring expressions and reducing reliance on specific terms. This masking strategy directly addresses the need for a robust RIS training framework, helping overcome the performance bottlenock caused by limited augmentation.
To fully exploit the advantages of imput masking, we propose a Distortion-awarr Contextual Learning (DCL. framework that processes inputs through two complementary paths: primary path with original inputs and secondary path with masked inputs. The primary path ensures baseline training stability and maintains model accuracy, while the secondary path enhances data diversity and robustness. The distillation loss aligns predictions from both paths, encouraging the model to make consistent predictions on both original and masked inputs. This DCL framework enhances feature robustness to masked inputs through the secondary path, while the primary path provides stability and mitigates potential harmful effects of severe distortion. Additionally, DCL acts effect aligns with findings from Mobahi et al. (2020), where self-distillation is shown to constrain model representations, reducing overfiuting and enhancing robustness process similarly achieved by aligning original and masked predictions in our framework.
regularizer, adding beneficial challenges to the RIS task. The regularization
Finally, we note that although MaskRIS shares certain surfaco-level similarities with the concurrent work CM-MaskSD (Wang al., 2024), it is conceptually and technically distinct. As illustrated in Figure 2, CM- MaskSD employs masking guided by cross cross-modal correlations, where a global text feature is compared against all image patch tokens (and vice versa for a global image feature with text tokens). This curclation-basee masking selects the most text-relevant patches and distills them through the [CLS]-token representation, resulting in an object-centric, token-level self-distillation scheme. However, this design inherently relies on a pre-aligned vision language representation, limiting its generalizability. In contrast, MaskRIS introduces holistic context masking across both image and text, which not only preserves spatial cues and avoids augmentation-induess semantic flips (e.g., "left" and "right", " color changes), but also yields richer diversity by exposing the model a broader range of masked contexts. Our Distortion-aware Contextual Learning (DCL) aligns segmenlation oulpuls from clean primary path and masked secondary path, enforcing pixel-wise consistency while maintaining stability and diversity. This enables reasoning under incomplete or ambiguous multi-modal cues; empirically, masking both modalities outperforms masking only one (sce Sec. 4.3). Importantly, MaskRIS is free from pre-aligned cross-modal features, architecture-agnoste and adds no parameters or inference overhead, thereby departing elearly from prior masking- based RIS met.ods.
Our contributions are as follows: (1) We provide a comprehensive analysis of data augmentation techniques for robust RIS training, identifying limitations in previous methods for the first time. (2) We propose a holistic context masking strategy that perturbs both image and text, increasing data diversity through varied,
3

---

Published in Transactions on Machine Learning Research (11/2025)
yet coherent partial contexts while reducing semantic conflicts across modalities. (3) We introduce Distortion- aware Contextual Tearning (DCT.), dual-path distillation framework that enforces pixel-level consistency between original and masked predictions. Unlike prior [CLS] token dependent schemes tied to specific architectures, our approach combines holistic masking with DCL to enable stable and archttccturengnnstii training. (4) Our method achieves significant performance gains over existing RIS methods, including weakly supervised approaches, establishing new state-of-the-art results on RefCOCO, RefCOCO+, and RefCOCOg datasets. Moreover, we show that the proposed framework also benefits Referring Expression Comprehension (REC), highlighting its broader applicability beyond RIS.
2
Related Work
Referring Image Segmentation. Referring Image Segmentation (RIS) aims to segment objects in images based on text descriptions. This field evolved from early WIN-RNNNLSTAN combinations for image and text processing (Margffoy-Tuas et al., 2018; Shi et al., 2018; Hu al., 2016; Li et al., 2018) to advanced approaches using cross-modal attention (Liu et al., 2017; 2021a; Ye et al., 2019; Hu et al., 2020; Hui et al., 2020; Ding et al., 2021) and pre-trained transformer encoders (Wang et al., 2022; Kim et al., 2022; Yang et al., 2022; Liu et al., 2023c). While previous studies have focused on image-text alignment, our study shifts focus to the training strategy in RIS, offering a complementary approach to enhance existing studics. This orthogonal perspective allows us to enjoy the performance benefits established by previous methods. Our concurrent work, NeMo (Ha et al., 2024), introduces a mosaic-based augmentation that combines a target image with carcfully selected negative samples. While effective, it requires additional overhead for constructing and retrieving from a negative sample pool. In contrast, our approach provides more efficient framework that enhances data diversity with a broader analysis of augmentation types and model robustness.
Masked Input Training. Masked input training is self-supervised learning technique where random parts of input data are masked, and the model learns to predict these masked elements from visible data. Originally successful in natural language processing (NLP) as masked language modeling (MLM) (Devlin et al,, 2018; Liu et al., 2019), it expanded to computer vision as masked image modeling (MIM) showing both training efficiency and strong generalization performance (Bao el al., 2021; Xie et al., 2022; He et al., 2022). These masking strategies are also employed to develop robust feature representations. RandomErasing (Zhong et al., 2020) and Cutout (DeVries & Taylor, 2017) erase randomly selected rectangle areas to reduce over- fitting and handle occlusion. MIC (Hoyer et al., 2023) leverages context clues from the target domain through masked image consistency for domain adaptation. SMKD (Lin et al., 2023) uses masked knowledge distillation for few-shot classification.
Recently, MagNet (Chng et al., 2024) applies masked text training to improve rision-language aligmment in RIS, where masked words are predicted using image features. Similarly, CM-MaskSD (Wang et al., 2024) applics masked input training using object-centric masking and CLIP-based patch-text alignment to refine feature representations. Although both methods improve cross-modal learning, their main focus is on feature alignment. In contrast, MaskRIS is inspired by the semantic conflicts from data augmentation in RIS. To address this issue, we performed systematic analysis of augmentatiion-indueed inconsistencies and their impact on RIS performance. Based on these findings, MaskRIS introduces holistic masking strategy and a distortion-aware training framework, explicitly designed to improve model robustness in RIS.
Self-distillation. Self-distillation leverages the model itself as both teacher and student, preventing the need for separate teacher networks to facilitate knowledge distillation (Hinton et al., 2015). PS-KD (Kim et al., 2021) demonstrates the utility of softening hard targets through teacher model predictions, acting an effective regularization strategy. Some studies (Zhang et al., 2019; Phuong & Lampert, 2019) explored self-distillation by employing an entire network as the leacher and an early exit nelwork as the student. CoSub (Touvron et al., 2023) improved model performance in visual recognition tasks with sub-model based self-distillation. MaskSub (Heo et al., 2023) introduced drop-based technique for sub-model self-distillation, improving model performance and cost efficiency for image classification tasks. To fully exploit the benefits of input masking, we employ self-distillation between the predictions of the original and masked input.
4

---

Published in Transactions on Machine Learning Research (11/2025)
YoU (96)
75
Wib DA
WVDA
16677
1BB00
76
boaelins
50
74
IoU
25
60
58
CRIS (CVPR'22)
ETRIS (ICCV'23)
CARIS (MM'33)
ReMamber (ECCV'24)
56
Base
Crop
hFlip
Color
I-Mask
T-Mask
(a) Performance w/ conventional augmentations
(b) Performance w/ individual angmentation on CARTS
Figure 3: Existing RIS methods show noticeable decline in their performance when applying conventional image augmentations (random cropping, color jittering, and horizontal flipping). In contrast, image masking (I-Mask) and text masking (T-Mask) improve model performance.
3 Method
In this section, we propose novel training strategy for RIS, called Masked Referring Image Segmentation (MaskRIS). MaskRIS aims to address the limitation of previous RIS (Sec. 3.1) with two key components: (1) input masking strategy (Sec. 3.2) for data augmentation in RIS and (2) dual-path training (Sec. 3.3) to maximize the benefits of the masking strategy.
3.1 Referring Image Segmentation (RIS)
RIS Pipcline. Given image #xx33 and text description specifying an object within the image, the RIS model outputs a pixel-level mask delineating the referred object. The text description is first tokenized into a sequence of words w {w], W2...., wt}, where w; R represents a word embedding with D as the embedding dimension and L as the number of words in the description. Here, H and W denote the height and width of the image, and represents the RGB channels. We formalize this process as follows:
fx (x,w),
(1)
where 1 E H.W is the predicted mask, generated through the interaction between the input image x and the tokenized words w by the neural network parameterized by 0.
In conventional RIS approaches, f consists of the image encoder (Liu et al., 2021b; Radford et al., 2021) and the text encoder (Devlin et al., 2018; Radford et al., 2021) to extract high-dimensional visual and language features, along with a vision-language fusion module that integrates these features to generate the segmentation mask. Training the RIS models essentially a pixel-wise binary classification task using the cross-entropy loss function:
1 N
Loely 9)
Σ y log(ūs).
(2)
where N denotes the total number of pixels, zi is the binary label at each pixel in the ground truth mask (1 for object, 0 for background), and 0s the predicted probability of the pixel belonging the object.
Limited Data Augmentation in RIS. Previous studies (Yang et al., 2022; Wang et al., 2022; Liu et al., 2023c) have only focused on aligning vision and language features to bridge the modality gap. Still, effective training techniques for RIS remain underexplored, limiting the potential of vision-language features. As vision-language models grow in size and complexity (Radford et al., 2021; Jia et. al., 2021; Alayrac et al., 2022), optimizing these training techniques becomes more crucial. Among training techniques, we regard data augmentation as the most promising approach, with the potential to enhance model robustness and generalization (Rebuffi et al., 2021; Li & Spratling, 2023; Yun et al., 2019; Cubuk et al., 2018; 2020).
Current RIS methods (Wang et al., 2022; Xu et al., 2023; Yang et al., 2022; Liu et al., 2023c) employ only simple resizing operations for data augmentations and don't utilize complex augmentations used semantic segmentation. This is due to the fact that the complex data augmentations in segmentation are incompatible with RIS as shown in Figure/ 1, often altering eliminating the referred objects. Consequently, these
5

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Iutra-object context masking
(b) Inter-object context masking "broccali just below spoon handle"
(c) Text context masking "middle light blue Uee
"center motorcycle"
Original Ingutt
CARIS
Onigiisl Irpat
CARIS
Originll Input
GI'
CARIS
3
"[MASK] guy light blu tle"
Ocdhured Input
GT
CARIS
Ooduded Irput
GT
CARIS
Occludee Inpur
GT
CARIS
Figure 4: The existing RIS method tends to be inaccurate when faced with occluded context. CARIS (Liu et al., 2023c) represents the SoTA method in RIS. Words highlighted in red represent occluded objects in the image (left and center) and masked words in the text query (right).
augmentations lead to decline in RIS model performance on ReíCOCO as demonstrated in Figure 3. Thus, believe tackling the limited data augmentation problem is essential for overcoming major performance bottleneck in RIS.
3.2 Masking Strategy for Image and Text
To overcome the limitations, we introduce a masking- augmentation that combines both image and text masking to generate diverse image-text training pairs. Unlike conventional data augmentation, which often distorts essential spatial and attribute details, image masking minimizes such distortions, prescrving this critical information while significantly expanding data diversily. In addition, image masking is beneficial in recognizing partially visible occluded objects, illustrated in Figure 4(a) and (b). State-of taate-of-thheart models, such as CARIS (Liu et al., 2023c), often struggle to identify partially visible objects. This issue highlights prevalent challenge: the presence of occlusions often leads to the failure of models trained on clear and unoccluded objects. Similarly, as seen from Figure 4(b), text descriptions often rely on contextual objects (e.g., "spoon handle" to describe the position of " "brocooii"),, suggesting that the impact of occlusion extends beyond the target objects themselves.
Beyond managing image occlusions, RIS requires flexibility in interpreting varied text descriptions, single referred object be described in multiple ways. This diversity in textual description is essential for generalizing to different referring expressions. While our image masking addresses visual occlusions, we introduce text masking to enhance the model's capability to interpret diverse and incomplete descriptions by training to roly on broader contextual cues. By masking parts of the text, we encourage the model to infer missing details, improving its understanding and reducing dependence on specific terms. For instance, as shown in Figure 4(c), CARIS can accurately identify a " "middle guy light blue tie" with complete text. However, struggles with partial descriptions, such as "[MASK] guy light blue tie", indicating the limitations in handling incomplete textual clucs. By incorporating text masking, we enable the model to adapt to various referring expressions, enhancing its comprehension and robust in RIS tasks.
We next describe our image and text masking techniques in detail, demonstrating how they address the aforementioned challenges in RIS. In the following, we specify the detailed process for both image and text masking, clarifying how these techniques are implemented to overcome the identified challenges and enhance the RIS model's performance.
Image Masking. We divide the input image & into non-overlapping patches and randomly mask a fixed ratio of patches. Following the MAE sampling strategy (He al., 2022), we create a binary mask M R #xW by sampling patches uniformly without replacement, where 1 indicates a masked patch. The masked image z is obtained by clement-wise multiplication: xM (1 M) @ x, where denotes the element- wiiee multiplication operation. This mechanism ensures that only subset of the image's patches is exposed to

---

Published in Transactions on Machine Learning Research (11/2025)
"middle guy lightblue tie"
"middle guy lightblue tie
RIS model
oe
RIS model
Original input
X data diversity
Original input
X distortion-aware training (no data distortion)
Shared weights
C
(a) Previous RIS training
"middle gay biue
"[MASKI guy light blue tie
RIS model
Ce
RIS model
Original imput
A uggmented input
data diversity
Masked input
data diversity
X distortion- -warr training
distortion aware training
(b) RIS trainingw/ conventional DA
(c) MaskRIS training
Figure 5: The overall framework of MaskRIS. Both image and text masking are employed to generate diverse image-text training pairs (Sec. 3.2). To maximize the benefits of the masking stratogy, Distortion- aware Contextual Learning (DCL) introduced (Sec. 3.3).
the model during training, compelling the neural network infer missing information and thereby enhancing its robustness and predictive accuracy.
While there might be concerns that our approach could hinder training by completely masking target objects, such cases are extremely rare in practice. For example, even with a relatively high masking ratio of 75%, the probability of a larget object being completely masked is only 0.19% remarkably low. This probability becomes even lower with reduced masking ratios. Furthermore, this potential issue mitigated during training as the same samples are seen multiple times. These statistics support that our masking approach effectively augments the training data while maintaining crucial visual information.
...... w using a probabilistic strategy in MLM (Devlin et al., 2018). Specifically, each word w; has a 15% chance of being masked. Of these, 80% are replaced with a [MASK] token, making invisible the model during training; 10% replaced with random word from the vocabulary, introducing noise and variability; and 10% remain unchanged, preserving the original context. This strategy allows the model to experience variety of textual scenarios, enhancing its ability to comprehend context and meaning even when parts of the text are masked or altered. Notably, as Wei & Zou (2019) witnessed, random word masking effectively preserves sentence meaning, further validating our method's balance between sentence coherence and data diversity.
Text Masking. We mask word tokens to generate masked word tokens w = [w w.......
3.3 Distortion-aware Contextual Learning
In the training phase, we employ both image and text masking simultaneously, enriching the diversity of the training dataset. Specifically, we formalize the processing of masked inputs through the model as follows:
j = fo(x w MM,
(3)
where gM denotes the model's output for the masked inputs. By adopting this strategy, MaskRIS learns to effectively segment objects under varying conditions of occlusion and incomplete descriptions. This simple integration effectively mitigates the intrinsic challenge of RIS that stems from the absence of an effective data augmentation strategy.
To further facilitate the RIS training, we introduce Distortion-awee aware Contextual Learning (DCL), as shown in Figure 5. This approach uses a primary path for processing original inputs and a sccondary path for masked inputs. The primary path focuses on preserving the original image context, ensuring training stability, and promoting distortion-aware training. Meanwhile, the secondary path introduces variability by processing
7

---

Published in Transactions on Machine Learning Research (11/2025)
masked data, which improves the model's robustness. The training objective of MaskRIS consists of two parts, the distillation loss Laist and the original pixel-wise classification loss Lee in Eq. 2, i.e., Ltttal = Cdist + Lee. In practice, we adopt a weighting factor between Lce and Ldist. While our default setting uses x= =0555 we also provide a detailed sensitivity analysis of this ratio in Appendix A.5, showing that MaskRIS is robust across wide range of choices.
The distillation loss is defined through the binary cross-entropy loss between the predictions from both original and masked inputs follows:
Cdist Lee (5g())) 1
i log )
(4)
where sg is the stop-gradient function, which blocks gradient pack-propagation. Inspired by Heo et al. (2023), MaskRIS employs self-distillation framework that leverages soft targets from the original inputs, reduc- ing training complexity and enhancing model generalization. This approach allows MaskRIS to effectively mitigate the learning instability caused by input masking while maintaining the benefits of increased data diversity and robustness.
4 Experiments
4.1 Experimental Setups
Datasets. We evaluate our method using three popular benchmarks in RIS: RefCOCO, RefCOCO+, and RefCOCOg. RefCOCO (Yu et al., 2016), all built on the MS COCO dataset (Lin et al., 2014). RefCOCO provides 142,209 expressions for about 50,000 objects across 19,994 images, emphasizing object locations and appearances. RefCOCO+ (Yu et al., 2016) offers over 141,564 expressions, excluding spatial language focus on object attributes. RefCOCOg (Yu et al., 2016) introduces more complexity into the evaluation by including 85,474 longer expressions (8.43 words per expression average) for 54,822 objects in 26,711 images; for this dataset, we report results on the UMD partition (Yu et al., 2016), following the previous studies (Wang et al., 2022; Liu et al., 2023b; Kim et al., 2023b). To further examine cross-dataset generalization beyond COCO-style images, we additionally evaluate MaskRIS on RefClef, a subset of the ImageCLEF dataset with more diverse natural scenes and object categories (see Appendix A.1 for details).
Evaluation Metrics. To evaluate model performance, we use mean Intersection-over-Uue Union (mloU), overall innrrectioo-ove--nnie (oloU), and P@X. While mloU calculates the average IoU across all test samples, oloU measures the ratio of the total intersection to the total union areas across all samples. P@X measures the percentage of test samples with an IoU above threshold X E (0.5, 0.7, 0.9). These metrics ensure a consistent and fair evaluation across different RIS methods.
Implementation Details. To validate MaskRIS, we applied to various RIS models (Xu et al., 2023; Wang et al., 2022; Yang et al., 2022; Liu al., 2023c). Designed as a plug-and-play training strategy, we strictly follow the original training settings and hypprparaamelers, such as learning rate, epochs, and batch size, without modification. Notably,
primarily implemented our method on CARTS (Liu et al., 2023c). leading SoTA method, unless stated otherwise. Images are resized to 448 448 for both training and testing. For image masking, we set 32 as the patch size.
4.2 Main Results
Comparison with State of the Arts. We compare MaskRIS with previous methods on three popular benchmarks. As shown in Table 1, MaskRIS consistently outperforms previous methods by significant margins. Specifically, compared to the second-best performing method, CARIS (Liu et al., 2023c), our approach improves oloU scores by 1.82pp, 1.33%p, and 2.25%p on the RelCOCO validation, Lest,A and testB sets, respectively, On RefCOCO+, our method leads by 1.37%p, 2.76%p, and 1.93%p on the validation, A, and testB splits, respectively. Even on the challenging RefCOCOg datasct, MaskRIS still outperforms CARIS by 0.89%p and 1.05%p on the validation and test splits. These results validate the effectiveness of our masking strategy for RIS model training.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Image
Text Encoder
RefCOCO bestA
RefCOCO+
RefCOCOg
Method
Encoder
val
test
val
Standard:
Training on the braining split of each datasct.
mloU
CRIS (Wang al., 2022)
RN101
CLIP
70.47
73.18
66.10
62.27
68.08
53.68
59.87
60.36
ETRIS (Xu et al., 2023)
RN101
CLIP
71.06
74.11
66.66
62.23
68.51
52.79
60.28
60.42
ReTTR (Td Sigal, 2021)
RN101
BERT
74.34
76.77
70.87
66.75
70.58
59.40
66.63
67.39
LAVT (Yang ct al., 2022)
Swin-B
BERT
74.46
76.89
70.94
65.81
70.97
59.23
63.34
63.62
CM-MaskSD (Wang et al., 2024)
VIT-L
CL.I
74.89
77.54
71.28
67.47
71.80
59.91
66.53
66.63
OGFormer (Tang et al., 2023)
Swin-B
BERT
76.93
78.70
73.32
68.56
73.76
61.72
67.57
67.83
MaskRIS
Swin-B
BERT
78.35
80.24
76.06
71.68
76.73
64.50
69.31
69.42
oloU
LAVT (Yang et al., 2022)
Swin-B
BERT
72.73
75.82
68.79
62.14
68.38
55.10
61.24
62.09
CGFormer (Tang al., 2023)
Swin-B
BERT
74.75
77.30
70.64
64.54
71.00
57.14
64.68
LQMFormerr (Shah ct al., 2024)
Swin-B
BERT
74.16
76.82
71.04
65.91
71.84
57 .59
64.73
66.04
NeMo (Ha et al., 2024)
Swin-B
BERT
74.48
76.32
71.51
62.86
69.92
55.56
64.40
64.80
ReMamber (Yang et al. 2024)
Mamba-B
CLIP
74.54
76.74
70.89
65.00
70.78
57.53
63.90
64.00
CARTS! (Tiiu et al,, 2023c)
Swin-B
BERT
74.67
77.63
71.71
66.17
71.70
57.46
64.66
65.45
MagNet (Chng et al., 2024)
Swin-B
BERT
75.24
78.24
66.16
58.
65.36
66.03
MaskRIS
Swin-B
BERT
76.49
78.96
73.96
67.54
74.46
59.39
65.55
66.50
Combined:
Training on the combination of three dafasets.
oloU
PolyFormer (Liu et al., 2023b)
Swin-B
BERT
74.82
76.64
71.06
67. 67.64
72.89
59.33
67.76
69.05
CARIS! (Liu al., 2023c)
Swin-B
BERT
76.67
79.85
73.07
68.23
73.98
59.59
67.42
MaskRIS
Swin-B
BERT
78.71
80.64
75.10
70.26
75.15
62.83
69.12
71.09
Table 1: Comparison with state-of-the-ard methods on three benchmark datasets. denotes the reproduced
results across all experiments.
"main guy"
"front zebra"
"back the second chair from left"
above dog"
"lady very
'hot dog on the
"main in blue shirt"
"main in red
"middle zebra*
Image
GT
CARIS
Ours
Image
GT
CARIS
Ours
Image
CT
CARIS
Ours
Figure 6: Qualitative examples of segmentation results on the RefCOCO dataset.
To further demonstrate the capability of our method, we conduct experiments on a larger, more comprehen- sive dalaset. This dataset a combination of the training sets from ReICOCO, ReSCOCO+, and ReICOCOg, following the approach used in previous studies (Liu et al., 2023b;c). Training on this combined dataset yields even better results, with oloU improvements of 1.17-4.04%p over previous methods. This result indicates that our method is effective across different lypes of data and highlights its generalizability. Figure 6 shows qualitative examples on ReICOCO, where our method more comprehensively captures the extent of objects by expanding or reducing their coverage (1st row and 2nd row). Additionally, it demonstrates robustness against occlusion (3rd row) and precisely identifies target objects alignment with the provided text de- scriptions (4th row). These results confirm the ability of our approach to enhance model robustness and leverage textual cues effectively.
Robustness Evaluation. A key strength of our approach lies in its robustness across visually and linguis- tically complex scenarios. To this end, we first evaluate our model's robustness to various image corruptions. We use the benchmark provided by ImageNet-C (Hendryeks Dictterich, 2019), which includes types of corruption, each with 5 severity levels. To evaluate the performance, we compute the oloU each severity level and average these scores for each corruption type. As shown in Figure 7(a), MaskRIS consistently out-

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Various image corruptions
(b) Linguistically complex situations
75
CARIS
oloU (%))
O000 (%)
65
Zoemi
Flur Glass
Blae
Mation
Blu
Ganeeian
Noue Shot
Note
Prost
Impolar
New Derbcus
Blar
Slaw
Gansian
Bler JPEd
Comp. Fog
Speckle
Note
Spanter
Counrnt
Saturate
Brighnes
Ocslovion
Robitive
Positio
Ordering
Figure 7: Robustness on various image corruptions provided by ImageNet-C (Hendrycks & Dietterich, 2019)
and linguistically complex situations. The results (oloU) are evaluated on the RefCOCO validation set.
Image
Text
RefCOCO testA
Method
Encoder
Encoder
val
test
Sraakly-superuie approach
mIoU
TRIS (Liu et al., 2023a)
RN50
CLIP
31.17
32.43
29.56
TRIS (Liu et al. 2023a) MaskRIS
RN50
CLIP
32.60 (+1.43)
34.26
(+1.83)
30.32 (+0.76)
SaG (Kim al., 2023a)
VIT-S/16
BERT
37.21
36.60
39.41
SaG (Kim al., 2023a)
MaskRIS
VIT-S/16
BERT
38.85 (+1.64)
37.70
(+1.1)
40.20 (+0.79)
approach
mloU
CRIS (Wang et al., 2022)
RN50
CLIP
69.52
72.72
64.70
CRIS (Wang et al., 2022)+ MaskRIS
RN50
CLIP
70.73
(+1.21)
74.06 (+1.34)
66.82 (+2.12)
ETRIS (Xu et al., 2023)
R101
CLIP
71.06
74.11
66.66
ETRIS (Xu et al., 2023) MaskRRIS
R101
CLIP
72.39 (+11.33)
74.99
(4-0.88)
68.55 (+188
oloU
LAVT (Yang et al., 2022)
Swin-B
BERT
72.73
75.82
68.79
LAVT (Yang et al., 2022) MaskRRS
Swin-B
BERT
73.79 (+1.06)
76.57 (+-0.75
70.31 (+1.52)
ReMamber' (Yang al., 2024)
Mamba-B
CLIP
72.37
74.85
68,29
ReMamber (Yang et al., 2024) + MaskRIS
Mamba-B
CLIP
74.02
(+1.65)
76.32 (+1.47)
69.06 (+0.77)
DETRIS (Huang et al., 2025)
DINOv2-B
CLIP
74.47
77.40
71.46
DETRIS (Huang et al., 2025) + MaskRIS
DINOv2-B
CLIP
76.08
(+1.61)
78.46 (+1.06))
72.84
(+ (+)3)
CARIST (Liu et al., 2023c)
Swin-B
BERT
74.67
77.63
71.71
CARIS (Liu et al., 2023c) + MaskRIS
Swin-B
BERT
76.49 (+1.82)
78.96
(+1133)
73.96 (+2.25)
Table 2: Compatibility of MaskRIS with various RIS methods. methods in both fully supervised and weakly supervised scttings.
MaskRIS consistently enhances existing
performs the previous SoTA method, CARIS (Liu et al., 2023c), with oloU improvements of 1.34 5.07%p. This demonstrates that MaskRIS effectively reduces overfitting to clean data, providing greater robustness against a wide range of visual distortions.
In addition, we evaluate MaskRIS's robustness to linguistically complex situations, such as occlusion, relative position, and ordering. For occlusion scenarios, we modify the input data by occluding parts of objects. For relative position and ordering, we select samples containing relevant linguistic keywords from the validation set. As shown in Figure 7(b), MaskRIS achieves superior performance in these complex scenarios the RefCOCO validation set, further highlighting its robustness to both visual and linguistic complexity.
Compatibility with Other Methods. Our training strategy is highly compatible, integrates seamlessly with various RIS methods, and achieves significant performance improvements across both weakly supervised and fully supervised settings, shown in Table 2. In the weakly supervised regime, where models rely solcly on textual descriptions without ground-truth masks, MaskRIS still yields notable gains. For instance, TRIS (Liu el al., 2023a) improves by +1.43%p on the validation set, +1.83%p on lestA, and +0.76%p on testBB Likewise, SaG (Kim et al., 2023a) is enhanced by up to +1.64%p, reaching 38.85 mloU on the validation set. In the fully supervisod sctting, MaskRIS demonstrates equally strong compatibility. CRIS (Wang et al., 2022) shows improvements of +1.21%p (val), +1.34%p (testA), and +2.12%p (testB), while ETRIS (Xu et al., 2023) gains up to +1.89pp. Beyond CNN-based baselines, transformer-based approaches such
10

---

Published in Transactions on Machine Learning Research (11/2025)
Image Encoder
Text
RefCOCO
Method
Encoder
val
testA
test
DetAcc
EEVG (Chen et al., 2024)
VIT-B
BERT
87.87
89.97
85.34
EEVG (Chen et al., 2024) MMskkRI
VIT-B
BERT
88.72 (+-0.5)
90.81 (+0.84)
85.36 (+-0.02)
SimVG (Dai et al., 2024)
VIT-B
BETT-3
86.94
88.97
82.87
SimVG (Dai et al., 2024)+
VIT-B
BETT-3
87.79
(+0.85)
90.02 (+1.05)
84.88 (+22.01)
C3VG (Dai et al., 2025)
VIT-B
BEIT-3
92.42
94.12
89.34
C3VG (Dai et al.,
VIT-B
BEiT-3
93.29 (+0.87)
94.86 (+0.74)
90.29 (+-0.95)
Table 3: Compatibility of MaskRIS with REC methods. MaskRIS consistently enhances existing methods.
IM
TM
DCL
P@0.5
P@0.7
P@0.9
oloU
87.73
80.20
39.60
74.67
X
X
87.76
80.52
38.73
75.31
X
V
X
87.72
80.45
39.26
75.32
V
X
88.00
81.35
40.11
75.71
X
V
88.60
81.86
41.08
76.02
88.34
81.19
39.24
75.76
88.62
81.95
41.19
76.49
Table 4: Impact of each component of MaskRIS on the RefCOCÓ validation set, where 1M (TM) refers
image (text) masking.
LAVT (Yang et al., 2022) and CARIS (Liu et al., 2023c) also benefit significantly, with CARIS achicving the highest improvements of +2.25%p (estB. More recent architectures, including ReMamber (Yang et al., 2024) and DETRIS (Huang et al,, 2025), further validate the generality of our method, consistently obtaining +0.8 2.0%p gains without architectural modifications. Overall, MaskRIS consistently improves RIS model performance across diverse architectures and training paradigms, demonstrating its adaptability and effectiveness without modifying the original designs.
Compatibility with REC Methods. The conflict between conventional image augmentations and Anguage-groundes lasks is not unique to RIS. Referring Expression Comprehension (REC) also relies on precise spatial and attribute alignment between text and image. For example, flipping an image containing the phrase "the man on the left" alters the semantic meaning for both RIS and REC. This confirms that the augmentation dilemma is indeed universal to referring tasks. To investigate this, we integrated MaskRIS into various RFC methods (Chen et al., 2024; Dai al., 2024; 2025). As shown in Table 3, MaskRTS consistently improves performance when combined with existing REC models by enhancing robustness to occlusion and incomplete linguistic cues while increasing data diversity. The findings confirm that MaskRIS is not task-specific. Its masking approach addresses the augmentation dilemma across both RIS and REC, making it general training strategy for referring tasks where traditional augmentations are unsafe.
4.3 Ablation Study
Impact of Each Component of MaskRIS. We analyze the impact of each MaskRIS component in model training. Table 4 shows segmentation results on the ReICOCO validation set. The results clearly show that masking images and text scparately improves model performance. However, the most significant improvement occurs when we apply masking to both images and lext together. This demonstrates that the combination of visual and textual masking strategies significantly improves the model's ability to accurately identify and segment the referred objects. In addition, Distortion-aware Contextual Learning (DCL) provides further performance gains. It effectively mitigates the learning instability caused by input masking, while maintaining the benefits of increased data diversity and robustness.
Figure 8 illustrates the benefits of image and text masking. CARIS often fails to correctly identify objects when they or their adjacent objects are partially obscured. However, by incorporating image masking, our approach achieves accurate object recognition (1st and 2nd rows). For example, in the scenario shown
11

---

Published in Transactions on Machine Learning Research (11/2025)
"baby girl"
"person with black shirt bike"
"broccoll the chopssicks"
"empay chair middle"
"leff marine"
"Iudy with gray hair in front"
Image
GT
CARIS
I-Mask
T-Mask
Both
Image
GT
CARIS
I-Mask
T-Mask
Both
Figure 8: Qualitative examples of masking strategies the RefCOCO dataset. T-Mask, T-Mask, and Both denote the results of applying image masking, text masking, and both, respectively.
in the first row with "baby girl" whose legs are obscured, CARIS inaccurately identifies the legs of an adult as part of the target object. On the other hand, our method successfully separates the obscured elements, demonstrating the superior accuracy of our method in complex environments. Conversely, text masking improves the alignment between the text description and the target object (3rd row). For example, with the "left marine", CARIS inaccurately identifies the target object by focusing too much on spatial information (i.e., ")eft"), as shown in Figure 4(c). Text masking intervenes in such cases, ensuring the accurate recognition of the target object. Furthermore, by integrating both image and text masking, our approach exploits the strengths of cach masking strategy.
4.4 Discussion: Comparison with CM-MaskSD
MaskRIS differs from CM-MaskSD (Wang et al., 2024) in three key aspects: purpose, masking strategy, and adaptability. First, MaskRIS explicitly handles the se- mantic conflicts caused by conventional augmentations in RIS, ensuring that transformations do not alter the contextual meaning of referring expressions. In contrast, CM-MaskSD focuses on improving patch-text alignment using CLIP's global vector but does not directly tackle augnnentation-induuel inconsistencies. This makes semantic conflict resolution a unique contribution of MaskRIS, Second, while CM-MaskSD employs object-centric masking, selectively targeting key objects keywords, MaskRIS utilizes holistic masking approach that applies masking across all image regions. This strategy enhances model generalization and robustness. As shown in Table 5, holistic masking consistently outperforms object-centric masking. Third, CM-MaskSD is architecture-deeendee due to its reliance on the [CLS) token, limiting its applicability to CLiP-based models. In contrast, MaskRIS is architectureagnostii and can be seamlessly integrated into various RIS and REC frameworks.
Masking
Image Enc.
Text Enc.
oloU
Object-centric Context Holistic
75.14
Swin-B
BERT
75.93
76.02
Table 5: Comparison of image masking ap-
proaches on the RefCOCO validation set.
5 Conclusion
In this study, introduce Masked Referring Image Segmentation (MaskRIS), effective training frame- work for Referring Image Segmentation (RIS) that combines image and text masking with Distortion-aware Contextual Learning (DCL). MaskRIS addresses the challenges of conventional data augmentation in RIS by minimizing semantic distortion and enhancing data diversity. This approach strengthens the model's robust- ness to occlusions and incomplete information, achieving new state-of-the-ar accuracies on the RefCOCO, RefCOCO+, and RefCOCOg datasets. Our results demonstrate MaskRIS's effectiveness and adaptability in both fully and superv-- settings, highlighting its potential as versatile and powerful baseline for ad vanciin the field of RIS.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Acknowledgments
Some parts of experiments are based on the NAVER Smart Machine Learning NSML (Kim et al., 2018) platform. This research was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (RS- RKS-2025-02211259, 25%), the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the MSIP (CS-2025-00520207, 25%, RRS-203-000219019 25%), and Samsung Electronics Co., Ltd (10230508-0619000), 25%).
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 227366,,376,---111 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint trXf::2106.0822 2021.
Jianbo Chen, Yclong Shen, Jianfeng Gao, Jingjing Liu, and Xiaodong Liu. Languago-based image editing with recurrent allentive models. In Proceedings of the IEEE Conference Compuler Vision and Pallern Recognition, pp. 3721-8729) 2018.
Wei Chen, Long Chen, and Yu Wu. An efficient and effective transformer decoder-based framework for multi-task visual grounding. In European Conference on Computer Vision, pp. 125-141. Springer, 2024.
Yong Xien Chng,, Henry Zheng, Yizeng Han, Xuchong Qiu, and Gao Huang. Mask grounding for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 01173-22583,, 2024.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXic preprint hkl::18050900 2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3504 133504-3514, 2021.
Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, and Wankou Yang. Simvg: A simple framework for visual grounding with decoupled multi-modal fusion. Advances in neural information processing systems, 37:121670- 111226670-121698, 2024.
Ming Dai, Jian Li, Jiedong Zhuang, Xian Zhang, and Wankou Yang. Multi-task visual grounding with coarse-to-fine consistency constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255. IEEE, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deop bidirectional transformers for language understanding. orXiv preprint arXiv:1810.04880 04805, 2018
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. Xin preprint ar Xiv: 7708.0055... 04552, 2017.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query gener- ation for referring segmentation. In Proceedings of the IEEE/CV CVF International Conference on Computer Vision, pp. 16321-122 161300, 2021.
Seongsu Ha, Chaeyun Kim, Donghwa Kim, Junho Lee, Sangho Lee, and Joonseok Lee. In Proceedings of the 18th European Conference on Computer Vision (ECCV), 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognilion, pp.
2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv:1903. 2019.
Bycongho Heo, Tackyung Kim, Sangdoo Yun, and Dongyoon Han. Augmenting sub-model improve main model. Xrxiv preprinl arx:::2306. 11339, 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arxiv preprinl arXiv:1503.0905 2015.
Lukas Hoyer, Dengxin Dai, Haoran Wang, and Lue Van Gool. Mic: Masked image consistency for context- enhanced domain adaptation. In Proceedings of the TEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11721 11732, 2023.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expressions. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part I 14, pp. 108-124. Springer, 2016.
Zhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang, and Huchuan Lu. Bi-diroctional relationship inferring network for referring image segmentation. In Proceedings of the IEEE/CVE conference on compuler vision and pattern recognition, pp. 4424 4433, 2020.
Jiaqi Huang, Zumnan Xu, Ting Liu, Yong Liu, Haonan Han, Kehong Yuan, and Xiu Li. Densely connected parfmetter-efficeen tuning for referring image segmentation. Tn Proceedings of the AAAT Conference on Artificial Intelligence, volume 39, pp. 3653 3661, 2025.
Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, and Jizhong Han. Linguistic structure guided context modeling for referring image segmentation. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part X 16, pp. 59-75. Springer, 2020.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for compu- lalional linguistics, 8:64 77, 2020.
Sahar Kazemzadeh,, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Dongwon Kim, Namyup Kim, Cuiling Lan, and Suha. Kwak. Shatter and gather: Learning referring im- age segmentation with text supervision. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 15547 15557, 2023a.
Hanjoo Kim, Minkyu Kim, Dongjoo Sco, Jinwoong Kim, Heungscok Park, Socun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, al. Nsml: Moet the mlaas platform with real- world case study. XrXiv preprint Xiv:: 1110:1810.000 310.09957, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEE/CVF international conference compuler vision, pp. 6567 6576, 2021.
Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjum Zeng, and Suha Kwak. Restr: Convolution-free referring image segmentation using transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 18145 18154, 2022.
Seoyeon Kim, Minguk Kang, and Jaesik Park. Risclip: Referring image segmentation framework using clip. arXiv preprinl (ifi:2306008499 2023b.
Lin Li and Michael Spratling. Data augmentation alone can improve adversarial training. arXiv preprint arXiv:2301.00 09879, 2023.
Muchen Li and Lconid Sigal. Reforring transformer: A one-step approach to multi-task visual grounding. Advances in neural information processing systems, 34:19652 19664, 2021.
Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2018.
Han Lin, Guangxing Han, Jiawei Ma, Shiyuan Huang, Xudong Lin, and Shih-Fu Chang. Supervised masked knowledge distillation for few-shot transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 19649 19659, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 18th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-756 755. Springer, 2014.
Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, and Alan Yuille. Recurrent multimodal interaction for referring image segmentation. In Proceedings of the IEEE international conference computer vision, pp. 1271-1280, 2017.
Fang Liu, Yuhao Liu, Yuqiu Kong, Ke Xu, Lihe Zhang, Baocai Yin, Gerhard Hancke, and Rynson Lau. Referring image segmentation using text supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2112--21134, 2023a.
Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and R. Man- matha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 18653 18663, 2023b.
Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, and Guanbin Li. Cross-modal progressive compre- hension for referring segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 44 (9):4761-4775, 2021a.
Sun-Ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie, Yongdong Zhang, and Ting Yao. Caris: Context- aware referring image segmentation. In Proceedings of the 31st ACM International Conference on Mulli- media, pp. 779 788, 2023c.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xrx:::1907. 1 11692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Xie preprint arxiv 11111051000111111 2017.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Edgar Margffoy-Tuay Juan C Pérez, Emilio Botero, and Pablo Arbeláez. Dynamic multimodal instance segmentation guided by natural language queries. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 630 645, 2018.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361, 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2085 2094, October 2021.
Mary Phuong and Christoph H Lampert. Distillation-based training for multi-exit architectures. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp.
2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Levestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness, Advances in Neural Information Processing Systems, 34:29935 29948, 2021.
Dhruy Shah, Blažej Osiáski, Sergey Levine, et al. Lm--av: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, pp. 492 204-004. PMLR, 2023.
Nisarg A Shah, Vibashan VS, and Vishal M Patel. Lqmformer: Language-aware query mask transformer for referring image segmentation. In Proceedingg of the IERE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12903-12913, 2024.
Hengean Shi, Hongliang Li, Fanman Meng, and Qingbo Wu. Key-word-aware network for referring expression image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 38 54, 2018.
Jiajin Tang, Ge Zheng, Cheng Shi, and Sibei Yang. Contrastive grouping with transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23500,, 2023.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégon. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski, Jakob Verbeek, and Hervé Jégou. Co- training 21 submodels for visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11701-11710, 2023.
Wenxuan Wang, Xingjian Hc, Yisi Zhang, Longteng Guo, Jiachen Shen, Jiangyun Li, and Jing Liu. Cm- masksd: Cross-modality masked self-distillation for referring image segmentation. IEEE Transactions on Multimedia, 26:6906-699 6916, 2024.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the CEE/ conference on computer vision and pattern recognition, pp. 11686-11695, 2022.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classifi- cation tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNIPPIJCNIP)) 2019.
Ross Wightman. Pytorch image models. https:/ http:://gghhhbb.o//riggaaaaarrrr
2019.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Yixuan Wu, Zhao Zhang, Chi Xie, Feng Zhu, and Rui Zhao. Advancing referring expression segmentation beyond single image. In Proceedings of the IEFF/CVV International Conference on Compuler Vision, pp. 2628 2638, 2023.
Zhenda. Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modcling. In Proceedings of the IEEE/CVV Conference Computer Vision and Pallern Recognition, pp. 9653 9663, 2022.
Zunnan Xu, Zhihong Chen, Yong Zhang, Yibing Song, Xiang Wan, and Guanbin Li. Bridging vision and language encoders: Parameter-efficient luning for referring image segmentation. In Proceedingss of the IEEE/CVP International Conference Computer Vision, pp. 17503 2023.
Yuhuan Yang, Chaofan Ma, Jiangchao Yao, Zhun Zhong, Ya Zhang, and Yanfeng Wang. Remamber: Refer- ring image segmentation with mamba twister. arXiv preprint arXiv:2403. 17899, 2024.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language- aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18155-18165, 2022.
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-allention network for referring image segmentation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10502 10511, 2019.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L. Berg. Modeling context in referring expressions. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer, 2016.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the FEEE/CV international conference computer vision, pp. 6023 6032, 2019.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVE internalional conference on compuler vision, pp. 3713 3722, 2019.
Zicheng Zhang, Yi Zhu, Jianzhuang Liu, Xiaodan Liang, and Wei Ke. Coupalign: Coupling word-pixel with sentence-mask aliguments for referring image segmentation. Advances Neural Information Processing Systems, 1414779-147747 2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001-13008, 2020.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Supplementary Material
Contents
A Additional Experiments
18
A.1 Evaluation on RefClef
18
A.2 Analysis of Image Masking Strategy
A.3 Analysis of Text Masking Strategy
19
A.4 Analysis of Distillation Strategy
20
A.5 Impact of Hyperparameters
20
A.6 Analysis of Computational Cost.
21
A.7 Analysis of Training Loss
21
B More Qualitative Examples
22
C Implementation Details
25
A Additional Experiments
Image
Text Encoder
Method
oToU
Encoder
CARIS (Liu et al., 2023c) CARIS + MaskRIS
75.21
Swin-B
BERT
76.06
Table A: Comparison on RefClef (ImageCL.FF) dataset. MaskRIS consistently improves over the baseline, demonstrating generalization beyond MS COCO benchmarks.
A.1 Evaluation on RefClef
Following the common practice of prior works RIS, our main experiments are conducted on MS COCO based benchmarks (RefCOCO, ReICOCO+, RefCOCOg). In addition, we validated MaskRIS on RefClef (Kazemzadeh et al., 2014), which is subset of the ImageCLEF dataset. Unlike MS COCO, RefClef includes more diverse natural scenes and broader set of object categories beyond MS-COCO's predefined classes.
As shown in Table A, MaskRIS outperforms the CARIS baseline (76.06 vs. 75.21), confirming that its effectiveness is not limited to COCO-style data. These results demonstrate that our holistic context masking and distortion-aware training framework generalizes well across datasets with varying scene compositions and calegory coverage, thereby reinforeing the broader applicability of MaskRIS.
A.2 Analysis of Image Masking Strategy
To rigorously validate the impact image masking on model training, we explored various mask sampling strategics and their effects. Figure A visually illustrates each method. Our baseline, patch-wise sam- pling (Heo et al., 2023: Hoyer et al., 2023; He al., 2022) divides images into non-overlapping patches and randomly masks certain patches. This approach is widely used due to its simplicity and effectiveness. We
18

---

Published in Transactions on Machine Learning Research (11/2025)
Image
Patch-wise
Grid-wise
Block-wise
Cutout
Figure A: Various types of image mask strategies. We use patch-wise sampling (He et al., 2022) as our default. For block-wise sampling, we follow BBiT(Bao et al., 2021) to remove large random blocks. For Cutout (DeVries & Taylor, 2017), we follow the implementation of timm (Wightman, 2019). For all strategies except Cutout, we maintain a consistent masking ratio of 75%.
Patch-wise Grid-wise Block-wise Cutout
oloU 76.49
76.14
75.90
76.07
Table Impact of imaeg masking strategy. The performance (oloU) evaluated across various masking strategies on the ReLCOCO validation set.
MaskRIS
WWM SpanBERT POS-based
oloU
76.49
76.03
76.46
76.52
Table C: Impact of text masking strategy. The performance (oloU) is evaluated across various masking strategies on the RefCOCO validation set.
also lested grid-wise sampling (He et al., 2022), where rows and columns are masked systematically rather than randomly. Additionally, we investigated block-wise masking, as used in BETT (Bao al., 2021), where masked, potentially climinating more important structural details by
contiguous regions of the image covering larger areas. Finally, we evaluated Cutout (DeVries & Taylor, 2017), which masks randomly selected rectangular section of the image.
Table B summarizes the results (oloU) of each mask sampling strategy on the ReiCOCO validation set. Remarkably, all the strategies we tested consistently outperform the previous SoTA method, CARIS (Liu et al., 2023c) (74.67 oloU), with significant margins. Among these strategies, our patch-wise random sampling approach achieved superior performance, demonstrating its ability to generate more varied training data than grid-wise sampling and minimize training data distortion more effectively than block-wis wise sampling. These results confirm the importance of selecting appropriate masking strategy to improve performance.
A.3 Analysis of Text Masking Strategy
To further validate the role of linguistic perturbation, we conducted an ablation study on various text masking strategies. Specifically, we compared four schemes. (1) Token-level random masking, which adopted in MaskRIS. (2) Whole-Wort Masking (WWM) (Cui al., 2021), which masks all sub-tokens of sampled word (e.g., masking "playing" removes both "play" and ##*ing")), often hiding entire entities or key terms. (3) SpanBERT-style masking (Joshi et al., 2020), which replaces contiguous spans of tokens (c.g., masking a two-word phrase like "on the"). This encourages the model to capture longer-range contextual dependencies. (4) Part-of-Speeed (POS)- random masking, where the target POS is treated as a tunable parameter. In our sctup, entity-bearing nouns and relational terms are preserved, while adjectives and adverbs are randomly masked. This approach recognizes the importance of entities while introducing variation through modifiers.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Method
val
test
test
DCL (Ours)
76.49
78. 78.96
73.96
Cross-distillation
76.68
78.92
73.21
Table D: Impact distillation stralegy. The performance (oloU) evaluated across various distillation strategies on RefCOCO.
16
32
64
112
oloU 76.58
76.49
76.49
76.47
76.01
Table E: Impact of patch size image masking, The performance (oloU) is evaluated across different patch sizes on the Re(COCO validation set. pa denotes the patch size.
A
0.1
0.25
0.5
0.75
1.0
oloU
75.48
76.62
76.19
76.16
71.67
Table F: Impact of À on DCL. procedure. The performance (oloU) is evaluated across various on the RefCOCO validation set.
Table C summarizes the performance on the ReICOCO validation set. All four strategies outperform the previous SoTA method, CARIS (Liu et al., 2023c) (74.67 oloU), with consistent margins. Among them, WWM tends to remove entire entities, leading slightly lower performance (76.03 oloU). Span masking performs comparably to token-level masking (76.46 vs. 76.49). The POS-aware strategy achieves the best result (76.52), highlighting the importance of preserving semantic anchors while introducing controlled lexical noise. These findings, combined with our image masking study, demonstrate that carefully selecting what to mask in both visual and textual modalities critical to achieving robust improvements.
A.4 Analysis of Distillation Strategy
We further investigated whether incorporating cross-modal contrastive loss (Wang et al., 2024) could enhance our distortion-awarc contextual learning (DCL) framework. As shown in Table D, adding cross- distillation yields a slight gain on the validation set (76.68 vs. 76.49), but this improvement does not generalize to the test splits. On RefCOCO testA, performance remains almost identical (78.92 vs. 78.96), and on testB even decreases (73.21 vs. 73.96). addition, this sctting substantially increases computational cost, as each training step requires three forward passes instead of one. These results confirm that coss-distillatton does not provide consistent accuracy benefits and results in unfavorable trade-off between performance and efficiency.
A.5 Impact of Hyperparameters
Patch Size of Image Masking. To further analyze the hyperparameters of image masking, examined the training results of different patch sizes for patch-wise image masking. Table E shows that our method consistently achieves superior performance regardless of patch size, outperforming the current SoTA method, CARIS (Liu et al., 2023c). Specifically, we observe that a patch size of yields the highest performance. Nevertheless, the results suggest that our method exhibits stable and strong performance as long as the patch size remains within reasonable range, highlighting its robustness patch size variations.
Loss Weight of DCL. As described in Sec. 3.3, our training objective of Distortion-aware Contextual Learning (DCL) is Liotal = Ldist + Lee. To balance the scale of the loss values with previous methods (Liu et al., 2023c) during the training phase, we adapt the equation to:
Liotal (1- Lasst + X Leer
(5)
where À is set 0.5 for all experiments to avoid extensive hyperparameter search for optimal performance. A À of 1.0 indicates training only with Loe, which is the equivalent to CARTS (Tiu et al., 2023c). In Table F.
20

---

Published in Transactions on Machine Learning Research (11/2025)
IM ratio
TM ratio
Pm
Pr
Pu
oloU
0.15
0.8
0.1
0.1
75.76
0.15
0.8
0.2
0
75.43
0.15
0.5
0.5
75.70
0.50
0.5
0.5
0
74.88
0.75
0.5
0.5
74.14
0.25
0.15
0.8
0.1
0.1
76.07
0.50
0.15
0.8
0.1
0.1
76.43
0.75
0.15
0.8
0.1
0.1
76.49
Table G: Impact of the masking ratio on the RefCOCO validation set. When word w, is selected based on the masking ratio, it has Pm probability of being masked with [MASK] token, Pr probability of being replaced with random word from the vocabulary, and Pu probability of remaining unchanged.
Method
Epoch
val
lestA
LestB
CARTS! (Liu et al., 2023c)
50
74.67
77.63
71.71
CARIST (Liu et al., 2023c)
100
74.80
77.38
70.63
MaskRIS
25
75.87
77.99
73.51
MaskRIS
50
76.49
78.96
73.96
Table H: Comparison of computational cost. All results (oloU) are evaluated on the RefCOCO dataset. MaskRIS outperforms CARIS even with half of the training epochs.
explore how varying À affects the DCL procedure. As shown in Table F, while the best performance is achieved with À = 0.25, the performance with À of 0.5 remains siguificantly better than the results without Cuist. These results confirm that MaskRIS does not require extensive hyperparameter search for À and performs reasonably well with À of 0.5.
Impact of Masking Ratio. We investigate how masking ratios affect model performance, focusing on the balance between masking images and text. Table G provides insights into the optimal use of masking in model training. For text masking, we follow the masking ratio used by BERT (Devlin et al., 2018). Note that we refer to the BERT sctting as it is, sctting Pa to 10% equivalent to combining Pm to 89% with p, to 11%. We observe a significant drop in performance as the text masking ratio increases, indicating that removing too much textual information interferes with model training. On the other hand, image masking shows an optimal improvement a masking ratio of 75%, suggesting that higher degree of image masking improves model performance. These observations highlight the balance of masking ratios across different modalities. Excessive text masking can negatively affect learning by causing the loss of crucial information, whereas increasing image masking up to certain threshold can be beneficial.
A.6 Analysis of Computational Cost.
MaskRIS needs additional computational costs due to the dual-path framework, which requires forwarding both the original and masked inputs. For fair comparison, we adjust the training epochs and examine the impact on performance. As shown in Table H, doubling the training epochs for CARIS does not enhance its performance, indicating that MaskRIS's improved performance is not a result of increased computation. Moreover, even when our method is trained for only 25 epochs, it still outperforms CARIS trained for 50 epochs. This highlights the efficacy of our masking strategy and dual-path approach in boosting performance efficiently. Notably, since MaskRIS does not change the model architecture add additional paramcters, it maintains the inference time the baseline.
A.7 Analysis of Training Loss.
We validate the effectiveness of MaskRIS through an in-depth analysis of training loss, comparing three distinct settings: CARIS (Liu al., 2023c), CARIS w/ Masking, and MaskRIS. For CARIS w/ Masking, we employ both image and text masking as data augmentation strategies, applying them with 50% probability. Figure B shows the training loss and the model performance changes various training epochs. Tn
21

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Train loss for the original imputs
(b) Train loss for the masked inputs
(c) Performance of RIS
0.125
0.3
Train Loss
0.100
Train Loss
olaU
70
0.075
0.2
0.050
0.025
0.1
10
20
30
40
10
20
30
40
10
20
30
40
Epoch
Epoch
Epoch
CARIS
CARIS w/ Musking
MaskRIS
Figure B: Training loss analysis of MaskRIS on RefCOCO. In the CARIS w/ Masking setting, we employ both image and text masking as data augmentation strategies within the CARTS (Liu et al., 2023c) framework. We visualize (a) the training loss for the original (i.e., unmasked) inputs, (b) the training loss for the masked inputs, and (c) the performance of the model on the RefCOCO validation set.
the baseline CARIS, training loss with the original imputs rapidly converges, while training loss with the masked inputs incrcases. This indicates that CARIS overfits clean inputs as training progresses. In contrast, CARIS w/ Masking effectively addresses this overfitting by incorporating input masking. However, it degrades loss convergence and disrupts learning stability. Our MaskRIS improves loss convergence for both original and masked inputs and at the same time significantly increases model performance.
B More Qualitative Examples
Figure C provides additional qualitative examples on the Re(COCO dataset across different occlusion sce- narios described in Sec. 3.2. We categorize our analysis into three cases: (a) intra-object context masking, where occlusions occur within the object of interest; (b) inter-object context masking, where occlusions occur within non-target objects; and (c) text context masking, where parts of textual descriptions are obscured.
In scenario (a), CARIS (Liu et al., 2023c) often struggles to recognize objects that are only partially visible, highlighting its difficulty in handling incomplete visual information. The challenge not limited to occlusions of the target objects. In scenario (b), CARIS (Liu et al., 2023c) also faces challenges when occlusions occur in non-target areas surrounding the target object, redueing its ability to identify the intended objects correctly. Similar limitations are observed in the text description domain, shown in Figure C(c). While CARIS can successfully recognize the targel objects with complete text descriptions, its performance drops when only partial text information is available. In contrast, our MaskRIS demonstrates robustness across these occlusion scenarios by improving its comprchension and utilization of diverse contextual dctails in the image and text data.
Figure D shows more qualitative examples on the RefCOCO dataset. As shown in Figure D(a), our MaskRIS accurately captures the target objects without over- or under-capturing them. In Figure D(b), it demon- strates strong robustness to occlusion, effectively identifying partially visible objects. Furthermore, lighted in Figure D(c), MaskRIS successfully aligns target objects with the given text descriptions. These results demonstrate that our approach significantly improves the robustness of the model and its ability to utilize textual clues for accurate object identification.
high-
We also provide failure cases in Figure (dd.. When the referred object an image or the given text descrip- tion is ambiguous or difficult to recognize, our approach sometimes struggles to identify the correct target. For instance, in the first example of the top row, MaskRIS fails to recognize the drawing ("flowers") on the "bottle", instead capturing different object. Another failure occurs when text descriptions include con- trasting relative positions, such as "Ieft" and "right", which confuse the model. Insufficient text descriptions also pose significant challenges. When text descriptions do not sufficiently characterize the larget object such as "yellow" or "her" the bottom row, MaskRIS captures the other objects as the target. These examples illusraate that while MaskRIS shows significant improvements in robustness and accuracy across various scenarios, it can still be limited in handling complex visual scenes or overly vague and ambiguous text descriptions.
22

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Intra-object context masking
"person holding snowboard"
"far left bowl"
"container with sliced strawberries"
"elephant on left"
Image
GT
CARIS
MaskRIS
Image
GT
CARIS
MaskRIS
(b) Inter-object context masking
"top
"giraffe back
"man on left"
"zebra in the back
Image
GT
CARIS
MaskRIS
Image
GT
CARIS
MaskRIS
(c) Text context masking
"denim jacket
"denim jacket MASK]
"mom holding child""
"[MASK] holding child
"guy middle back striped shirt"
"guy in middle [MASK] striped shirt"
"guy sitting "
"[MASK] sitting
Image
GT
CARIS
MaskRIS
Image
GT
CARIS
MaskRIS
Figure C: Qualitative examples under various occluded contexts on the ReiCOCO dataset. Although CARIS (Liu et al., 2023c) tends to inaceurate under occluded contexts, MaskRIS produces accurate predictions, demonstrating robustness occlusion and incomplete information. For text context mask- ing, the word highlighted red is masked.
23

---

Published in Transactions on Machine Learning Research (11/2025)
(a) Precise Coverage
"person holding snowboard"
"cookic the middle"
"first wine glass"
"wine glass In front next to water carafe"
Image
CT
CARIS
MaskRIS
Image
GT
CARIS
MaskRIS
(b) Robustness to oclusion
"girl in purple haody
"sheep on far left"
"man left of red ubrella
"motorcycle far right
Image
GT
CARIS
MaskRIS
Image
GT
CARIS
MaskRIS
(c) Precise A Aligmment with Text
"main shirt playing tennis
"little dress"
"catcher"
arms crossed
Image
GT
CARIS
MaskRIS
Image
GT
CARIS
MaskRIS
(d) Failure Cases
"bottle with flowers on labe)
"player on left on right home
"yellow" "
"ber"
Image
CARIS
MaskRIS
Image
CARIS
MaskRIS
Figure D: More qualitative examples on the RefCOCO dataset. MaskRIS mitigates the limitations of CARIS (Liu et al., 2023c) by (a) capturing target objects more precisely, (b) being robust to occlusions, and (c) effectively using various textual clues. (d) We also provide failure cases.
24

---

Published in Transactions on Machine Learning Research (11/2025)
C Implementation Details
Most of our experimental results are based on CARIS (Liu et al., 2023c). For the image encoder, we used the Swin-Base Transformer (Liu et al., 2021b), pre-trained on ImageNet-22k (Deng et al., 2009), and for the text encoder, we employed BERT-Base (Devlin et al., 2018). The maximum length of the text is set to 20 words. We used the AdamW (Loshchilov & Hutter, 2017) optimizer with a weight decay of 0.01. We applied different learning rates of 1e 5 and le 4 encoders and the others, respectively, with a polynomial learning rale schedule with power of 0.9. The model was trained for 50 epochs with batch size of 16, and the input images were resized to 448 448.
To validate the adaptability of MaskRIS with other RIS methods, we integrated it into several existing models in both fully and weakly supervised settings, as shown in Table 2. Importantly, we follow the original training recipes of each method without any changes to the original architecture. This simple integration demonstrates the flexibility of MaskRIS and its seamless compatibility with existing frameworks.
25


Reviews:
[{"review_id": "1iVFVQw2nS", "reviewer": "Reviewer_3PtX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper is overall clear and easy to follow.\n- MaskRIS is conceptually simple, does not require architectural changes, and is applicable to a range of existing RIS models.\n- Experimental results across RefCOCO/g/+ benchmarks and different model structures demonstrate the effectiveness of the MaskRIS.\n\n### Weaknesses\n- Dataset scope is somewhat limited. All images are from MS-COCO datasets. Could the authors experiment with  datasets from other image sources?\n- Certain implementation details are missing. E.g. What is the ratio of the CE loss and the distillation loss? Does this ratio affect the results?\n- Ablations could be more comprehensive. E.g. It would be valuable to assess the sensitivity of performance to the choice of patch size in image masking.\n- The evaluation would be more convincing if it included additional qualitative and quantitative results demonstrating MaskRIS’s robustness under both image and text distortions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper first analysis the limitation of existing data augmentation methods for referring image segmentation. It then proposes MaskRIS, a framework aiming to improve robustness for referring image segmentation via image masking, text masking, and Distortion-aware Contextual Learning (DCL). Experimental results demonstrates the method's effectiveness on the RefCOCO series benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses section.", "broader_impact_concerns": "Not much.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sxEQ5koF5j", "reviewer": "Reviewer_gCJq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* The paper addresses a practical question of referring image segmentation: augmentation can distort the object in the image that the text is referring to. Solving this problem is meaningful for RIS tasks.\n* The masking mechanism is simple and effective\n* The ablation experiment is comprehensive.\n\nWeakness/Questions \n* The data augmentation dilemma seems to be universal to the REC and RIS task? How would the same augmentation technique apply to REC tasks?\n* The distortion-aware loss seems to give marginal improvement while requiring twice the inference time during training. Do you have some cost vs performance analysis on this part? \n* Would this method be applicable to those VLM-based generalists like CogVLM-Grounding?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a masking-based data augmentation method to assist with referring image segmentation training. Unlike traditional methods, which could potentially distort the image and cause a performance drop, the proposed method applies random masks to both image and text, enabling a performance boost.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Add performance on REC tasks and some analysis", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tfFAnB3gYs", "reviewer": "Reviewer_2isH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n1.\tThe idea of this paper is intuitive and easy to understand.\n\n2.\tLeverage masking strategy and self-distillation framework for RIS is reasonable.\n\n3.\tThe framework is compatible with various RIS architectures, making it potentially useful as a plug-and-play method.\n\nWeakness\n\n1.  Limited technological contribution. The proposed MaskRIS is largely inspired by CM-MaskSD [1]. The CM-MaskSD already introduced a masked self-distillation method (i.e., Cross-Modality Masked Self-Distillation, Section III.B). However, the authors do not explicitly discuss the difference between their Distortion-aware Contextual Learning (DCL) module and the CM-MaskSD framework, making it unclear what the true contribution is. The main contribution appears to be replacing CM-MaskSD’s object-centric masking strategy with a holistic masking approach.  Therefore, I concern their might not meet the bar for TMLR and may not be of enough interest to the TMLR audience.\n\n2. Unclear experiment setting. The authors report their own reproduced results for CARIS [2] under the \"Standard\" setting, while using the originally reported results for CARIS under the \"Combined\" setting. Given that the authors do not provide the source code and CARIS is open-source, the reason behind these choices should be clarified.\n\n3. Missing comparison to recent SoTA methods. The comparison mainly includes methods from 2023, while overlooking newer RIS SoTA methods (e.g., [1, 4, 6]). It is also necessary to evaluate the integration of MaskRIS with recent SoTA RIS methods, such as MagNet [3] and ReMamber [6].\n\n4. Concerns about generalizability. The authors claim that MaskRIS is a plug-and-play approach for various RIS models. However, Table 4 shows experiments mainly on text and image encoders that lack explicit alignment knowledge. There is insufficient evidence that the masking strategy is suitable for RIS models with cross-modal alignment knowledge (e.g., CLIP-based RIS [1,4]).\n\n5. Need more ablation studies. For example, what is the performance of random patch masking? What is the performance of different word masking strategies? What is the performance of MaskRIS to different distillation strategies (e.g., using cross-modal contrastive loss)?  What is the performance of combine MaskRIS to recent SoTA methods (e.g., MagNet[3], ReMamber [6] or CM-MaskSD[1]).\n\nReferences\n\n[1] Wang, Wenxuan, et al. \"Cm-masksd: Cross-modality masked self-distillation for referring image segmentation.\" IEEE Transactions on Multimedia 26 (2024): 6906-6916.\n\n [2] Sun-Ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie, Yongdong Zhang, and Ting Yao. Caris: Context aware referring image segmentation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 779–788, 2023c.\n\n[3] Chng, Yong Xien, et al. \"Mask grounding for referring image segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Huang, Jiaqi, et al. \"Densely connected parameter-efficient tuning for referring image segmentation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.\n\n[5] Kim, Seoyeon, et al. \"Extending clip's image-text alignment to referring image segmentation.\" arXiv preprint arXiv:2306.08498 (2023).\n\n[6] Yang, Yuhuan, et al. \"Remamber: Referring image segmentation with mamba twister.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces MaskRIS, a method addressing semantic distortion in data augmentation for Referring Image Segmentation (RIS), motivated similarly to CM-MaskSD [TMM 2024] by proposing an input masking strategy that simultaneously masks image patches and textual tokens, and a Distortion-aware Contextual Learning (DCL) module that employing self-distillation to enhance robustness against occlusions. The proposed method achieves SoTA performance on three RIS benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major:\n\n1. Clarify technological contribution. Clearly highlight the unique technological contributions of the proposed method and explicitly differentiate the DCL module and holistic masking strategy from existing masking RIS works (e.g., CM-MaskSD) in the introduction section. \n\n2. Clarify experimental settings. Justify and clearly explain the choice of using reproduced results for CARIS under the \"Standard\" setting and reported results under the \"Combined\" setting, given the open-source availability and add pseudo code.\n\n3. Expand comparisons to recent SoTA methods. Include results and integration studies with more recent RIS methods (e.g., MagNet, ReMamber, CM-MaskSD), and provide discussion on compatibility and improvements.\n\n4. Prove MaskRIS’s generalizability. Evaluate the effectiveness of MaskRIS on a CLIP-based RIS models.\n\n5. Extend ablation studies. Please refer to W.5.\n\nMinor:\n\n1. It is recommended to move Section A.1 from the Supplementary to main paper.\n\n2. It is recommended to compare the different between MaskRIS and existing masking RIS works (e.g., CM-MaskSD) in your motivation figure (i.e., Figure 1) \n\n3. It is recommended to compare the performance with CM-MaskSD on quantitative and qualitative experiments (i.e., Table1 and Figure 7)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)