Instruct2See: Learning to Remove Any Obstructions Across Distributions
Authors: Junhang Li
Yu Guo
Chuhua XIAN
Shengfeng He
Venue: ICML 2025 poster (2025)
Keywords: computer vision
image restoration
Primary area: applications->computer_vision

Abstract: Images are often obstructed by various obstacles due to capture limitations, hindering the observation of objects of interest. Most existing methods address occlusions from specific elements like fences or raindrops, but are constrained by the wide range of real-world obstructions, making comprehensive data collection impractical. To overcome these challenges, we propose Instruct2See, a novel zero-shot framework capable of handling both seen and unseen obstacles. The core idea of our approach is to unify obstruction removal by treating it as a soft-hard mask restoration problem, where any obstruction can be represented using multi-modal prompts, such as visual semantics and textual instructions, processed through a cross-attention unit to enhance contextual understanding and improve mode control. Additionally, a tunable mask adapter allows for dynamic soft masking, enabling real-time adjustment of inaccurate masks. Extensive experiments on both in-distribution and out-of-distribution obstacles show that Instruct2See consistently achieves strong performance and generalization in obstruction removal, regardless of whether the obstacles were present during the training phase. Code and dataset are available at https://jhscut.github.io/Instruct2See.

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Junhang Li Yu Guo 23*
Chuhua Xian
Shengfeng He
In-distribution
Out-of-distribution
Obstruction Representation
RainDrop
Cobles
Snow
(e) Ingut
"NEeMvee the SEM SEM Trw.siTTaaeeeee Okstrruttow
(b) Restormer
Mask Decoder
Visual Encoder
Text Encoder
"Stains adherad the surface the ca window."
(c) Latka
(d) Instruuueeeee
Visual
Textual
Mask
Semantic
Command
Figure 1: We present Instruct2See, zero-shot framework for obstruction removal that handles arbitrary obstructions. effectively tackles both soft tramsaarett) and hard (opaque) obstructions, while demonstrating robust effect in both in-distribution (seen) and out-of-distribution (unseen) scenes. The right part shows how represent unseen obstructions.
Abstract
through rros-attention unit to enhance textual understanding and improve mode control. Additionally, a tunable mask adapter allows for dynamic soft masking, enabling real-time adjust- ment of inaccurate masks. Extensive experiments on both in-distribution and dutttf-disttibution obstacles show that Instruct2See consistently achieves strong performance and generalization in obstruction removal, regardless of whether the obstacles were present during the training phase. Code and dataset are available at https: // jhscut. g githhu.. 1o/ Itss.tsstuuu Sec
Images are often obstructed by various obstacles due capture limitations, hindering the observa- tion of objects of interest. Most existing meth- ods address occlusions from specific elements like fences or raindrops, but are constrained by the wide range of real-world obstructions, mak- ing comprehensive data collection impractical. To overcome these challenges, propose In- struct2See, novel zero-shot framework capable of handling both seen and unseen obstacles. The core idea of our approach is to unify obstruction removal by treating as a soft-hard mask restora- tion problem, where any obstruction can rep- resented using multi-modal prompts, such as vi- sual semantics and textual instructions, processed
1. Introduction
Obstruction removal a challenging task that involves re- covering clean scenes occluded by unwanted obstacles or unpredictable natural phenomena. Existing methods often focus on specific types of obstructions (Zhang al., 2019; Wen et al., 2019; Du et al., 2020; Chugunov et al., 2024; Quan et al., 2023a; Huang et al., 2024b; Zhou et al., 2023; Li et al., 2024) by relying on predefined categories and spe- cific training datasets. However, this reliance limits their generalization, often resulting in poor or invalid removal of occluders outside the training distribution. Therefore,
Equul contribution 'Scoool of Computer Science and Engi- neering. South China University of Technology "School of Com- puting and Information Systems, Singapore Management Uni- versity "School of Navigation, Wuhan University of Technology. Correspondence to: Chuhua Xian and Shengfeng <chhx- ian scut.edu.en, shengfenghe sereecgghheum.ucuu.......
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
is crucial to enable models to grasp the underlying physi- cal properties of occlusion to enhance image quality under complex and varied conditions.
bridge the gap between the estimated and actual masks, reweighting the predicted mask into a soft mask that dy- namically adapts to the testing scene. In summary, the key contributions of our work include:
While many advanced methods (Tsogkas et al., 2023;
Chugunov et al., 2024; Zhou et al., 2023; Dai et al., 2023;
We introduce the first unified obstruction formulation and a novel zero-shot paradigm capable of handling any obstruction by integrating obstacle positions with multi multi-modal prompts, including visual semantics and text instructions.
Zhang et al., 2023a; Li et al., 2024; Chang et al., 2024; Chen et al., 2023a) continue to target specific obstructions,
the diversity of real-world obstacles makes designing and training separate models for each type inefficient and im- practical (Guo et al., 2024). As a result, there is growing interest in all-in-one restoration models (Li et al., 2020: Chen et al., 2021; Han et al., 2022; Wang et al., 2023b; Vala- narasu et al., 2022; Li et al., 2022a; Özdenizei & Legenstein, 2023), which aim to handle multiple complex scenarios with single model. However, despite these advancements, such models remain constrained by their training datasets. As shown in Fig. (b), restoration methods trained on multi- scene datasets struggle to handle unseen scenarios and lack the flexibility to adapt based on user input. This limitation is especially problematic in dynamic worldworld applications, such as autonomous driving and intelligent roboties.
We develop a dynamic soft masking strategy that au- tomatically refines inaccurate masks for occluding ob- structions using a tunable adapter.
Comprehensive experiments demonstrate the superior effectiveness of our model in obstruction removal, as well as its strong zero-shot generalization to unseen obstructions outside the training distribution.
2. Related Work
Obstruction Removal. The task of obstruction removal aims to clear unwanted obstructions from scene, improving its visibility. Many existing methods are tailored to specific types of degradation, such as deraining (Zhang et al., 2023b; Wang et al., 2023a), desnowing (Quan et al., 2023b; Chen et al., 2023b), and raindrop removal (Qian al., 2018; Li et al., 2024). While these approaches are effective for in- dividual obstructions, they struggle with handling multiple degradation types simultaneously, often requiring separate models for each. To address this limitation, all-in-one image restoration models have been developed. For example, Liu et al. (2021) proposed a method that separates an image into obstruction and background layers using layered decompo- sition, improving visibility through obstructions. Li et al. (2022a) introduced AirNet, which incorporates an additional encoder with contrastive learning to distinguish between var- ious degradation types. Potlapalli et al. (2023) presented PrompuR, a flexible plugin module that uses lightweight prompts to handle multiple image restoration tasks. Histo- former was introduced to employ histogram equalization techniques within a neural framework enhance and re- store degraded images (Sun et al., 2024). Despite these advancements, all-in-one models still face challenges when dealing with degradation types beyond their training data, limiting their effectiveness in real-world applications.
An alternative approach to obstruction removal is image in- painting techniques (Zeng et al., 2019; Suvorov et al., 2022), which repair or fill in missing or occluded regions by gen- erating plausible pixel values that blend with the original scene. While inpainting can produce visually convincing re- sults, these reconstructions often lack realism. For example, as shown in Fig. (c), inpainting can fill occluded areas, but the reconstructed textures, such as the woman's face, often appear unnatural. Applying these inaccurate results to downstream tasks, like object detection, depth estimation, or video analysis, can lead to errors and negatively impact practical applications.
In this work, we revisit the problem of obstruction re- moval through the lens of unified masking and introduce Instruct2See, a method that transcends traditional training- dependent solutions by generalizing beyond specific data distributions (Fig. (d)). Our distribution-agnosiic ap- proach formulates obstruction removal as soft-hard mask restoration problem, where any obstruction can be repre- sented by integrating visual semantic embeddings and text instructions, as provided by visual-language model (right part of Fig. 1). By seamlessly integrating obstruction po- sitions, visual semantics, and textual instructions, our ap- proach redefines obstruction removal as an adaptable pro- cess that fluidly transitions between hard and soft masking, effectively capturing the complexity and diversity of real- world obseured scenarios. To be specific, visual semantics help recover missing information caused by occlusions, lead- ing to more accurate scene reconstruction, while the text instruction serves a prior prompt to guide various removal tasks. Additionally, we design tunable mask adapter to
Image Inpainting. With advancements in parallel comput- ing and deep learning, numerous image inpainting methods have been developed to restore missing or damaged regions in digital images with natural and coherent content. CNN- based methods, such as PEN-Net (Zeng et al., 2019) and LaMa (Suvorov et al., 2022), have proven efficient for gen- crating local textures, but they often struggle to capture global context and handle complex patterns. To address
2

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
these limitations, transformer-based and diffusion-based models have been proposed. Deng et al. (2021) introduced the Contextual Transformer Network, which uses multi- scale, multi-head attention to capture long-range dependen- cies and global context through self-attention. Similarly, Li et al. (2022c) developed the Mask-Aware Transformer, which selectively aggregates non-local information using a dynamic mask, ensuring high fidelity and diversity in re- stored images. Diffusion-based methods like RePaint (Lug- mayr et al., 2022) combine denoising diffusion probabilistic models with conditional inpainting to iteratively generate high-quality results. More recently, Grechka et al. (2024) proposed GradPaint, a diffusion- based method that lever- ages gradient guidance to enhance the quality and coherence of inpainted regions, producing artifact-free restorations.
Inputs
Maska
Inputs
Fence
Fine Woolen
RainDrrpp
Snow
Flore
Scratch
In-distribution Obstacles
iitttf-distributtion Obstacles
Figure 2: Examples of in- and out- dittof-distrbbution obstacles.
obstruction removal, characterized by diverse and varied obstructions, ZSL approach is essential for handling a wide range of unseen obstacles effectively.
Despite these efforts, applying image inpainting methods directly to obstruction removal is challenging due to limi- tations in cross-domain applicability and the distinct focus of these methods. While inpainting aims to generate visu- ally plausible results, obstruction removal requires precise restoration to ensure data integrity for further analysis. Un- realistic outcomes can lead to errors in downstream tasks. Nonetheless, rethinking obstruction removal from the per- spective of image inpainting presents a promising avenue for future exploration.
3. Distribution-Agnosti Formulation
Seen Obstructions. As illustrated in Fig. 2, we focus on three typical but distinct types of obstructions in the training data: fences, raindrops, and flares. These obstructions were chosen for their diversity in visual characteristics and mask extraction difficulty. Fences represent obstructions with sharp distinction from the background, making it relatively straightforward to extract the mask. Therefore, we adopt a hard masking strategy for fences, as shown by the purple process in Fig. 3. In contrast, raindrops pose a challenge due to their blurred boundaries with the background and their random distribution across the image. Similarly, flares also exhibit soft, blurred edges, but their occurrence is more pre- dictable, often appearing around point light sources. Given the difficulty in extracting accurate masks for raindrops and flares, which have indistinct boundaries, we employ a soft masking approach, indicated by the green process in Fig. 3.
Vision-L Language Models (VLMs). VLMs (Zhang et al., 2024) have gained significant attention for their ability to jointly interpret visual and textual information. Pre-trained VLMs, such as CLIP (Radford et al., 2021), have demon- strated improved performance across range of downstream tasks by integrating visual and textual representations. CLIP employs a contrastive learning approach to align image and text embeddings, while distancing mismatched pairs in the embedding space. This alignment enables CLIP to perform zero-shot learning, recognizing unseen objects and concepts based on textual descriptions, achieving remarkable results without task-specific fine-tuning. In this work, we integrate the pre-trained CLIP with a prompt module to effectively leverage contextual information about degradation types, enhancing the performance of obstruction removal.
Unseen Obstructions. In addition the seen obstructions present in the training data, this work also targets more complex and varied unseen obstructions, including power cables, yarn, snow, rain streaks, scratches, and others, with examples depicted in Fig. 2. Depending on the degree of boundary ambiguity between the obstruction and the background, and the difficulty in extracting an appropriate mask, we apply soft masking for semi- tami-trranpprrent occlusions like shadow and rain streaks, where the edges are blurred. For more opaque occlusions, such power cables, yarn, and scratches, we employ hard masking due to the clearer boundary between the obstruction and the background.
Zero-Shot Learning (ZSL). ZSL is an advanced machine learning paradigm that enables models to recognize and understand instances they have never encountered during training. Unlike traditional supervised learning, which relies on labeled examples for each class, ZSLL leverages auxiliary information such as semantic attributes, textual descrip- tions, or word embeddings to generalize from seen to un- seen classes. ZSL has shown significant potential in various applications, including image classification (Naeem al., 2024), object detection (Huang et al., 2024a), and object counting (Zhu al., 2024), offering solution for scenarios with limited labeled data dynamic class distributions. In
Unified Imaging Description. The overarching objective of this work is to remove unwanted obstructions and restore the occluded background. This problem can be modeled mathematically as follows:
[(x) B(x) o (1 M())) R(x) o M(),, (1)
where T(x:) represents the input image containing obstruc-
3

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
GPT
Irput Images
Ground Truth
Hask Detecto
Acopter
Text Command
Datapase
Saft Masking
Currutt Cancet
Recansfruch
Outputs
Layer Norn
Switch
Self Attortion
Input Images
Attention
Obstruction Removal
Layer Nor
Indirnatiane
Obstruction Removal Medel
Tumable
Freage
Sefermece Phose
Instruc125ec See
Visual
Tokers
Pedtiicn Shooding
Enbeciiiny
Layer Nerm
o
Trinapirent Disenation...
Tokers
Text Commond
Pricas
Marking
Image
Muuii-modd modal Promps
Proces
User
Cutputs
Instructions
Multi-modal Prompt Generation
Common preses
Transformer Black
Figure 3: Flowchart our Instruct2Sce. Instruct2Sce accepts instructions (randomly sampled from database of instructions generated by GPT-4 during the training phase or inputted by the user during the inference phase) to flexibly activate soft masking and control the obstruction removal model for optimal capabilities.
tions, (x)) is the underlying background image to be recov- ered, R(xr) denotes the obstruction components, and (xr a binary mask. Here, refers to the pixel index. This for- mulation enables the decomposition of the input image into background and obstruction components using the mask (x:) to separate them.
in Eq. (1) as the model input, we first aim mitigate the negative impact of R model generalization. We introduce I as the input to the restoration network, defined as:
((x) = [(x) R(x) o M(x).
(2)
To achieve this, we utilize trained mask detector D(-) to estimate the mask M from T. This estimated mask used as the final mask for hard masking, while a tunable adapter A(.) (see Sec. 4.1) 1) employed for soft masking, effectively compensating for inaccuracies in M during the restoration process. The process is formulated as:
Generalization Unseen Obstructions. As outlined in Eq. (1), obstruction removal is inherently an ill- ill-posed prob- lem, it involves estimating the background scene (xx) from single input image I (xx) while accounting for the obstructions R(xx) represented by the mask ((r). Most deep learning-basse methods tackle this challenge by taking [(x) as input and predicting (xx) as output, relying on a network to learn the complex mapping from /(x) + B(x). However, these approaches are heavily data-dependent and typically perform well only obstructions within the train- ing distribution, becoming less effective when encounter- ing -it-of--istribution obstructions. This limitation arises primarily from the significant variation in the obstruction component R(x) across different classes
A(D(I(z))) if soft masking, if hard masking.
M(x) M(x D(T(2)),
Using M we remove the obstruction R(x) from the original image to obtain í. With this pre-processing, the obstruc- tion removal task is simplified to:
f(x) B(x) o (1-M)=))
(4)
To address this challenge, our approach focuses on improv- ing the model's ability generalize by explicitly consider- ing the variability in the obstruction component R(x). By designing the model to handle wide range of obstruction types beyond the training data, we enhance its capacity to perform well on unseen obstructions. This enables the model to maintain robust performance even when faced with occlusions that deviate significantly from those encountered during training.
reconstruct the clear background scene, we develop a Transformer-based obstruction removal framework (detailed in Sec. 4.3) that learns the mapping (i, (i,, T being the text instruction. Multi-modal prompts (see Sec. 4.2) are integrated using cross-attention unit to guide the reconstruction process. Finally, the network parameters θ are optimized by minimizing the objective function E(())
with
E(O) Σ -i,M,T,O))
(5)
4. Proposed Method
Fig. 3 outlines the flowchart of the proposed method, Unlike existing obstruction removal approaches that directly use
where N the number images, and f(-) represents the restoration operation.

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
4.1. Mask Generation
visual prompts help prevent overfitting and enhance zero- shot generalization, reducing dependence on specifie train- ing data.
Hard Masking and Motivation. As shown by the purple process in Fig. 3, for obstructions with clear boundaries, such as fences, we apply an accurate mask to explicitly mark the regions requiring restoration, termed hard mask- ing. However, this approach struggles when dealing with obstructions with ambiguous boundaries, such as raindrops, as it lacks flexibility in handling uncertain occlusion regions. This limitation ultimately affects restoration performance. To overcome this, we propose soft masking approach, il- lustrated by the green process in Fig. 3, enabling the model to autonomously adjust the mask based on the obstruction's s characteristics.
Soft Masking Switch. Based on the imput instructions, we use the text embeddings F (T) to activate the soft masking mode. Specifically, let the text representations for semi- transparent and opaque obstacles be denoted as (T(T,) and I' (T,),, respectively; the similarity between these represen- tations and the I, (T) can be expressed as follows:
Pe(T) (TT). (T.(T))II
sim sim(T,T.)
for E {o, (7)
Tunable Adapter for Soft Masking. The tunable adapter is designed to improve reconstruction performance by mit- igating the negative impact of inaccurate boundary esti- mates. Specifically, the tunable adapter takes the initial mask, estimated by the mask detector or manually provided, as input and outputs an optimized mask. The input first passes through a convolutional layer with batch normaliza- tion and ReLU activation. Subsequently, multiple Trans- former blocks, incorporating self-attention units and feed- forward networks, are used extract relevant features. A final convolutional layer produces the output mask. The key role of the adapter is to dynamically adjust the mask, allowing the model to determine the extent and regions where the mask should be applied based on the image fea- tures and occlusion conditions. This adaptive mechanism enhances flexibility, enabling selective restoration without strict reliance on the initial mask regions.
After calculating the similarity index, we apply the Softmax function to normalize the vector such that the sum of its probabilities equals 1. When the similarity between I' (?') and r, (T.)) exceeds a threshold 9, we enable the adapter strategy for adaptive adjustment of the mask.
4.3. Obstruction Removal
We develop restoration network based on Restormer (Za- mir et al., 2022), utilizing a Transformer based encoder- decoder architecture for image restoration tasks.
Cross-attention for Multi-modal Prompts. To efficiently use the prompt information, integrate cross-attention mechanisms into each Transformer block. By concatenat- ing the embeddings from the text and visual prompts and feeding them into the Transformer blocks, we improve the model's ability to leverage multi-modal cues during the restoration process. The cross-attention unit is defined as:
4.2. multi-modal Prompt Generation
Using only i and M as inputs presents two key challenges: l) the model lacks understanding of the required masking strategy for targeted restoration, and 2) struggles ex- tract high-level semantic information from the incomplete image, especially when encountering unseen obstructions, leading to less accurate results. To address these issues, we introduce a multi-modal prompting strategy that leverages both text instructions and visual semantic embeddings to guide the image restoration process. Specifically, we input the text instruction T' and the original image into the CLIP model's text and visual encoders (T,, r.,) to generate re- spective embeddings, which are concatenated to form the multi-modal prompt P E IR,,, where L is the number of tokens. This can be expressed as:
K Q.Kg
Q
Cross-Au(Q,, Kp. Vpp) = Softmax
Vps
(8)
where a temperature factor, and K₂ and V, represent the key and value obtained from the multi-modal prompt. while Q represents the query derived from the degraded image feature.
5. Experiments and Analysis
5.1. Experiment Settings
Implementation Details. Our Instruct2See framework implemented in PyTorch 1.12.0 and trained on system equipped with AMD EPYC 7543 32-Core CPUs and 8 NVIDIA L40 GPUs. We train the model using the AdamW optimizer (3) = 0.9, 32 = 0.999, weight decay of x 10-4) and LI loss, over 300K iterations. The initial learning rate is set to 3 x 10 4 A progressive learning strategy is employed, starting with a patch size of 128 x 128 and a batch size of 1. The patch size is progressively updated to 128 x 128,
P = (TT))
R..
(6)
A cross-attention mechanism is then applied to integrate these prompts with the image features, guiding the re- construction process. By incorporating text prompts, the model's ability to adapt its masking strategy improves, while
5

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Table I: PSNR, SSIM, CLIP Score (CLIPS), and LPIPS comparisons of different methods on seen obstructions. The best and second best results are highlighted in bold and underlined.
Mothod
Venue
Fence
Flare
Roindrop
Average
PSNRT
SSIMT
CLIPST
LPIPS.
PSNRT
SSIMf
CLIPST
LPIPS
PSNR?
SSIM?
CLIPST
LPIPS)
PSNRT
SSIMT
CLIPST
LPIPS,
Restormer
CVPR22
29.86
0.9170
0.9045
0.0986
25.41
0.9162
0.9547
0.0772
30.07
0.9542
0.9670
0.0482
28.45
0.9291
0.9421
0.0746
CVPR22
0.8492
0.8964
0.9225
0.0787
27.52
0.9013
0.0922
PrompiR
NeurIPS23
24.59
0.7423
0.8957
0.1521
25.43
0.9187
0.9604
0.0721
31.95
0.9668
0.9679
0.0471
27.32
0.8759
0.9413
0.0904
WGWSNet
CVPR23
23.19
0.7878
0.8561
0.2884
25.87
0,9192
0.9486
0.0710
32.89
0.9671
0,9706
0.0486
27.32
0.8914
0.9251
0.1360
Histolormer
ECCV24
28.05
0.9001
0.8971
0.1134
25.19
0.9195
0.9541
0.0712
31.59
0.9614
0.9584
0.0548
28.28
0.9270
0.9365
0,0798
XRestormer
ECCV24
27.11
0.8972
0.9037
0.1430
24.89
0.9185
0.9487
0.0743
30.55
0.9583
0.9557
0.0482
27.52
0.9247
0.9360
0.0924
Instruci2See
ICML25
30.15
0.9079
0.9093
0.0941
25.15
0.9202
0.9509
0.0694
32.64
0.9680
0.9723
0.0446
29.31
0.9320
0.9442
0.0694
Fence
Flare
Rrin
(8) Inputs
(b) PromptiR
(c) WGWSNel
(d) Histoformer
(e) XRestomer
(f) Instruci2See
(g) Ground Truth
Figure 4: Visual comparisons of our method with other approaches on seen obstructions.
160 x 160, 192 x 192, and 256 x 256 at iterations 15,000, 80,000, 60,000, and 45,000, respectively. We also apply horizontal and vertical flips for data augmentation.
Additionally, 500 raindrop test image pairs are included. For unseen obstructions, we sourced 100 test images each from the rain streak dataset (Yang et al., 2017), snowy dataset (Liu et al., 2018), and stroke dataset (Lugmayr et al., 2022). We also tested our method on special obstacles to verify its zero-shot capability.
Datasets. We utilize 3,984 images for model training. For the fence obstacle, select 897 clear images from the BSD dataset (Martin al., 2001) and the UCID dataset (Schaefer & Stich, 2003), and generate paired data using the fence synthesis method from (Du et al., 2018). Additionally, 987 clear images from the Flickr24K dataset (Zhang et al., 2018) and 987 flare images from the Flare7K dataset (Dai et al., 2022) are used to create flare image pairs. We also include 2,100 training image pairs from the VRDS dataset (Wu et al.,
5.2. Comparisons with the State-of-the-Arts
Results on Seen Obstructions. The quantitative evalu- ation results for seen obstructions are presented in Table 1. We compare the obstruction removal performance of various methods by using detected masks. While our pro- posed method is slightly outperformed by certain state-of- the-art methods in specific tasks based on PSNR, SSIM,
2023). For testing, we apply the same synthesis strategy create fence test dataset with 100 image pairs. Moreover, flare test dataset with another 100 image pairs is used.
6

---

Instruct2Sce: Learning to Remove Any Obstructions Across Distributions
Table 2: PSNR, SSIM, CLIP Score (CLIPS), and LPIPS comparisons of different methods on inseen obstructions. The best and second best results are highlighted in bold and underlined.
Method
Venue
Rain Streak
Snow
Stroke
Average
PSNR1
SSIM
CLIPS
LPIPS
PSNRT
SSIMP
CLIPS
LPIPS|
PSNR
SSIMP
CLIPS1
LPIPS)
PSNRT
SSIM
CLIPSt
LPIPS
Restormer
CVPR22
26.19
0.8381
0.9133
0.2385
28.81
0.9013
0.9355
0.1508
21.14
0.8173
0.8012
0.2061
25.38
0.8522
0.8833
0.1984
TransWeathrr
CVPR22
26.70
0.8341
0.8989
0.2450
29.53
0.8926
0.9455
0.1317
18.27
0.6752
0.8014
0.3598
24.83
0.8006
0,8819
0.2455
Prompti
NeurIPS23
24.04
0.7197
0.3869
0.2211
26.01
0,7457
0.9370
0.1185
29.39
0.9021
0.8533
0.0970
26.48
0.7892
0.8924
0,1455
WGWSNet
CVPR23
29.68
0.9111
0.9422
0.1411
29.54
0.8944
0.9439
0.1273
28.25
0.8722
0.8791
0.1222
29.18
0.8927
0.9217
0.1302
Histoformer
ECCV24
27.99
0.8634
0.8854
0.1870
32.40
0.9203
0.9421
0.0909
28.07
0.8761
0.8379
0.1206
29.49
0.8866
0.8885
0.1328
XRessommr
ECCV24
28.05
0.8560
0.9138
0.2108
31.31
0.9170
0.9567
0.1040
19:00
0.7588
0.7754
0.2472
26.12
0.8439
0.8820
0.1873
Instruct2Sec
ICML25
29.82
0.8907
0.9296
0.1639
34.85
0.9283
0,9618
0,0639
29.45
0.9067
0,8507
0.0936
31.37
0,9086
0.9140
0.1071
CLIPS, and LPIPS metrics, the overall results clearly high- light the strengths and advantages of our approach. Notably, our method achieves a PSNR that is 0.86 dB higher than the second-bes best method, Restormer, demonstrating its su- perior ability to preserve image quality. Furthermore, shown in Fig. 4, visual comparisons across the three ob- struction removal tasks consistently emphasize the strengths of our approach, particularly in reconstructing fine details and maintaining scene coherence. For additional numeri- cal experiments under ideal conditions (using ground truth masks), please refer to the Appendix.
Table 3: PSNR and SSIM comparisons of our method with inpainting/cditing based methods unseen obstructions. The best results are highlighted in bold.
Method
Venue
Rain Streak
Snow
Stroke
Average
PSNR SSMTTT SSIMM P PSNR SSIM SSMM
LaMa
WACV22 29.07 0. 18888 32.32 0.9108 28,10 0,8728
29.83 0,3898
RePaint
CVPR22
28.78
0.8865 32,20 0,9064
23.78
28.25 0.8662 0.6072
0.8059
DiffEdit
ICLR23
23.88 0.6561 24.23 0.6732 11.65
19.92 0.6455
Insimcr2SeeICM1..5 29.82 0.8907 34.85
0.9067
31.37 0.9086
in the zero-shot obstruction removal. Table 3 presents the quantitative evaluation results for these methods and ours across three classic obstacle removal scenarios. In particu- lar, the editing- based method (DiffEdit) has poor numerical performance, while inpainting- based methods fail to accu- rately reconstruct details. These stem from their design as image inpainting and editing tools, which strictly con- strain changes to masked regions and lack the flexibility to handle arregllarshappd holes. The results indicate that, despite using obstacle masks as inputs, existing inpainting and editing methods still struggle to effectively address this problem. In contrast, our method, which incorporates a tun- able mask adapter and multimodal feature representation of obstacles, demonstrates superior performance in zero-shot obstacle removal tasks. Fig. 6 further visualizes additional obstacle removal results. It is evident that while LaMa and RePaint exhibit some obstacle removal capabilities, residual obstacles remain. Conversely, our Instruct2See effectively handles various situations and robustly removes obstacles.
Results on Unseen Obstructions. To further evaluate the zero-shot learning capability of our model, we conducted experiments on images containing unseen obstructions. Ta- ble 2 presents the PSNR, SSIM, CLIPS, and LPIPS results for obstruction removal on three classic obstacles not in- cluded in the training. With the exception of slightly lower CLIPS compared to WGWSNet, our method consistently delivers the best results across the unseen obstruction. As shown in Fig. 5, visual comparisons across additional ob- structions, such as yarn, scratches, spots, and power cables, further demonstrate the strong generalization ability of our proposed method. While existing methods often struggle or show limited effectiveness in addressing out-of-distribution obstructions, our approach consistently produces more ac- curate and realistic restorations. This confirms the perfor- mance boost provided by our multi-modal prompt strategy and tunable adapter, which enable effective zero-shot leam- ing and allow our model to capture the nuances of unseen obstructions. These results validate the robustness and flex- ibility of our method, making it a promising solution for real-world applications where diverse and unpredictable obstructions are common.
5.3. Ablation Study
Effectiveness of Network Modules. Table 4 (a) presents a comprehensive quantitative evaluation of different module configurations on three seen and three unseen obstructions. The results clearly show that the baseline model alone pro- duces poor results, indicated by the low PSNR and SSIM scores. Introducing mask improves performance slightly, but the enhancement remains limited. This is primarily be- cause the model struggles to distinguish between different types of obstructions and cannot effectively select the ap- propriate masking strategy. Additionally, without proper
In addition, the inpainting- and editing-based based methods hibit strong zero-shot capabilities and perform well in var- ious object removal or image editing tasks, making them viable solutions for the obstruction removal task. There- fore, for a comprehensive comparison, we evaluated our proposed method against two inpainting methods (LaMa (Suvorov et al., 2022) and RePaint (Lugmayr et al., 2022)) and one editing method (DiffEdit (Couairon et al., 2023))
7

---

Instruc(2See: Learning to Remove Any Obstructions Across Distributions
Yarns
Scratches
Spots
Power Cables
(a) Inputs
(b) PromptIR
(c) WGWSNet
(d) Histoformer
(e) XRestormer
(f) Instruct2See
Figure 5: Visual comparisons our method with other approaches on unseen obstructions.
understanding of scene semantics, the model generates unre- alistic and anomalous restoration outcomes. In contrast, in- corporating the cross-attention mechanism, which integrates textual instructions with visual semantics, significantly im- proves performance, leading to PSNR increase of 1.95 and an SSIM increase of 0.0113. This mechanism enables the model to better grasp the contextual relationship be- tween the obstruction and the surrounding scene, producing more coherent and realistic restoration results. Finally, the tunable adapter can further enhance the ability to handle obstructions with blurred boundaries, resulting in optimal effects across all metries. This verified that model can adaptively manage different obstruction scenarios, leading to more refined and effective restoration process.
Table 4: PSNR and SSIM comparisons of integrating dif- ferent modules (a) and using different prompt strategies (b). Pt and P. represent text and visual prompts, respectively.
mask
CA
Adapter PSNR*
SSIM
P.
PSNR*
SSIMI
27.05
0.8920
28.65
0.9063
28.05
0.9004
29,73
0.9168
4
30.00
0.9117
30.25
0.9215
4
0.9250
0.9250
(a)
(b)
When using only the textual embedding, the model can adopt the correct strategy to handle both sharp and blurred mask boundaries. However, due to the absence of complete image semantics from occluded regions, the model often produces unrealistic or inconsistent results. Using only the visual encoder strategy better compensate for the se- mantic loss caused by obstacle removal, thereby achieving better results than introducing only the text encoder. Finally, by integrating both visual semantics and textual instructions through a multi modal prompt, the model can easily handle obstruction removal tasks for both in-distribution and out-of-
Effectiveness of Different Prompts. Table (b) compares performance using different prompt strategies. Without prior prompts, the model performs poorly, primarily due confusion over the correct masking strategy and lack of semantic understanding, especially for unseen obstructions.
8

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Strake
(a) Inpul
w/o Adapler
(c) Full Model
Ground Truth
Figure 8: Visual comparisons using our tunable adapter on a raindrop obstruction case.
Spots
adapter for soft masking to address inaccuracies in occlu- sion regions. This adapter dynamically adjusts the mask, enabling model determine the extent of mask applica- tion based on the image features, rather than being confined to predefined mask area. To evaluate the function and effectiveness of our proposed adapter, we conducted an ab- lation study. As shown in Fig. 8, a comparison between the adjusted and original masks demonstrates that the adapter effectively refines the mask for uncertain obstructions, such raindrops. The original mask often overly covers the restoration area, leading to loss of detail and suboptimal reconstruction. In contrast, the adjustments made by the tunable adapter ensure more accurate and reliable restora- tion by preserving crucial details in the occluded regions. Additionally, the ablation study reveals that the tunable adapter significantly improves the model's adaptability to various obstructions with ambiguous boundaries, resulting in PSNR increase of 0.93 and an SSIM gain of 0.0133 com- pared to the fixed mask approach. These findings confirm that the tunable adapter not only optimizes mask coverage but also plays a vital role in refining the restoration process.
Snow
(a) Imputs
LaMa
(c) RePaint
(d) Instruci2See
Figure Visual comparisons of our method with inpainting
methods on unseen obstructions.
Modia Ground Figure Visual comparisons of using different prompt strategies on a snow obstruction case.
distribution obstacles. The multi-modal prompt strategy not only improves the model's interpretability but also strength- its ability generalize to unfamiliar obstructions.
6. Conclusion
In this work, proposed Instruct2See, novel zero-shot obstruction removal framework designed to effectively ad- dress challenges posed by both in-distribution and out- of-distribution obstructions. By leveraging multi-modal prompts that integrate visual semantics and textual descrip- tions through cross-attention mechanism. Instruct2See demonstrated superior performance in accurately recon- structing occluded scenes. The inclusion of tunable adapter for soft masking further improved adaptability, al- lowing the model to handle ambiguous boundaries with greater flexibility. Extensive experiments validated the effi- cacy and generalization capabilities of Instruct2See, high- lighting its potential as a robust solution for real-world ob- struction removal tasks.
To further illustrate the impact of multi-modal prompts, we compared different strategies in snow obstruction case, as shown in Fig. 7. Without any prompt, the model shows only a slight reduction of the snow obstacles, Introduc- ing a textual prompt (Fig. 7(c)) allows the model to focus more masking strategy, leading to better suppression of the obstruction. However, using only the textual prompt introduces unnatural artifacts in the reconstruction, as the model lacks complete semantic information from the oc- cluded regions. By incorporating the visual encoder from the visual-language model, we effectively compensate for missing semantic details during obstruction removal. This approach preserves the model's robust zero-shot learn- ing capability while enabling it to extract relevant details and accurately represent various obstructions, even those not encountered during training. Consequently, the multi- modal prompt strategy delivers superior visual restoration and enhanced obstruction suppression performance.
Limitations. Our method targets small, numerous obstacles (e.g., raindrops and fences) and explores optimal strategies for removing both opaque and translucent obstacles. It is not intended for large obstructions, as focuses on scene recovery through contextual cues. In such cases, inpainting techniques may be more effective.
Effectiveness Tunable Adapter. We designed a tunable

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Acknowledgements
Du, C., Kang, B., Xu, Z., Dai, J., and Nguyen, T. Accurate and efficient video de-fencing using convolutional neural networks and temporal information. In ICME, pp. 1-6, 2018.
This work supported by the Guangdong Natural Sci- ence Funds for Distinguished Young Scholars (Grant 2023B151502009) the GuangDong Basic and Applied Ba- sic Research Foundation (2025A (1551011244 the National Research Foundation, Singapore under its Al Singapore Pro- gramme (AISG Award No: AISG3-GV-2023-011) and the Lee Kong Chian Fellowships.
Du, Y., Han, G., Tan, Y. Xiao, C., and He, S. Blind image denoising via dynamic dual learning. IEEE TMM, 23: 2139-2152, 2020.
Grechka, A., Couairon, G., and Cord, M. Gradpaint: Gradient-guided inpainting with diffusion models. CVIU, 240: 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guo, Y., Gao, Y., Lu, Y., Zhu, H., Liu, R. W., and He, S. Onerestore: A universal restoration framework for composite degradation. ECCV, 2024.
Han, J., Li, W. Fang, P., Sun, C., Hong, J., Armin, M. A., Petersson, L.,, and Li, H. Blind image decomposition. In ECCV, pp. 218-237, 2022.
References
Chang, W., Chen, H., He, X., Chen, X., and Shen, L. Uav- rainlk: A benchmark for raindrop removal from uav aerial imagery. In CVPR, pp. 15-22, 2024.
Huang, P., Zhang, D., Cheng, D., Han, L.., Zhu, P., and Han, M-rrfs: memory-based robust region feature synthesizer for zero-shot object detection. IJCV, pp. 1-22, 2024a.
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., and Gao, W. Pre-trained im- age processing transformer. CVPR, pp. 12299-12310, 2021.
Huang, Q. Xu, C.,, Li. G., Wu, Z., Liu, S., and He, S. Portrait shadow removal via self-exemplar illumination equalization. In ACM MM, pp. 7474-7482, 2024b.
Chen, S., Ye, T., Bai, J., Chen, E., Shi, J., and Zhu, Sparse sampling transformer with uncertainty-drive driven ranking for unified removal of raindrops and rain streaks. In ICCV, pp. 13106-131177 2023a.
Li, B., Liu, X., Hu, P., Wu, Z., Lx, J., and Peng. X. All-in- one image restoration for unknown corruption. In CVPR, pp. 17452-17462, 2022a.
Chen, Z., Sun, Y., Bi, X., and Yue, Lightweight image de-snowing: A better trade-off between network capacity and performance. Neural Networks,
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision- vision-language understanding and generation. In ICML, 2022b.
2023b.
Chugunov, I., Shustin, D., Yan, R., Lei, C., and Heide, F. Neural spline fields for burst image fusion and layer separation. In CVPR, pp. 25763 25773,,2773,.. 2024.
Li, R., Tan, R. T., and Cheong, L.-F. All in one bad weather removal using architectural search. In CVPR, pp. 3175- 3185, 2020.
Couairon, G., Verbeek, J., Schwenk, H., and Diffedit, M. Diffusion-bart based semantic image editing with mask guid- ance. In ICLR, 2023.
Li, W., Lin, Z., Zhou, K., Qi, L., Wang, Y., and Jia, Mat: Mask-aware transformer for large hole image inpainting. In CVPR, pp. 10758-10768. 2022c.
Dai, Y., Li, C., Zhou, S., Feng. R., and Loy, C. C. Flare7k: A phenomenological nighttime flare removal dataset. NeurlPS, 35:3926-3937 2022.
Li, Y., Monno, Y., and Okutomi, M. Dual-pixel raindrop removal. IEEE TPAMI, 2024.
Liu, Y.-F,, Jaw, D.-.., Huang, S.-C., and Hwang, J.-N. Desnownet: Context-aware deep network for snow re- moval. IEEE TIP, 2766)3304 2018.
Dai, Y., Luo, Y., Zhou, S., Li, C., and Loy, C. C. Nighttime smartphone reflective flare removal using optical center symmetry prior. In CVPR, pp. 20783-20791, 2023.
Liu, Y.L.,, Lai, W.-S., Yang, M.-H., Chuang, Y.YY,, and Huang, J.-B. Learning to see through obstructions with layered decomposition. IEEE TPAMI, 44(1 8)8877788002 2021.
Deng, Y., Hui, S., Zhou, S., Meng, D., and Wang. J. Learn- ing contextual transformer network for image inpainting. In MM, pp. 2539-2538, 2021.
10

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Lugmayr, A., Danelljan, M., Romero, A., Yu, F. Timofte, R., and Van Gool, L. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pp. 11461- 11471,2000 2022.
Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong, N., Goka, H., Park, K., and Lempitsky, V. Resolution-robust large mask inpaint- ing with fourier convolutions. In WACV, pp. 2149-2155 2022.
Martin, D., Fowlkes, C., Tal, D., and Malik, A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, volume 2, pp. 416-423, July 2001.
Tsogkas, S., Zhang, F., Jepson, A., and Levinshtein, A. Efficient flow-guided multi-frame de-fencing. In WACV, pp. 1838-1847, 2023.
Valanarasu, J. M. J., Yasarla, R., and Patel, V. M. Tran- sweather: Transformer-based restoration of images de- graded by adverse weather conditions. In CVPR, PP. 2353-2363, 2022.
Naeem, M. F., Xian, Y., Gool, V., and Tombari, 12dformer+: Learning image to document summary at- tention for zero-shot image classification. IJCV, pp. 1-17, 2024.
Wang. C., Zheng. Z., Quan, R., Sun, Y., and Yang, Y. Context-aware pretraining for efficient blind image de- composition. In CVPR, pp. 18186-18195, 2023a.
Özdenizci, O. and Legenstein, R. Restoring vision in ad- verse weather conditions with patch-based denoising dif- fusion models. IEEE TPAMI, 4336611034, 2023.
Wang, Y., Ma, C., and Liu, J. Smartassign: Learning smart knowledge assignment strategy for deraining and desnowing. In CVPR, pp. 3677-3686, 2023b.
Potlapalli, V., Zamir, S. W., Khan, S., and Khan, F. Promptir: Prompting for all-in-one image restoration. In NeurlPS, 2023.
Wen, Q.. Tan, Y., Qin, Liu, W., Han, G., and He, S. Single image reflection removal beyond linearity. In CVPR, pp. 3771-3779, 2019.
Qian, R., Tan, R. T., Yang, W., Su, J., and Liu, J. Attentive generative adversarial network for raindrop removal from single image. In CVPR, pp. 2482-22222222 2491, 2018.
Wu, H., Yang. Y., Chen, H., Ren, J., and Zhu, L. Mask- guided progressive network for joint raindrop and rain streak removal in videos. In MM, pp. 7216-7225, 2023.
Quan, Y., Huang, H., He, S., and Xu, R. Deep video demoiréing via compact invertible dyadic decomposition. In ICCV, pp. 12677-12686, 2023a.
Yang, W., Tan, R. T., Feng, J., Liu, J., Guo, Z., and Yan, S. Deep joint rain detection and removal from a single image. In CVPR, pp. 1357-1366, 2017.
Quan, Y., Tan, X., Huang, Y., Xu, Y., and Ji, H. Image desnowing via deep invertible separation. IEEE TCSVT, 33(():3133-315 2023b.
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, S., and Yang, M.-H. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pp. 5728- 5739, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, 2021.
Zeng, Y., Fu, J., Chao, H., and Guo, B. Learning pyramid- context encoder network for high-quality image inpaint- ing. In CVPR, pp. 1486-1494 2019.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al. Sam 2: Segment anything in images and videos. arXiv preprint ar XrXiv:2408.007 14, 2024.
Zhang, D.,, Ouyang, J., Liu, G., Wang, X., Kong, X., and Jin, Ff-former: Swin fourier transformer for nighttime flare removal. In CVPR, pp. 2824-2832, 2023a.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In MICCAI, pp. 234-241, 2015.
Zhang, H., Xu, X., He, H., He, S., Han, G., Qin, J., and Wu, Fast user-guided single image reflection removal via edge-aware cascaded networks. IEEE TMM, 22(8): 2012 2112-202,,22 2019.
Schaefer, G. and Stich, M. Ucid: An uncompressed color image database. In Storage and retrieval methods and applications for multimedia 2004, volume 5307, pp. 472- 480. SPIE, 2003.
Zhang. J., Huang, J., Jin, S., and Lu, S. Vision-language
models for vision tasks: A survey. IEEE TPAMI, 2024.
Sun, S., Ren, W., Gao, X., Wang, R., and Cao, X. Restor- ing images in adverse weather conditions via histogram transformer. ECCV, 2024.
Zhang, X., Ng, R., and Chen, Q. Single image reflection separation with perceptual losses. CVPR, pp. 4786 4794, 2018.

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Zhang, Z., Wei, Y., Zhang, H., Yang, Y., Yan, S., and Wang, M. Data-driven single image deraining: A comprehensive review and new perspectives. PR, 103:109740, 2023b.
Zhou, Y., Liang, D., Chen, S., Huang. S.-J,, Yang. S., and Li, C. Improving lens flare removal with general-purpose pipeline and multiple light sources recovery. In ICCV, pp- 12969-12979, 2023.
Zhu, H., Yuan, J., Yang. Z., Guo, Y., Wang, Z., Zhong, X., and He, S. Zero-shot object counting with good exemplars. ECCV, 2024.
12

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
A. Implementation Details
A.1. Model Inference
As outlined in Algorithm 1, our model operates in two modes during inference: opaque obstruction removal using hard masking and transparntt obstruction removal using soft masking. The process begins with the image I, containing the obstruction, and text instruction T. The overall obstruction removal procedure is divided into three key steps: mask generation, multi-modal prompt generation, and obstruction removal.
Algorithm Instruct2See Model Inference
1: I: input image, B clear background, M: initial mask, M: adapted mask, 7: input image cutout by M, T: text instruction, T3: "semi-transparent obstacle" text, D()): mask detector, A(-): tunable adapter, R):,;;)): Obstruction Eliminator, re(:): visual language model's ' text encoder, (-): visual language model's visual encoder, Ps: text prompt, Pa: visual prompt, P: multi-modal prompt, concat: embedding splicing operation
Input: I.T
2: M -- D(T) (Initial mask generation.}
3: Pt Ix(T)
4: P. (T(
5: if ifsim (T.)).(T))) >> 0 then
6: M + A(M) {Soft masking.)
7: else
8: M = M (Hard masking.}
9: end if
P concat Pol (Multi-modal prompt generation.)
11: get i by cutting out the region in M from I
12: B t- T-R M, P) {Obstruction removal.)
13: return B
Mask Generation. We first extract the initial mask M from the input image I using mask detector (as described in line 2 of Algorithm 1). For obstructions like rain streaks and snow, which are more challenging to segment, we employ a U-Net-based model (Ronneberger et al., 2015) to generate the initial mask. For other obstructions, we use the Segment Anything Model 2 (SAM2) (Ravi et al., 2024). Depending on the type of obstruction, different masking strategies are applied: for opaque obstructions with clear boundaries, we directly use M the final mask M (lines 7-9), while for semi-transparent obstructions with blurred edges, refine M using tunable adapter to improve performance (lines 5-7).
Multi-Modal Prompt Generation. We process the inputs / and T using the text and image encoders of the Contrastive Language- Image Pre-training (CLIP) model (Radford et al.. 2021) to obtain textual and visual embeddings (Pr, P.). These embeddings are then concatenated to generate the multi-modal prompt P. as described in lines 3, 4, and 10 Algorithm 1.
Obstruction Removal. With the refined mask M and the multi-modal prompt P, we first use M mask out the obstructions in L, generating 7. Then, I and M are concatenated along the channel dimension and, along with P. input into the obstruction removal model R--) (lines 11-13). A cross-attention module within R(-) fuses the image features with the multi-modal prompt. Specifically, features from the image map extracted using convolution as the query, while the multi-modal prompt generates key and value vectors via two independent linear layers. These vectors are fused using multi-head attention mechanism, guiding the network to effectively remove unknown obstructions.
A.2. CLIP Usage
In the use of CLIP!, the visual encoder is employed to extract visual features from the original image to compensate for the loss of visual semantics caused by obstacle cutout. Since the pre-trained CLIP visual encoder already possesses strong semantic representation capabilities, we do not perform additional fine-tuning on this module. The text embeddings, however, provide specific removal prompts to the model. Due to the relatively few specific instructions for obstacle removal CLIP's original training, the original embedding space may not be suitable for this task (i.e., the embeddings generated by text instructions for the same goal may exhibit significant variability). Therefore, we only fine-tune CLIP's text encoder. The
We utilize the CLIP ViT-B/32 model.
13

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Contrastive Text Encoder Fine-tuning
Text Command Database
Trinslucent objest in the Lmsage.
"Trändlusent object bloskes."
Enccder
Text
The pichort contRing translmeent barrier. Please the tranclusent dbject from the image.
"This imagge sontDing opaque obstacle."
An opeque object is otstrrcttng part of the image, makinn it olfffouutt to see what lies behindd it.
Share Weight
Ta
T4
"The view in the image is obstrwuaeed by a semii transppeent objest, mulkinng it diffiouutt what lies behinds it
The image contains ai opaque objest that needs to addrressed to impprove visibiliiy. opaque obstruction.
Text Encoder
Ti
T:T: TiT
TeTaTTTT
"OpRquE"
Figure 9: Contrastive fine-tuning of CLIP text encoder.
Table PSNR and SSIM comparisons of different methods on seen obstructions. The scheme using the GT mask as input designed to demonstrate the obstruction removal capabilities of each model under ideal conditions.
Method
Fence
Flare
Raindrop
Average
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR1
SSIM
Restormer
29.86
0.9170
25.41
0.9162
30.07
0.9542
28.45
0.9291
GT mask
29.62
0.9166
25.38
0.9145
32.04
0.9651
29.01
0.9321
Weather w GT mask
26.93
0.8492
25.18
0.9040
30.44
0.9508
27.52
0.9013
29.12
0.8727
26.05
0.9150
30.58
0.9510
28.58
0.9129
PromptR
24.59
0.7423
25.43
0.9187
31.95
0.9668
27.32
0.8759
w GT mask
26.41
0.7842
25.63
0.9193
32.71
0.9691
28.25
0.8909
WGWSNet
23.19
0.7878
25.87
0.9192
32.89
0.9671
27.32
0.8914
GT mask
26.88
0.8467
25.50
0.9193
32.26
0.9648
28.21
0.9103
Histoformer GT mask
28.05
0,9001
25.19
0.9195
31.59
0.9614
28.28
0,9270
32.29
0.9382
25.96
0.9106
32.29
0.9636
30.18
0.9375
XRestormer 27.11
0.8972
24.89
0,9185
30.55
0,9583
27.52
0,9247
w GT mask
30.57
0.8972
25.44
0.9176
31.02
0.9599
29.01
0.9249
Instruct2See
30.15
0.9079 0.9329
25.15 25.83
0.9202
32.64
0.9680
29.31 30.49
0.9320
w mask
32.12
0.9203
33.52
0.9706
0.9413
fine-tuning strategy for the CLIP text encoder is shown in Fig. 9. We first collected the text instructions corresponding to each image in our training dataset to construct text command database. This database contains two categories: instructions for removing opaque obstacles and instructions for removing semi-transparent obstacles. Subsequently, we fine-tuned the model using a contrastive pre-training strategy similar to CLIP.
More specifically, in each iteration, we randomly select text instructions in the database and use the CLIP text encoder to generate text embeddings for opaque obstacles (T2 and Tạ) and two text embeddings for semi-transparent obstacles (Ti and Ts). Subsequently, the same category as positive samples, while the values calculated between different categories are designated negative samples. Finally, we perform contrastive training based on the clip loss (Radford et al., 2021).
calculate the cosine similarity between each pair and designate the values calculated between
Additionally, the tunable adapter is only activated for semi remi-transparent obstacles. To selectively enable this function based on the input instruction, we set up two-word embeddings: "opaque" and "semi-trannspaeent.. By calculating the cosine similarity between the instructions embedding and these two embeddings, we can determine whether to activate the adapter module. Therefore, this fine-tuning strategy allows our model to more accurately judge when enable the adapter.
B. More Experimental Results
B.I. More Numerical Results
To demonstrate the upper bound of seen obstacle removal under ideal conditions, we compared the restoration performance of various methods using both detected masks and ground-tree -trtt masks. As shown in Table 5, the results show that higher- quality masks benefit most methods. our task, the mask detector is plug-and-play, and future adoption more powerful
14

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
PSNR
SSIM
Fence
Fence
Stroke
Stroke
Flare
Flare
Reslormer
TransWeather
RaimDrop
RaimDrop
PromptiR
Snow
Snow
WGWSNet
Histoformer
XRestormer
Rain Streak
Rain Streak
Instruct2See
Figure 10: PSNR and SSIM comparisons of different methods on seen and unseen obstructions.
mask detectors could further enhance the obstruction removal capability. Additionally, we present radar chart (Fig. 10) comparing the PSNR and SSIM results for three types of seen obstacles and three classic unseen obstacles. The results demonstrate that the proposed model delivers robust performance across various scenarios, validating its effectiveness and strong generalization ability.
B.2. Zero-Shot Single Obstruction Removal
This section presents additional examples of removing various unseen obstructions, Fig. 11 compares the results of different methods on three typical obstruction removal tasks. It is evident that TransWeather and XRestormer perform poorly in the stroke removal task, failing to handle such cases effectively. Other methods also produce distorted facial details during restoration. For traitrranpprrent obstructions, such as raindrops and snow, these methods fail to properly capture the relationship between the obstruction and mask, leading to ineffective or minimal removal.
In contrast, our method employs hard-soft masking strategy, allowing smooth transitions between hard and soft masking. This enables it to capture the complexity and diversity of real-world occlusion scenarios effectively. Fig. 12 showcases further experiments on uncommon obstructions, demonstrating the zero-shot generalization capability of our method. This advantage stems from our distribution-agnoss approach, which formulates obstruction removal as a soft-hard masking problem, representing any obstruction through the integration of visual semantic embeddings and textual instructions.
B.3. Zero-Shot Multiple Obstruction Removal
Fig. 13 displays three visualization cases multiple obstruction removal. It is evident that our method can accurately represent specified obstructions through multi-modal prompts and masks, and easily climinate them using the designed model. From the results, it appears that only when there are occlusions between multiple obstacles does the elimination of one obstacle inevitably affect another. The order of obstruction elimination does not have significant impact on the results.
Table 6: PSNR and SSIM comparisons of using different prompt generation strategies.
Model
PSNR1
SSIM
Instruct2See + CLIP
30.93
0.9250
Instruct2See + BLIP
31.01
0.9235
B.4. Metric Influence of Using Different Prompt Generation Models
In this section, we compared the effects of using CLIP's and BLIP's encoders to generate textual and visual embeddings on obstacle removal results. The experimental results are shown in Table 6. Clearly, whether using CLIP (Radford al., 2021) or BLIP (Li et al., 2022b), model can generate robust obstacle removal effects, proving that model can adapt to commonly used pretrained encoders.
C. Failure Cases using Incorrect Description
Fig. 14 illustrates two examples of using incorrect descriptions. In the stroke removal case, when an instruction of a semi-transparent obstruction removal used for scene with an opaque obstruction, our model tends to treat the original
15

---

Instruc(2See: Learning to Remove Any Obstructions Across Distributions
(a) Imputs
Restormer
(c)
TransWeather
PromptiR
(e) WGWSNet
(f) Histaformer
XRestormer
Instruci2See
Ground Truth
Figure 11: Visual comparisons on three classic unseen obstructions (rain streak, snow, and stroke).
opaque obstruction as part of the real information, leading to suboptimal results. Accurately describing the obstacle as an opaque obstacle can easily resolve this issue. Similarly, in the shadow removal case, using an opaque obstruction description will make the masked image content completely invisible, resulting in restoration that does not match reality, especially in cases of large-area occlusions. However, correcting the instruction to remove the transparent obstacle can solve this problem.
D. Complexity Analysis
In this section, we present the model sizes various methods and calculate their Floating Point Operations (FLOPs) and runtime on 224x224 images. As shown in Table 7, although our model has the highest number of parameters due the introduction of cross-attention module integrated with multi-modal prompts and an adjustable mask adapter, its FLOPs and inference speed remain at moderate level compared to competing methods. In the future, will consider maintaining the model's strong zero-shot generalization capabilities while reducing computational costs.
Table Comparisons of parameters, FLOPs, and runtime between.
Model
Venue
Parameters (M)
FLOPs (G)
Runtime (ms)
Restormer
CVPR22
26.13
118.60
49.37
TransWeather
CVPR22
38.06
3.57
19.64
PrompuI
NeurIPS23
35.59
132.26
53.95
WGWSNet
CVPR23
4.70
96.65
88.39=0.35
Histoformer
ECCV24
16.62
86.79
83.13=0.82
XRestromer
ECCV24
22.34
155.49
100.67::044
Instruct2See
ICML25
56.69
146.23
84.28=0.61
16

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Shadow
Watermarks
ABC
Watermark
na
Travel
na
Inputs
Histoformer
Instruct2See
Inputs
Histoformer
Instruct2See
Figure 12: Visual comparisons more uncommon obstructions.
17

---

Instruct2See: Learning to Remove Any Obstructions Across Distributions
Input
Spots removal
Soratches removal
Remove scratches, then spots
Remove spots, then scratches
Ground Truth
Input
Fence removal
Scratches removal
Remove scratches, then fence
Remove fence, then scratches
Ground Truth
Input
Fence removal
Watermark removal
Remove watermark, fence
Remove fence, then watermark
Ground Truth
Figure 13: Visual results multiple obstruction removal.
Inpul (Stroke)
Outpul_S
Qulput_
Input (Shadow)
Inpul.
Input S
Oqaqus atstruction
Tixc: Pissas remuave couu--trakssavveet obstsoies the humdee' fane.
opaque
Comil-nnnaspareec plsttrcction.
Toxt: Coaque blech my view
Tharallcact shoolows affect auy oicervation.
Figure 14: Visual comparisons of using different text descriptions.
18


Reviews:
[{"review_id": "CXXqn2Yifu", "reviewer": "Reviewer_CuLC", "summary": "This paper studies obstruction removal of 2D images. The proposed model is a zero-shot method that can handle both seen and unseen obstacles in open vocabulary. The method is to obtain a mask of the obstructions and inpaint/repaint the image with a transformer. The results are claimed to be state-of-the-art.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer CuLC's valuable feedback. Our responses to the weaknesses and questions are listed below.\n\n**R-W1 Mask detector design:** This concern appears to stem from a misunderstanding. We have already clarified the role and details of the mask detector in Appendix A.1 of the submission. The mask detector is an off-the-shelf component and not central to our contribution. Our primary focus is a novel, distribution-agnostic framework for obstruction removal, which introduces (1) adaptive soft/hard recovery strategies for both opaque and semi-transparent obstructions, and (2) multi-modal prompt integration for precise obstacle representation. The framework is fully plug-and-play and can seamlessly incorporate more advanced mask detectors to further enhance performance without modification.\n\n**R-W2 Comparison with diffusion-based method:** Indeed, our submission demonstrates that diffusion-based models struggle with irregular and complex obstructions. In Table 4 and Figure 12 of Appendix C.3, we compared our method against **Repaint**, a representative diffusion-based inpainting approach, and validated the strong zero-shot generalization ability of our framework through both qualitative and quantitative results.\n\nAs requested, we further evaluated **DiffEdit** for unseen obstacle removal (PSNR↑/SSIM↑):\n\n\n| Method | Rain Streak | Snow | Stroke | Average |\n| - | - | - | - | - |\n| DiffEdit     | 23.88/0.6561         | 24.23/0.6732         | 11.65/0.6072         | 19.92/0.6455         |\n| Instruct2See | **29.82**/**0.8907** | **34.85**/**0.9283** | **29.45**/**0.9067** | **31.37**/**0.9086** |\n\nThe results show that DiffEdit performs poorly. This stems from its design as an image editing tool, which strictly constrains changes to masked regions and lacks the flexibility to handle irregular-shaped holes. Even when provided with detailed VLM-generated prompts, DiffEdit fails to achieve meaningful completions and often hallucinates entirely new objects. These limitations highlight the inadequacy of diffusion-based editing methods for this task and underscore the effectiveness of our proposed framework.\n\n**R-W3 Usage of input strategy like Instruct-Pix2Pix:** We conducted a direct comparison between our final strategy and the channel-wise concatenation approach (original + masked image), as in Instruct-Pix2Pix. The results are shown in the table below.\n\n\n| Method                 | PSNR↑    | SSIM↑     |\n| - | - | - |\n| Masked Image                  | **30.93** | **0.9250** |\n| Original + Masked Images | 30.19     | 0.9173     |\n\nIncorporating the raw image as input significantly degrades performance rather than improving it. This is because our training data includes three types of degradations, and direct access to the original image encourages the model to memorize specific degradation patterns, which harms zero-shot generalization.\n\nWhile we agree that conventional inpainting methods struggle with semi-transparent obstructions due to the occluded content being partially visible yet unknown, our method addresses this effectively through a soft-masking strategy. Moreover, original image features are already embedded implicitly via CLIP’s encoder, enabling semantic completion without overfitting to training-specific obstructions.\n\n**R-W4 Description of the restoration model:** The restoration model follows a standard encoder-decoder architecture, a widely adopted design in image processing tasks. As this component is not the core focus of our work (the key is the distribution-agnostic framework), we provided only a brief overview in the main text. However, to ensure methodological transparency and reproducibility, we will include a detailed description of the network architecture in the appendix.\n\n**R-W5 More evaluation metrics:** As suggested, we incorporated three additional evaluation metrics—**LPIPS↓**, **CLIP Score↑**, and **User Study (US)↑** (0–1)—to provide a more comprehensive assessment of obstacle removal performance across both **seen** (Fence, Flare, RainDrop) and **unseen** (Rain Streak, Snow, Stroke) scenarios. Due to space limitations, we reported results for three representative methods. Even under these expanded metrics, our method consistently outperforms baselines. Full comparisons across all metrics and methods will be included in the final submission.\n\n\n| Method       | Seen Obstruction               | Unseen Obstruction             |\n| - | - | - |\n| Restormer    | 0.0746/0.9421/0.74             | 0.1984/0.8833/0.80             |\n| Histoformer  | 0.0798/0.9365/0.84             | 0.1328/0.8885/0.70             |\n| XRestormer   | 0.0924/0.9360/0.68             | 0.1873/0.8820/0.60             |\n| Instruct2See | **0.0694**/**0.9442**/**0.92** | **0.1071**/**0.9140**/**0.84** |\n\n**R-W6 Too small image size:** Thank you for your suggestion. We will modify the experimental images to enhance the comparability between methods.", "claims_and_evidence": "- Extensive experiments in both qualitative and quantitative manners effectively support the claim of state-of-the-art results.", "methods_and_evaluation_criteria": "- The core of this method contains two parts: (1) a mask detector to predict the obstruction mask, and (2) an inpainting model (that leverages cross-attention, prompts, etc.) to inpaint or repaint the image. \n- However, it is completely unclear how this mask detector is designed, constructed, or trained. \n    - The word \"mask detector\" only appears 5 times in the whole paper, merely used as a black box. \n    - It is unclear whether the mask detector requires the user to input the type of obstructions or multi-modal clues or only with image input.\n    - Given that it is a crucial and integral model component, this becomes a substantial flaw in this paper's technical perspective. \n- For the inpainting part, the model itself is straightforward and well-motivated. However, it is also a very typical task for diffusion-based models with the out-of-the-box training-free inpainting method DiffEdit.\n    - DiffEdit strictly constraints the modifications within the unmasked part from its implementation, so there will be nothing unrelated changed outside of the mask. The strong capability of diffusion models also have the potential to outperform the proposed method's novel part.\n    - I think \"applying the predicted mask to Stable diffusion (XL or 3/3.5) + VLM-generated obstruction-free prompt + DiffEdit inpainting\" should be considered a naive baseline for comparison. \n    - One issue of the naive baseline is some bad support of semi-transparent obstructions as the semi-occluded covered part is unseen to the model. And a very typical way is to follow Instruct-Pix2Pix to channel-wise concatenate the original image in the input, while still inpaint the masked image. If time permits, I would also like to see this results.\n- The architecture of the proposed inpainting model is not very clear. It looks like both a VLM that can generate images and a ViT that do dense prediction. I would request more detailed explanation of this architecture, given no code is provided.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "- The experiment is extensive, and the results can well support the claims. \n- The metrics are mainly PSNR and SSIM. It would be better if there were also LPIPS metrics, along with VQAScore, GPTScore, or user studies.", "relation_to_broader_scientific_literature": "This paper proposes a novel method for inpainting in dealing with obstruction removal tasks.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "- All the figures are small sizes. Even though there is a zoomed-in area, the area itself is too small compared with the full image. Therefore, it is unable to verify whether other parts of the images contain artifacts.", "other_comments_or_suggestions": "Please refer to the reviews above. Specifically, \n- Please provide a detailed description of how the mask detector works.\n- Please add the naive diffusion inpainting baseline, at least the training-free one. Note that the prompt can be generated by a VLM, given the image and then removal of all descriptions of the obstruction.\n\nThis information is provided in the rebuttal. Therefore, I would like to raise the reviewing score from 2 to 3.", "questions_for_authors": "Please refer to the reviews above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OdOZkpMVsI", "reviewer": "Reviewer_17U3", "summary": "The paper provides a novel pipeline for “obstruction removal”: the combined task of identifying unwanted obstructions in an image and filling in the masked regions with plausible pixels. The method proposes a) using a hybrid adaptable masking strategy to identify the occluders and b) learning a text and vision conditioned model to make the inpainting process “context-aware”. These changes lead to competitive quantitative and qualitative results on seen occluders, and better performance with unseen occluders.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer 17U3's valuable feedback. Our responses to the weaknesses and questions are listed below.\n\n**R-W1 Lack of key details:** We emphasize that our method targets a fundamentally different problem than prior approaches such as Restormer and PromptIR. These existing restoration methods assume prior knowledge of obstruction characteristics and are designed to handle specific, predefined categories (e.g., rain, snow, flare). As a result, they lack generalization ability: a model trained for snow removal, for example, fails on rain without retraining, even when the distributions are similar.\n\nIn contrast, our work breaks this constraint through two key innovations: (1) a distribution-agnostic obstruction formulation, and (2) a flexible soft-mask recovery strategy integrated with multimodal obstruction representation. Together, these enable zero-shot generalization to diverse and unseen obstructions. This direction is both novel and underexplored, marking a significant advancement in the theory and practice of obstruction removal.\n\nWe will further strengthen the distinction between our method and existing work in the Introduction and Related Work sections to ensure clarity.\n\nAdditionally, we clarify that our original submission already includes: (1) a detailed rationale for dataset selection (see Section 3), and (2) complete details on prompt/text conditions, including their integration in Algorithm 1 (Appendix A) and illustrative examples in Figure 8 (Appendix B). As most of this information is provided in the appendix and may be easily overlooked, we will revise the main text to reorganize the structure or better guide readers to these supporting materials.\n\n**R-W2 Unreasonable distribution between main text and appendices:** Thank you for your suggestion. We will reorganize the content distribution of the main text and appendices.\n\n**R-W3 & Q2 Application scope and problem definition:** Thank you for your feedback. All obstruction categories, both seen and unseen, are sourced from publicly available datasets to ensure reproducibility.\n\nThe seen classes (Fences, Raindrops, Flares) were selected for their diversity in shape and structure, providing a strong foundation for generalizing to irregular obstructions. These include grid-like, point-based, and diffuse patterns, offering broad coverage of common obstruction forms.\n\nThe unseen classes (Rain Streaks, Snow, Stroke, Power Cables, Yarn, Scratches) were chosen to introduce appearance-level and geometric shifts from the training distribution. For example, snow and yarn differ in opacity and texture; rain streaks and power cables differ in orientation and continuity. This setup enables a meaningful evaluation of zero-shot generalization across varied obstruction types.\n\nWe will expand Section 3 to clarify these distinctions and include additional descriptions of seen/unseen differences.\n\nRegarding limitations: Indeed, the limitation of our method under intended usage has already been discussed in Section 6 of the paper. Our approach is specifically designed for small, spatially sparse obstructions and is not suitable for cases where large regions are occluded, as this exceeds the model’s implicit semantic completion capacity. To make this limitation more explicit, we will include additional visual examples and further elaborate on typical failure modes, even within the intended application scope.\n\n**R-Q1 Comparison with diffusion-based method:** Some important comparisons may have been overlooked. As shown in Figure 12 and Table 4 of Appendix C.3, our method demonstrates superior zero-shot generalization compared to inpainting baselines. This includes Repaint, a representative diffusion-based approach. In addition, in response to Reviewer CuLC’s Question W2, we extended our analysis to include DiffEdit, a diffusion-based image editing method. Together, these results provide comprehensive evidence of the advantages of our method over diffusion-based baselines.\n\n| Method       | Rain Streak                      | Snow                             | Stroke                           | Average                          |\n| ------------ | -------------------------------- | -------------------------------- | -------------------------------- | -------------------------------- |\n| LaMa         | 29.07/0.8858                     | 32.32/0.9108                     | 28.10/0.8728                     | 29.83/0.8898                     |\n| RePaint      | 28.78/0.8865                     | 32.20/0.9064                     | 23.78/0.8059                     | 28.25/0.8662                     |\n| DiffEdit     | 23.88/0.6561                     | 24.23/0.6732                     | 11.65/0.6072                     | 19.92/0.6455                     |\n| Instruct2See | **29.82**/**0.8907** | **34.85**/**0.9283** | **29.45**/**0.9067** | **31.37**/**0.9086** |", "claims_and_evidence": "In general, this paper backs up the claims it makes with reasonable evidence. The main claim of the paper is that their method performs on par, and sometimes worse or better than baselines on the task of obstruction removal. This seems to be well justified through standard PSNR / SSIM metrics. Evaluation is split between seen and unseen object classes, and many ways of visualizing the difference (eg. tables, qualitative visualization, comparison plot across baselines eg. Fig 5) are used to justify it. For different parts of the model (use of masking instead of end-to-end, cross attention module for conditioning, and adaptiveness in the masking), an ablation study is provided to show quantitative improvements.", "methods_and_evaluation_criteria": "At a high level, the method seems to be well motivated and clearly explained. However, key details which are present in the figures (eg. text / image encoders) are not included in writing. There were a lot more details provided in the appendix (a clear algorithm for inference, what actually goes into the text encoder), but this was not clear in the main paper. I believe the methods section needs to be augmented with more details about how corresponding text is generated for the datasets, and how a user will actually interact with the system using prompts. \n\nA quick glance at a close baseline (PromptIR) seems to suggest that there are many other datasets / benchmarks being used in this family of work. As I am not an expert in obstruction removal, it is unclear to me whether any of these datasets are transferable to the current work. I believe the work should augment it's evaluation criteria reading to better describe why particular datasets were chosen, and add a section in the supplementary to describe why datasets used in comparable works were left out.", "theoretical_claims": "This paper has no theoretical claims.", "experimental_designs_or_analyses": "Other than the question of how the dataset was chosen and organized, the paper follows standard experimental design and analysis in comparison to a range of baselines, showing many quantitative and qualitative results and using standard metrics in computer vision for image quality. I reviewed all experimental details, including both tables and all details. Assuming that correct datasets were chosen, experiments are sound and clear.", "relation_to_broader_scientific_literature": "In general, I find the work to be similar to other broad works in vision that use a generative model to solve under-constrained problems in a feed forward manner. The paper will benefit significantly from more description comparing the work in a more detailed manner to other naive approaches (simply using the easiest strategy of inpainting after masking with a foundation diffusion model for image generation) and more problem-specific approaches, while highlighting key differences at both levels. The paper has a very sparse description of related works and differentiation from baselines. While many relevant works seem to be cited, the description don’t clearly highlight the differences in the method in terms of technical contribution. Particularly, I believe revisions are needed to related work to highlight how the key components of the method differ from Restormer, PromptIR, or other close performing baselines. Further,  it is important to characterize the scope in which ease of these baselines work. To this end, many claims in the “Obstruction Removal” section of related work are very general. Lines 083 (“still face challenges”), 085 (“limiting their effectiveness in real world”), etc do not clearly state the differences in failure modes of the two methods. \n\nThe writing of the work would benefit from augmenting it to highlight clearly how the model choices enable specific improvements in unseen classes of obstructions. For example, one might expect that conditioning additionally on text should reduce Bayes error of the prediction problem, or using cross attention (instead of another naive conditioning strategy) might enable more precise conditioning on fine-grained features, etc. These will then specifically expand the cases in which the method works compared to baselines. Presently, the writing does not follow this kind of structure clearly motivating what the method enables.", "essential_references_not_discussed": "I am generally unfamiliar with research in this sub-field and would not know if key baselines were not listed here.", "other_strengths_and_weaknesses": "This likely involves a summary of the other sections of the review with a few additional points. \n\nStrengths\n\nFor the experiments the paper includes, they are sound and clear to the extent they are described. The paper is full of qualitative visualizations showing test cases in which the method works well, and all baselines are included in these visualizations.\n\nWeaknesses\n\nWriting is missing key details on related work, dataset choice and prompting / text conditioning details of method.\n \nA large part of some critical important parts of the method and results have been put in the appendix / supplementary despite many repeated results being in the main paper. This includes the description of how the CLIP text encoder is finetuned, some failure analysis and comparison to image inpainting methods. \n\nThe paper does an insufficient job of limiting the scope of what works with their method, and providing technical motivation for the precise unseen cases that the technical changes should help (or not help) on. For example, after reading the paper, it is clear to me that the method performs better on three sub-classes of unseen obstacles, and some miscellaneous novel obstacles by looking at the qualitative and quantitative results. However, it is not clear why how these unseen classes precisely differ from the seen classes, how large the distribution shift is, or how robust the method will be to other forms of perturbations. It is also clear from the appendix that one failure mode is providing the unintended input text, but there is no analysis presenting the failure mode of the method when it is used as intended. For example, why were the three particular unseen classes chosen to evaluate? And what are some out of distribution unseen classes.", "other_comments_or_suggestions": "As a result of 1) incomplete descriptions of related work, 2) lack of failure analysis/shortcomings, 3) lack of technical description precisely characterizing what kind of “unseen” objects we’d expect this method to work on, and 4) belief that the work would be a stronger fit for a vision rather than a machine learning conference due to lack of ML-direct writing, the present review is only that of weak acceptance. The review is especially borderline in belief.\n\nGiven lack of full understanding of benchmarks of this particular sub-field, I am keen to see the other reviews and will be open to change  depending on author / other reviewer’s clarifications.", "questions_for_authors": "1) Why is it that post-masking, an image diffusion model with one of many inpainting / imputation strategies cannot be applied for this task? Such models, since they’re trained at scale will naturally handle a wide range of objects - potentially even in a zero shot way? What prevents numbers from such an approach to be reported alongside other baselines?\n\n2) How was the dataset selection for this method done? What are the standard datasets in this field and why was a subset of BSD chosen for evaluation? Why can’t datasets used by prior work (eg. Prompt IR and Urban100) be also evaluated for this method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cw7rttbEax", "reviewer": "Reviewer_91FV", "summary": "In this paper, the author propose the Instruct2See, which is a zero-shot framework for removing both seen and unseen obstructions from images. It formulates obstruction removal as a soft-hard mask restoration problem, integrating multi-modal prompts via cross-attention. A tunable mask adapter refines masks for semi-transparent obstructions. The results demonstrate the outperforms state-of-the-art methods on PSNR and SSIM while generalizing well to unseen cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer 91FV's valuable feedback. Our responses to the weaknesses and questions are listed below.\n\n**R-W1 Domain restributions:** Our model is trained on natural image datasets, and as such, it performs well across a wide range of domains within the natural image space, such as urban scenes, landscapes, wildlife, and daily objects, without restriction. However, for domains outside this distribution, such as medical imaging, the model cannot be expected to produce meaningful restorations, as the training data does not include such content.\n\nHowever, our framework itself is domain-agnostic. By training on appropriately curated domain-specific data (e.g., medical images), the same architecture and methodology can be applied to obstruction removal in those domains. We will clarify this point in the revised manuscript to better reflect the scope and potential of our approach.\n\n**R-W2 Explanation of running efficiency:** Thank you for your question. The efficiency of our method is largely due to the encoder-decoder architecture of the restoration model. Although the model has a relatively high parameter count, this does not translate into high computational cost. This is because deeper modules operate on low-resolution feature maps, allowing for higher channel dimensions without significantly increasing FLOPs or runtime. This design decouples parameter size from computational load, enabling efficient resource use while maintaining strong performance.\n\n**R-W3 More details of experiment:** Thank you for the suggestion. We will revise the Experiment Settings section to include additional implementation and training details for improved clarity and reproducibility.\n\n**R-W4 Minor spelling error:** Thank you for pointing this out. We will carefully review the manuscript to correct any spelling errors.\n\n**R-Q1 Overlapping obstruction removal:** Thank you for your comment. We would like to clarify that our original submission includes experiments on images with multiple overlapping obstructions, with results presented in Appendix C.2 and Figure 11. These experiments demonstrate that our method can accurately identify and represent overlapping obstructions using multimodal prompts and masks, and that our model effectively removes them, further validating the robustness of our approach.", "claims_and_evidence": "The paper claims \"Remove Any Obstructions.\" However, I would like to know whether there are any domain restrictions or size limitations.\n\n\n\nOther claims I think are clear.", "methods_and_evaluation_criteria": "I think it makes sense.  \n\nHowever, I would like to understand the computational efficiency. Although Table 6 presents the results, I am still concerned about why the proposed model has the largest number of parameters, yet the FLOPs and runtime are not significantly higher.", "theoretical_claims": "The paper lacks formal theoretical proofs but provides a clear mathematical formulation.", "experimental_designs_or_analyses": "I think the results are promising, and the comparisons are comprehensive. However, I suggest that the author provide more implementation details.", "relation_to_broader_scientific_literature": "The work builds on vision-language models (CLIP). It advances the field by unifying multi-modal prompts for zero-shot generalization, addressing limitations of task-specific and all-in-one models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "I am concerned about how the model performs in specific domains, such as the medical field.", "other_comments_or_suggestions": "Typo: L802 As shown in...", "questions_for_authors": "Can the method handle overlapping obstructions (e.g., rain + snow) without interference?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sC8ahAgmOi", "reviewer": "Reviewer_4Rdn", "summary": "This paper introduces a zero-shot framework for image restoration that can handle a wide range of obstructions, including those not seen during training. Overall, the paper contributes a flexible, distribution-agnostic method for obstruction removal that harnesses multi-modal cues and dynamic masking to achieve robust performance across diverse and unpredictable obstacles.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer 4Rdn's valuable feedback. Our responses to the weaknesses and questions are listed below.\n\n**R-W1 Limited scope of experimentation:** Thank you for your comment. We believe some of our experimental results may have been inadvertently overlooked. In both the main paper and supplementary materials, we present extensive zero-shot evaluations across a wide range of obstructions, including **rain streaks, snow, strokes, power cables, spots, scratches, yarn, shadows, watermarks**, and **complex multi-obstruction scenarios**. These results collectively validate the effectiveness and generalization capability of our method.\n\nUnlike object removal, where datasets are abundant and well-defined, obstructions are often **irregular in shape, appearance, and distribution**, making comprehensive evaluation more challenging. To address this, we have curated and tested on **all publicly available obstruction data** that aligns with our problem definition. We will revise the manuscript to highlight these results more prominently. Additionally, we welcome suggestions on any other publicly available datasets we may have missed and are happy to include further evaluations as needed.\n\n**R-W2 Limited technical contribution:** Thank you for your feedback. We respectfully believe this concern may arise from an underappreciation of the conceptual novelty behind our approach. Our method introduces a new formulation of obstruction removal as a distribution-agnostic soft masking problem, which fundamentally unifies the treatment of both opaque and semi-transparent obstructions within a single framework.\n\nThis perspective shifts away from traditional category-specific pipelines and reframes obstruction removal as a context-aware reconstruction task, where the model learns to reason about partially visible content regardless of obstruction type. Prior methods typically rely on known obstruction categories or retraining per class, whereas our method enables zero-shot generalization to unseen obstruction types—an ability that existing approaches lack.\n\nMoreover, our flexible soft-mask recovery strategy, combined with multi-modal prompt integration, allows the model to adaptively handle obstructions with varying degrees of transparency, shape complexity, and semantic ambiguity. We also show that even with access to ground-truth masks, conventional methods fail to generalize across distributions, underscoring the need for our proposed formulation.\n\nThis unified, distribution-agnostic perspective and the demonstrated generalization across diverse and unseen scenarios represent a meaningful advancement in both the theoretical framing and practical capabilities of obstruction removal.\n\n**R-Q1 CLIP fine-tuning strategy:** In our framework, the primary role of the text encoder is not to align text with visual features in the conventional sense, but rather to help the model interpret user instructions, particularly in understanding the transparency attributes of obstructions described in the prompt. To achieve this, fine-tuning the text encoder is essential.\n\nFor example, before fine-tuning, the cosine similarity (after softmax) between the user instruction *\"There are raindrops in the image, please remove them\"* and the two core commands *\"remove opaque obstructions\"* and *\"remove transparent obstructions\"* were 0.5374 and 0.4626, respectively. After fine-tuning, these shifted to 0.00004 and 0.99996, respectively, indicating a dramatic improvement in semantic alignment. This refinement is critical for guiding the model’s recovery strategy based on the nature of the obstruction.\n\nAdditionally, Section 5.3 (Ablation Study) in the main paper provides experimental evidence for the importance of textual and visual conditioning. We further support this with a qualitative case study in Appendix D.2, illustrating the performance gains achieved under different conditioning strategies.", "claims_and_evidence": "The submission puts forward several key claims, and overall, many of these claims are supported by a comprehensive set of experiments and analyses.", "methods_and_evaluation_criteria": "The contrast experiment and evaluation criteria make sense.", "theoretical_claims": "I’ve checked the correctness of the proofs in the paper.", "experimental_designs_or_analyses": "I check the soundness/validity of any experimental designs or analyses.", "relation_to_broader_scientific_literature": "The key contributions and ideas include:\n\n• Adaptive Masking with a Tunable Adapter: Depending on whether an obstruction has clear (hard) or ambiguous (soft) boundaries, the model dynamically adjusts the mask. This adapter refines the initial mask estimates, enabling more accurate removal, particularly for semi-transparent obstructions like raindrops.\n\n• Zero-Shot Generalization: Extensive experiments show that Instruct2See not only performs well on in-distribution obstructions (like fences, raindrops, and flares) but also generalizes effectively to unseen types of obstructions (e.g., power cables, yarn, scratches).\n\n• Empirical Results: The paper provides comprehensive quantitative and visual comparisons with state-of-the-art methods. The proposed approach often demonstrates improved PSNR/SSIM scores and superior visual quality, especially in challenging, real-world scenarios where traditional models may fail.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n\n1. The paper proposes a unified obstruction removal method by formulating the task as a soft-hard mask restoration problem, effectively integrating visual and textual information, which demonstrates strong theoretical innovation.\n\n2.  It adopts a multi-modal prompting strategy along with an adaptive mask module, enhancing the model’s generalization ability in handling unseen obstructions, with extensive experiments validating its effectiveness.\n\nWeakness:\n\n1. The experiments on unseen obstructions are relatively limited in scope; further expanding the evaluation range may better demonstrate the robustness of its zero-shot learning capability.\n\n2. The technical contribution is somewhat limited; while the multi-modal prompting and mask recovery techniques are effective, they do not substantially deviate from established methodologies, indicating a reliance on existing concepts rather than offering groundbreaking innovations.", "other_comments_or_suggestions": "None.", "questions_for_authors": "In Supplementary Material B, we observe that the authors trained CLIP to better adapt to the dynamic soft masking approach. In fact, is it reasonable to train only the text module within the multimodal framework, and could not training the vision encoder affect consistency?  Furthermore, based on our concerns mentioned above, we would like to ask whether the use of a multimodal model is necessary, or if it is possible to use separate modules for visual and textual input.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GzskxXUfAu", "reviewer": "Reviewer_zzkC", "summary": "This paper tackles the problem of obstruction removal in images with transformer-based generative models. The key design lies in the modeling of obstructions and also the alignment between types of obstructions and language descriptions (e.g. \"rain drops\", \"fences\"). The learned model achieves comparably better results on several benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer zzKc's valuable feedback. Our responses to the weaknesses and questions are listed below.\n\n**R-W1 Minor performance improvements:** We would like to emphasize that the primary goal of our method is not to maximize performance for any specific obstruction type, but to achieve robust zero-shot generalization across a wide range of unseen obstructions. From this perspective, our approach shows a clear advantage over existing methods, including those that claim to be category-agnostic and those we adapted to operate in zero-shot settings.\n\nInstead of tailoring the model to known obstruction types, our distribution-agnostic formulation treats all obstructions using a unified approach. This applies regardless of their appearance, transparency level, or spatial structure. As a result, our method delivers consistent performance across both seen and unseen categories, something existing methods cannot achieve without retraining or manual tuning.\n\n**R-W2 Selection of more representative cases:** Thank you for your suggestion. We will replace more representative experimental samples for visualization.", "claims_and_evidence": "The authors claimed the following contributions:\n\n- **first unified obstruction formulation**: The authors did provide a formulation, but it is a general and broad formulation that previous works also shared.\n\n- **zero-shot paradigm for obstruction removal, incorporating multi-modal prompts**: The authors did include self-switching modules given both visual and textual prompts. The dynamic soft masking strategy seems new to me.\n\n- **comprehensive experiments, strong zero-shot capability**: The unseen experiments did suggest the model is somewhat superior to previous models.", "methods_and_evaluation_criteria": "The overall design of the model is intuitive, the modeling of soft and hard masking also seems reasonable for the obstruction types at hand. One minor concern is on the depth of exploration over the design of the text-aligned masking strategy as it seems currently it also relates to tasks like inpainting and editing.", "theoretical_claims": "The theoretical claims are intuitive and easy to follow.", "experimental_designs_or_analyses": "Though the authors achieve comparatively better results against existing methods, the overall improvement seems marginal.", "relation_to_broader_scientific_literature": "The designs could similarly be adapted to tasks like generation, editing, etc.", "essential_references_not_discussed": "There are no obvious missing references.", "other_strengths_and_weaknesses": "As mentioned earlier, the current soft masking strategy seems to go beyond obstruction removal alone, the authors might want to dive deeper in to this learning framework", "other_comments_or_suggestions": "I guess more representative images in the qualitative visualization could be beneficial for understanding the effectiveness of the proposed method.", "questions_for_authors": "See the Strength and weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers found the proposed method to be effective and the results promising. The rebuttal successfully addressed reviewer comments and all reviewers recommend acceptance. The authors are encouraged to strengthen the final version by incorporating reviewer suggestions, particularly by integrating additional results, clarifying technical contributions beyond prior work, and ensuring key content currently in the appendix is moved to the main paper with improved linkage between the two. 

Reviewer CuLC also raised two remaining concerns to be addressed in the revision: (1) while the mask detector is an off-the-shelf component and not a claimed contribution, the paper lacks analysis on how its selection affects performance; (2) although various types of obstructions were evaluated, they share similar characteristics, which may limit generalizability. These issues should be acknowledged and discussed.
Decision: Accept (poster)