SeeThruAnything: Learning to Remove Any Obstructions Across Distributions

Authors: Junhang Li, Yu Guo, Chuhua XIAN, Shengfeng He
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Obstruction Removal, Zero-shot, Prompts

Abstract:
Images are often obstructed by various obstacles due to capture limitations, hindering the observation of objects of interest. Most existing methods address occlusions from specific elements like fences or raindrops, but are constrained by the wide range of real-world obstructions, making comprehensive data collection impractical. To overcome these challenges, we propose SeeThruAnything, a novel zero-shot framework capable of handling both seen and unseen obstacles. The core idea of our approach is to unify obstruction removal by treating it as a soft-hard mask restoration problem, where any obstruction can be represented using multi-modal prompts, such as visual semantics and textual commands, processed through a cross-attention unit to enhance contextual understanding and improve mode control. Additionally, a tunable mask adapter allows for dynamic soft masking, enabling real-time adjustment of inaccurate masks. Extensive experiments on both in-distribution and out-of-distribution obstacles show that SeeThruAnything consistently achieves strong performance and generalization in obstruction removal, regardless of whether the obstacles were present during training.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729742632333

thread_type:
official_review

<thread_object>
creator: reviewer_tqqw

<content>
summary:
The paper introduces a zero-shot obstruction removal framework to handle both seen and unseen obstructions in images. The idea is to formulate obstruction removal as a soft-hard mask restoration task, leveraging multi-modal prompts to enhance generalization. The framework incorporates a tunable mask adapter that dynamically refines inaccurate masks during the restoration process. The authors show that their method achieves superior performance over state-of-the-art techniques across a wide range of both in-distribution and out-of-distribution obstructions, demonstrating its flexibility and robustness across diverse occlusions.

strengths:
1. The introduction of the soft-hard mask prediction task intuitively enhances the model’s generalization ability, making it more adaptable to various obstruction types.
2. The paper conducts thorough experiments to validate the effectiveness of the multiple components of the framework.

weaknesses:
1. While the generalization largely stems from the mask prediction process, the paper lacks a detailed analysis of the quality and generalizability of the predicted masks. Are there any quantitative metrics to evaluate mask quality on both seen and unseen objects?
2. There is no comparison with in-painting methods in experiments. It would be valuable to see a comparison with more recent diffusion-based in-painting methods. For example, [1,2].
3. The performance on seen categories is not consistently superior to prior works.


[1] Grechka, Asya, Guillaume Couairon, and Matthieu Cord. "GradPaint: Gradient-guided inpainting with diffusion models." Computer Vision and Image Understanding 240 (2024): 103928.

[2] Lugmayr, Andreas, et al. "Repaint: Inpainting using denoising diffusion probabilistic models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

questions:
1. In cases where an image contains multiple obstructions (e.g., raindrop and power cable), how does the model handle prompts to remove only one type of obstruction? Can it selectively remove the specified obstruction without affecting others?
2. In Sec 5.1, the patch sizes mentioned (128, 160, 192, 256) seem unreasonable large. Should this refer to the image resolution instead?
3. Is the model capable of handling obstructions that exhibit significant differences from seen obstructions, such as in the case of reflection elimination? Can the authors provide some visualizations with existing datasets or real-world photos?
4. How do the authors initialize the model? Are pre-trained weights beneficial?
5. What is the model size, and how does it perform in terms of inference speed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730393329774

thread_type:
official_review

<thread_object>
creator: reviewer_aewh

<content>
summary:
This paper proposes a new Obstruction Removal method SeeThruAnything to reconstruct a clear original image given a degraded image and the estimated occlusion mask as input. To deal with different obstructions with or without ambiguous boundaries, SeeThruAnything utilize a transformer-based tunable adapter to convert hard masking to soft masking and use different maskings for different obstructions during inference. To better recover the original clean image, this paper also utilizes CLIP to extract multi-modal information from corrupted images and text commands like "remove semi-transparent obstructions" as a condition for their network. A cross-attention is used to inject this multi-modal information into their model. The proposed method obtains competitive performance compared to SOTA on seen obstructions and SOTA performance on unseen obstructions.

strengths:
1. The proposed method is quite simple and well motivated, which has a potential to become a common baseline for future works in this field.
2. The proposed method obtains competitive performance on seen obstructions and SOTA performance compared to previous methods.

weaknesses:
1. The proposed tunable mask detector seems to be heavy. It would be best to mention the number of parameters and the flops for your method and the compared method so that we can distinguish the performance improvement brought by the increasing parameters.
2. The proposed method use the corrupted images with obstructions removed as input. The obstructions are removed according to inaccurate estimated obstruction masks. However, previous works mainly take degraded images with unremoved obstructions as input. There is no ablation study to prove the advantage of your design.
3. The images and texts in Figure 1 might be too small. It is difficult to distinguish the comparison in figure 1.

questions:
1. As the method is mainly tested on synthetic corrupted data, how does it perform for images with multi-type obstructions?
2. What mask detector is used? Is it the same one for compared methods?
3. What is the exact text command used in your method? Do you use different text commands for different obstructions? How does it perform when only using some consistent text commands like "remove obstructions".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629775476

thread_type:
official_review

<thread_object>
creator: reviewer_uqlu

<content>
summary:
The article presents SeeThruAnything, a novel zero-shot framework designed to effectively remove various types of obstructions in images. SeeThruAnything employs multi-modal prompts—combining visual and textual inputs—processed through a cross-attention unit for enhanced contextual understanding. It also features a tunable adapter for mask adjustments. Extensive experiments demonstrate that SeeThruAnything excels in both familiar and unfamiliar obstacle scenarios, showcasing strong performance and generalization capabilities in obstruction removal tasks.

strengths:
1. The paper is well-written and easy to follow.
2. The paper demonstrates that SeeThruAnything is highly effective in removing obstacles, particularly in generalizing to invisible obstacles outside the training distribution. 
3. The paper conceptualizes obstacle removal as a problem of soft and hard mask recovery, offering significant insights into the future research directions of this field. By integrating visual tokens with text tokens, the model’s capacity for generalization in open-world scenarios is substantially enhanced.

weaknesses:
1. The technical contribution of the paper is limited. The use of multi-modal prompts and mask recovery techniques, although effective, may not significantly depart from established methodologies, suggesting a reliance on existing concepts rather than groundbreaking innovations.
2. Generalization Limitation. While SeeThruAnything demonstrates the capability to remove unseen obstacles, these obstacles are often fundamentally similar in nature (e.g., raindrops and rain streak, fences and yarn). This is underscored by the observation that the performance of WGWSNet and PromptIR on rain streaks and strokes is nearly comparable to, or even surpasses, that of SeeThruAnything.

questions:
As you mentioned, the original configuration of other methods cannot achieve zero-design tasks. How do you give them this ability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679709425

thread_type:
official_review

<thread_object>
creator: reviewer_ms8c

<content>
summary:
The paper introduces an in-painting method to remove real-world obstructions from images. The method uses multimodal prompts from a pretrained CLIP model as conditioning to the in-painting transformer model and shows good improvement over prior art.

strengths:
The paper is well written, the method and architecture is clearly explained and tested on a wide range of benchmarks. The proposed method shows good improvement over recently published methods in the domain.

weaknesses:
1. The paper does not include any examples of textual prompts. The only examples are in Figure 1 and 3, CLIP text encoder is not explicitly trained on instructions like "Remove the semi-transparent obstruction", and the image-text datasets used to train CLIP models typically have captions describing the foreground which may or may not describe the type of occlusions. It is unclear how the embedding space of CLIP's text encoder is capable of embedding such instructions. 
2. The paper does not provide details on CLIP model used for generating multimodal prompts. 
3. In Table 4. ablation does not include "visual prompt only" setting. An interesting ablation would be to use different text embedding models apart from CLIP.

questions:
My main concern with this work is the use of CLIP text encoder, as it is typically not trained on textual instructions as depicted in Figure 1 and 3. is it possible that any text encoder would work in this setup? Also, ablation on textual prompts would be good to have, i.e. what level of detail is necessary in the prompt to achieve a good in-painting result. I am willing to update the score based on the response to above questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732547423492

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We sincerely appreciate your valuable feedback and the critical concerns raised. We believe some of these concerns stem from misunderstandings, which we have addressed thoroughly in the revised manuscript. Specifically, our distribution-agnostic obstruction formulation, which unifies obstruction removal tasks, is the key innovation enabling generalizability to both seen and unseen obstructions. This formulation is not only impactful for obstruction removal but also holds significant potential for broader applications in image recovery tasks. We kindly invite you to review the clarifications and improvements in our revised manuscript and hope you will reconsider your ratings in light of these contributions.

Thank you for your time and thoughtful consideration.

Sincerely,

The Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733118266324

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Reconsideration of Rating

comment:
Dear Reviewers,  

Thank you for your thoughtful comments and feedback. We have provided detailed responses to address all concerns, including clarifications and revisions. We would greatly appreciate it if you could reconsider your rating to reflect the improvements and explanations provided in our replies.  

Thank you for your time and consideration.  


Sincerely,\
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
All the reviewers provided the negative ratings. Although the paper has some merits, e.g., competitive results, the reviewers pointed out a few critical concerns about 1) technical contributions compared to the prior work, 2), technical clarity like used prompts and details of CLIP text encoder training, 3) generalization limitation and performance on seen cases. After taking a close look at the paper, rebuttal, and discussions, the AC agrees with reviewers' feedback and hence suggests the rejection decision. The authors are encouraged to improve the paper based on the feedback for the next venue.