MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation

Authors: Jiawen Wang, Yinda Chen, Xiaoyu Liu, Che Liu, Dong Liu, Zhiwei Xiong
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: domain adaptation, unsupervised learning, masked image modeling, semantic segmentation, complementary masking

Abstract:
Recent works have correlated Masked Image Modeling (MIM) with consistency regularization in unsupervised domain adaptation. However, they merely treat masking as a special form of deformation on the input images and neglect the theoretical analysis, which leads to a superficial understanding of masked reconstruction and insufficient exploitation of its potential in enhancing feature extraction and representation learning. In this paper, we reframe masked reconstruction as a sparse signal reconstruction problem and theoretically prove that the dual form of complementary masks possesses superior capabilities in extracting domain-agnostic image features. Based on this compelling insight, we propose MaskTwins, a simple yet effective learning strategy that integrates masked reconstruction directly into the main training pipeline. MaskTwins uncovers intrinsic structural patterns that persist across disparate domains by enforcing consistency between predictions of images masked in complementary ways, enabling domain generalization in an end-to-end manner. Extensive experiments verify the superiority of MaskTwins over baseline methods in natural and biological image segmentation. These results demonstrate the significant advantages of MaskTwins in extracting domain-invariant features without the need for separate pre-training, offering a new paradigm for domain-adaptive segmentation.

Decision: reject
Reviewer scores: 8, 3, 5

Reviews and discussion:
<thread>
creation date:
1730554332554

thread_type:
official_review

<thread_object>
creator: reviewer_9p4o

<content>
summary:
This paper addresses the image segmentation problem in the context of unsupervised domain adaptation (UDA). Instead of using the random masks, authors design a dual-form complementary masked images to enhance the generalization of the overall framework. Authors argue that robust, domain-invariant features by enforcing consistency learning upon masked images can be learned. Extensive experiments on six experiments demonstrate the effectiveness of the proposed method.

strengths:
-- The presentation of this paper is clear.

-- The idea of this paper is easy to understand.

-- The results look good on multiple datasets.

weaknesses:
-- It is difficult to justify the contribution of this paper. The overall framework is based on many well-structured techniques, such as AdaIN, EMA-based pseudo label, consistency learning loss, and so on. There lacks an effective ablation study to clarify the gain that is actually taken by the envisioned dual-form complementary masked method. In the ablation study section, only the patch size and the mask type and mask ratio are reported. 

-- The contribution is a bit incremental and over-claimed. At a high level, the envisioned dual-form complementary masked method can be regarded as the constrained entropy minimization like MEMO (Test Time Robustness via Adaptation and Augmentation) . Meanwhile, it seems that the proposed dual-form complementary masked method can also be applied to general classification problem, where we can mask the image in a same manner.  This can verify the authors' first contribution, "This perspective bridges the gap between masked image modeling and signal processing theory, potentially opening new avenues for future research". Applying the proposed method to more general tasks can further show the effectiveness of the proposed method.

-- In the introduction section, the authors clarify that "This insight is grounded in the principles of compressed sensing" and "We provide a theoretical foundation for masked reconstruction by reframing it as a sparse signal reconstruction issue". However, in the method section, there is no any presentation about these descriptions, which is very vague. Authors may reorganize the manuscript carefully.

questions:
Please the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680442690

thread_type:
official_review

<thread_object>
creator: reviewer_e8te

<content>
summary:
The paper proposes MaskTwins for unsupervised domain-adaptive image segmentation. The key method is dual-form complementary masking, where masked image modeling (MIM) is used to generate dual complementary views of images. This approach enables robust feature learning by enforcing prediction consistency across complementary masked images, allowing for adaptation to target domains without additional learnable parameters. MaskTwins demonstrates strong performance improvements over previous unsupervised domain adaptation (UDA) techniques, achieving state-of-the-art results on diverse datasets, including both natural and biological images.

strengths:
- This paper provides a solid theoretical basis for the use of dual-form complementary masks.  

- The method could be easily integrated into existing frameworks with minimal overhead.

weaknesses:
- This paper is similar to [1*]. Therefore, it seems that the complementary masking techniques are not novel.

(1) Both papers propose complementary masking techniques to promote robustness. 

(2) Both methods use consistency mechanisms to enforce feature learning. 

[1*] Shin U, Lee K, Kweon I S, et al. Complementary random masking for rgb-thermal semantic segmentation[C]//2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024: 11110-11117.

- The method relies on pseudo-labels generated by an exponential moving average (EMA) teacher model for unsupervised target domain training. This dependence on pseudo-label quality could introduce noise into training if the initial pseudo-labels are inaccurate.

- The ablation study of different components in the model should be provided.

questions:
See section weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706143778

thread_type:
official_review

<thread_object>
creator: reviewer_2ev4

<content>
summary:
The paper introduces a new methodology based on masked image modeling for unsupervised domain adaptation. The paper shows theoretically that the complementary masking strategy outperforms random masking on information preservation, generalization and feature consistency. The proposed methodology is evaluated on natural image segmentation, mitochondria semantic segmentation and synapse detection.

strengths:
The paper is overall very enjoyable to read. In particular, i find that:

* The paper is clearly written, well-situated in the literature and is easy to read and understand.
* The proposed methodology is original and constitutes and important bridge between SSL and UDA.
* The proposed methodology excels in its simplicity, in particular compared to adversarial UDA frameworks. In particular, i find it noteworthy that the proposed methodology is effective on 3D input (synapse detection), in which it is generally notoriously hard to stabilise UDA training.
* The evaluation is thorough and includes comparisons to a wide range of UDA baselines.
* The theoretical foundation is sufficient for the study’s objectives, but could be strengthened to enhance rigor.

weaknesses:
The main short comings of the paper seems to be the limited scope of the theoretical analysis and the ablation study.

* First the theoretical analysis does not connect the proposed methodology to the vast literature on the theory for unsupervised domain adaptation, in particular [1], [2], [3]. Instead the theoretical results relate the methodology to a naive masking strategy, which might still provide benefits over other methodologies. 

* Second, the ablation study does not study the effectiveness of individual parts of the methodology. I find that it is central to include a comparison to a simpler masking strategy, such as the random baseline from the theoretical analysis and to study the impact of each element in the loss. Instead the authors choose to only ablate details on the patch size, mask type and masking ratio.

References:

[1] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of repre- sentations for domain adaptation. In NIPS, pages 137–144, 2006.

[2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151–175, 2010.

[3] Zhang, Yuchen, et al. "Bridging theory and algorithm for domain adaptation." _International conference on machine learning_. PMLR, 2019.

questions:
Ablation study:

* How does the framework perform if the naive masking strategy is used compared to complementary masking? (I.e. no complementary loss). And more generally, what is the performance gain of each element in the loss? (i.e. complementary vs. consistency vs. supervised loss.)
* What is the benefit of using an EMA teacher-student framework and what is the effect of not including AdaIN?

Evaluation:

*  How many folds/runs are the evaluation results based on?
*  How stable is the training? (i.e. the variance of the runs.)


Overall i find that the paper is important and I am inclined to reconsider my score if above points are addressed in the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732632236398

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for their valuable comments and suggestions, and we hope our responses adequately address your concerns. The revised version of the manuscript has been uploaded. Furthermore, we are happy to provide further details on any aspects of our responses that may require additional clarification or elaboration.

Once again, we appreciate the reviewers’ time and insightful feedback and look forward to receiving further input.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733302116850

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers once again for their valuable time and feedback.

We have carefully incorporated each reviewer's constructive comments, making corresponding revisions and clarifying possible misunderstandings. We are encouraged to see that Reviewer 2eV4 have raised the score during discussion. Although not all reviewers have responded to our latest replies, we believe the major concerns should be adequately addressed. 

We hope our efforts will be taken into consideration in your final justification.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents MaskTwins, leveraging dual-form complementary masking for domain-adaptive image segmentation, supported by a theoretical framework connecting masked image modeling to sparse signal reconstruction. While the theoretical contributions are appreciated, the reviewers consistently raised concerns about the lack of empirical validation to substantiate these contributions beyond standard benchmark evaluations. The limited scope of ablation studies and marginal performance improvements over random masking strategies also raised questions regarding the claims. Although the authors provided detailed responses and additional experiments during the rebuttal period, they were not able to fully address these issues, leaving questions about the practical impact of the theoretical insights. Therefore, further revisions are recommended for this work.