MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Authors: Jiawen Wang
Yinda Chen
Xiaoyu Liu
Che Liu
Dong Liu
Jianqing Gao
Zhiwei Xiong
Venue: ICML 2025 poster (2025)
Keywords: domain adaptation
unsupervised learning
masked image modeling
semantic segmentation
complementary masking
Primary area: applications->computer_vision

Abstract: Recent works have correlated Masked Image Modeling (MIM) with consistency regularization in Unsupervised Domain Adaptation (UDA). However, they merely treat masking as a special form of deformation on the input images and neglect the theoretical analysis, which leads to a superficial understanding of masked reconstruction and insufficient exploitation of its potential in enhancing feature extraction and representation learning. In this paper, we reframe masked reconstruction as a sparse signal reconstruction problem and theoretically prove that the dual form of complementary masks possesses superior capabilities in extracting domain-agnostic image features. Based on this compelling insight, we propose MaskTwins, a simple yet effective UDA framework that integrates masked reconstruction directly into the main training pipeline. MaskTwins uncovers intrinsic structural patterns that persist across disparate domains by enforcing consistency between predictions of images masked in complementary ways, enabling domain generalization in an end-to-end manner. Extensive experiments verify the superiority of MaskTwins over baseline methods in natural and biological image segmentation.
These results demonstrate the significant advantages of MaskTwins in extracting domain-invariant features without the need for separate pre-training, offering a new paradigm for domain-adaptive segmentation. The source code is available at https://github.com/jwwang0421/masktwins.

MaskTwins:
Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Jiawen Wang 12
Yinda Chen
12*
Xiaoyu Liu
Che Liu
Dong Liu
Jianqing Gao Zhiwei Xiong' 12
Abstract
1. Introduction
Inspired by Masked Language Modeling (MLM) (Devlin, 2018; Brown, 2020) in natural language processing, Masked Image Modeling (MIM) (Bao et al., 2022; He al., 2022; Xie et al., 2022b) has achieved remarkable success in self- supervised visual representation learning, MIM learns se- mantic representations by deliberately obscuring parts of the input and then reconstructing the missing information based on the unmasked parts, e.g., normalized pixels (He et al., 2022; Xie et al., 2022b), HOG feature (Wei al., 2022), discrete tokens (Bao et al., 2022; Dong et al., 2023), deep features (Zhou et al., 2021; Dong et al., 2022) or fre- quencies (Xie et al., 2022a; Liu et al., 2023). The success of these methods can be attributed to their ability to force models learn robust and generalizable features, even in the face of incomplete or corrupted input data. By simu- lating real-world scenarios where visual information may be partially occluded or distorted, masked reconstruction techniques enable models to develop more comprehensive understanding of visual concepts.
Recent works have correlated Masked Image Modeling (MIM) with consistency regulariza- tion in Unsupervised Domain Adaptation (UDA). However, they merely treat masking as spe- cial form of deformation on the input images and neglect the theoretical analysis, which leads to a superficial understanding of masked recon- struction and insufficient exploitation of its po- tential in enhancing feature extraction and rep- resentation learning. In this paper, we reframe masked reconstruction as a sparse signal recon- struction problem and theoretically prove that the dual form of complementary masks possesses su- perior capabilities in extracting domain-agnostic image features. Based on this compelling in- sight, we propose MaskTwins, a simple yet ef- fective UDA framework that integrates masked reconstruction directly into the main training pipeline. MaskTwins uncovers intrinsic struc- tural patterns that persist across disparate do- mains by enforcing consistency between predic- tions of images masked in complementary ways, enabling domain generalization in an end-to-end manner. Extensive experiments verify the supe- riority of MaskTwins over baseline methods in natural and biological image segmentation. These results demonstrate the significant advantages of MaskTwins in extracting domain-invarian fea- tures without the need for separate pre-training, offering new paradigm for domain-adaptes segmentation. The source code is available at //mss//ii/./ssssss....mm
Analogously, consistency regularization in unsupervised domain adaptive segmentation learns domain-invariant fea- tures by enforcing consistency between the predictions of transformed images and their original counterparts. In Un- supervised Domain Adaptation (UDA), consistency regu- larization based methods (Choi et al., 2019; Araslanov & Roth, 2021; Melas-Kyriazi & Manrai, 2021) typically uti- lize a variety of augmentations, like affine transformations, color jittering and cutout (DeVries, 2017). Recently, MIC (Hoyer al., 2023) uses masked image consistency to learn context relations. Meanwhile, Shin et al. (2024) and Yang et al. (2025) have preliminarily explored the complementary masking in multi-modal tasks. However, these methods rely on specific multi-moda, datasets and neglect the potential of complementary masking in a single modality. Moreover, the lacked theoretical analysis results in a cursory under- standing of masked reconstruction and a failure to fully harness complementary contexts for feature extraction and representation learning.
Theory contribution Uniiversity of Science and Technology of China 'Institute of Artificial Intelligence. Hefei Comprehensive National Science Center DDta Science Institute, Imperial College London "IFLYTEK CO., LTD. Correspondence Zhiwei Xiong <zwxiong @ ustc.eda edu.en>>, Jianqing Gao
In this paper, we propose novel perspective on masked reconstruction by reframing it as a sparse signal reconstruc- tion problem and utilize it to design an effective strategy
Proceedings of the 4 International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
for domain-adaptive segmentation. Our theoretical analysis reveals that the dual form of complementary masks pos- sesses superior image feature extraction capabilities. This insight is grounded in the principles of compressed sens- ing (Donoho, 2006), suggesting that complementary masks can provide a more comprehensive sampling of the input space, Building upon this theoretical foundation, we in- troduce MaskTwins, a simple yet effective framework for domain-adaptive segmentation. MaskTwins leverages the consistency constraints of complementary masks to extract domain-invariant features. This approach not only advances the theoretical understanding of masked reconstruction but also provides a practical framework for improving perfor- mance on domain- -adative vision tasks.
moment alignment (Chen et al., 2019; Liu et al., 2020b), ad- versarial learning (Tsai et al., 2018; Luo et al., 2021; Zheng & Yang. 2022) and self-training (Zou et al., 2018; Mei et al., 2020; Zhao et al.. 2023). Methods based on statistical mo- ment a alignment aim to minimize the domain discrepancy employing an appropriate statistical distance function such as entropy minimization (Chen et al., 2019) and Wasserstein distance (Liu et al., 2020b). Adversarial training methods achieve domain invariant feature extraction with GAN framework (Goodfellow et al., 2014). To overcome the chal- lenges of instability in adversarial learning, Zheng & Yang (2022) adaptively refine the distribution of training data by aggregating the weak models. In self-training, pseudo la- bels (Lee et al., 2013) are created for the unlabeled target domain using confidence thresholds (Zou et al.. 2018; 2019; Mei al., 2020), pseudo-label prototypes (Zhang al., 2019a; 2021; Jiang et al., 2022) uncertainty (Zheng & Yang, 2021). Recently, Hoyer et al. (2023) explore context relations while Zhao et al. (2023) learn pixel-wise repre- sentations to boost the quality of pseudo-labels, Different from these UDA methods, our proposed MaskIwins inte- grates the context relationships by enforcing complementary masked consistency without introducing extra learnable pa- rameters. The dual-form masked image consistency enables the learning of complementary clues, which further boosts the extraction of domain invariant features.
Our contributions can be summarized as follows:
1. We provide a theoretical foundation for masked re- construction by reframing it as a sparse signal recon- struction problem, offering new insights into the effec- tiveness of complementary masks. This perspective bridges the gap between masked image modeling and signal processing theory, potentially opening new enues for future research.
2.
We propose MaskTwins, UDA framework that forces consistency between predictions of dual-form complementary masked images without introducing extra learnable parameters. Furthermore, we study the complementary masking strategy and first prove its capability in extracting domain-invariant features. Our theoretical analysis provides conceptual guidance for the broader application of complementary masking.
Masked Image modeling (MIM) methods are showing great promise in visual self-supervised representation learn- ing for their ability to learn robust and generalizable fea- tures from incomplete cornupted input data, enhancing the models' comprehension of visual concepts. Many target signals have been conceived for the masked reconstruc- tion, encompassing raw pixels (He et al., 2022; Xie et al., 2022b), HOG features (Wei et al., 2022), discrete visual tokens (Bao et al., 2022: Dong et al., 2023), frequencies (Xie al., 2022a; Liu et al., 2023) and deep features (Zhou et al., 2021; Dong et al., 2022). Wang et al. (2023) fur- ther explore the reconstruction process multiple scales while Kong & Zhang (2023) interprete MIM in unified framework. However, these works mainly treat masked re- construction as a pre-training strategy but neglect its poten- tial for downstream tasks related to domain generalization. Hoyer et al. (2023) preliminarily explore the masked tar- get image in the UDA setting and conclude that masked image consistency substantially boosts UDA performance through additional context clues. Recently, complementary masking has been superficially utilized with cross-modal thermal imaging (Shin et al., 2024) and depth estimation (Yang et al., 2025). Yet, a thorough theoretical foundation for the effectiveness of masked images in domain adapta- tion remains to be established. In this work, we introduce a novel reconceptualization of the masked reconstruction as a sparse signal reconstruction problem and refine the theory
3. We demonstrate the superiority of our approach through extensive experiments, showing significant improvements over baseline methods in both natural and biological image segmentation. Our results indi- cate that MaskTwins can enhance model robustness and adaptability across diverse domains.
2. Related Works
Unsupervised domain adaptation (UDA) addresses the critical problem of performance degradation in target do- mains through the effective exploitation of both labeled source domain data and unlabeled target domain data. By bridging the domain gaps, UDA has emerged as a versatile solution to enhance model robustness in various computa- tional domains, demonstrating promising results various computer vision tasks such as natural image semantic seg- mentation (Tsai et al., 2018; Mei et al., 2020; Jiang et al., 2022) and medical image segmentation (Bermúdez- Cermidezz-Chcón et al., 2018; Liu et al., 2020a; Wu et al., 2021). UDA solu- tions are broadly categorized into three groups: statistical
2

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
of complementary masks. By surpassing the constraints of domain-specific customization, MaskTwins employs strategic complementary masking technique the input data, ensuring more holistic and nuanced understanding of the intrinsical data patterns.
Theorem 2 (Generalization Bound), Assume & is L- Lipschitz and f is 8-smooth. For any ō (0,1), 1), with probability at least 1- 8:
(C()) (f)...L))))) S CLBBA/VE,
(6)
[C(f) (.)()) S CBLBB (A +HWC) /VT.
3. Theoretical Analysis
Consistency regularization typically leverages rich set of augmentations. Nonetheless, focusing excessively on se- lecting the most appropriate parameters and perturbation functions makes them depart from the simple principle of consistency. Inspired by MIC (Hoyer et al., 2023), we ex- pect the performance of masked consistency in UDA. We further take insights from the paradigm of masked recon- struction (Bao et al., 2022; He et al., 2022; Xie et al., 2022b) and present theoretical analysis addressing the proper- ties of masked training in visual tasks to provide formal foundation for the complementary masking strategy. This analysis focuses on information preservation, generaliza- tion bounds, and feature consistency. Detailed proofs of all results are provided in Appendix A.
where A = 2 (2g B supxex [ and C₁ C2 are constants.
Theorem 3 (Feature Consistency). Define the feature con- sistency error as FCE(X1,,X) = [l - f(X2)l2. Then any 8 (0, 1), with probability at least 8:
FCE(X XI-D) S Cio Ciov klogk//H///)
(8)
FCE(X Xna) S C2 (a C2
(oVklog
+ #/BB1///HOo////////////
Definition 1 (Complementary Mask). Let DE E 10,1)"x be a binary matrix, where each element Dij Bernoulli(0.5). The complementary mask pair is defined as
where C₁, C2 are constants.
Remark 1. The theoretical results demonstrate the advan- tages of complementary masking. Specifically, complemen- tary masks offer better information preservation, tighter generalization bounds, and improved feature consistency, compared random masking. These properties are critical for extracting comainn-nvariian features, which are essential in cross cross-domain tasks such as domain adaptation.
(D,1 - D), where 1 is the all-ones matrix of size H x W. Definition 2 (Random Mask). Let R E {0, HXW be a binary matrix where each element R.j Bernoulli(0.5) in- dependently. The random mask pair is defined as (Ri, R2), where R1 and R2 are independent random masks.
Assumption (Visual Data Model). The input image E R. XC is generated by the model X ++E+N N, where S reprexents a sparse signal component, F. represents environmental factors, and N o20 is additive Gaussian noise.
4. Method
4.1. Overview
Assumption 2 (Feature Extraction Framework). We con- sider a feature extraction framewerrk with the objective func- tion:
The MaskTwins framework for UDA in semantic seg- mentation is detailed in Figure 1. The objective is to train a neural network fo that effectively generalizes to the target domain, given a labeled source domain dataset x = {Ix; : v C D and an unlabeled target do- main dataset X { C D', The framework oper- ates by generating two complementary masked versions of each target image z²"" denoted as D 2x and (1 - D) @ aTf. where D is a binary mask. A teacher model fo. updated via the Exponential Moving Average (EMA) of the student parameters, generates pseudo-labels for the target domain. (3) The student's predictions, together with the pseudo-labels from the teacher model, are used to compute the target- domain losses, while a supervised loss is computed using the labeled source data. This iterative process adapts the model to the target domain, leveraging both the supervised source information and the unsupervised adaptation to the target domain.
C(f) = Ex(((((x)),
(1)
where f RXHXXC + R2t is the feature extraction func- tion, and e: R² x R + R. is the loss function.
Theorem (Information Preservation). For any input image X. define the information preservation metric 12) = ((X)) Then:
XIDD) xD
(2)
(XR,, XRm) = (R1 O X, R2 2 X)
(UIP(X,.XI-D)) 2 :
(4)
(or(IP(X,,xI-D)) S Var(IP(Xnn,) (X.X.N.)))) Xn.)),
(5)
where: 2 denotes element-wise multiplication, (2) and (3)
represent complementary masking images and random masking images respectively.
3

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Source Image X
Pred. ps
Supervised Loss Laup
Complemenlary Masked Loss L
Encoder
Decoder
Stop Gradien.
Consistancy Learning Loss Lett
Network fv
MaskTwins Strategy
Masked
AdaIN
Weight update
Target Image X
Pscudo Lahel
Complementary Mirked
Ensedder
Devoder
Encudee
Decoder
Network fe
JMA teacher fo
Encoder
Decader
Putch-wise binury music
Network
(Xg, m)
(1% pf
Figure 1: The overall framework of MaskTwins. Given the labeled source data, caleulate the segmentation prediction ps with the network fe, supervised by basic segmentation loss L For the target domain, we obtain the predictions of complementary masked target images, constrained by the pseudo-labels Py that are generated based on the unmasked image by an exponential moving average (EMA) teacher fa- Furthermore, MaskTwins proposes the complementary masked loss between dual-form complementary masked images for deep consistency learning.
4.2. Complementary Masked Learning
where 17 are the pseudo-labels, Lor(::) refers to the stan- dard cross entropy loss, defaults to 0.5 to ensure balanced learning from the complementary masks. Since there is no ground truth available for the target domain, teacher model fo predicts the pseudo-label for the unmasked target image:
Building upon the theoretical analysis, we now describe the core complementary masked learning approach in Mask- Twins. This strategy employs patch-wise binary masks to generate dual complementary views of the target images. Specifically, for each target image 2T._ a binary mask D is sampled from a Bernoulli distribution:
[c argmax fo (z(z)) (a])),
(14)
where è is one category and the pseudo-label is converted into a one-hot categorical form via the Iverson bracket [-]-
mbb+1:(m++166 ~ Bernoulli - r), nb+1:(n++)6
(10)
The parameters the teacher network f are updated using Exponential Moving Average (EMA) of the parameters of the student network fo (Tarvainen & Valpola, 2017);
where r is the mask ratio, is the patch size, and m and n are patch indices. The dual-form complementary masked images are then obtained by dlement-wise multiplication: XT
X (T (1-D)oX"), (11)
Φc+1 - aΦi + (1-a)),.
(15)
where / denotes training step and is the EMA decay rate. The teacher model averages the weights of previous student models over time, leading to temporally stable and reliable predictions on the target domain.
These complementary views encourage the model extract robust, domain-invatie features by enforcing consistency learning upon masked images. To effectively learn from dual-form complementary contexts, we introduce two kinds of consistency losses. First, we constrain the consistent pre- diction of complementary masked images, which enables the network to integrate the dual-form clues. The comple- mentary masked loss is accordingly defined as:
This complementary masking strategy ensures that the model learns from diverse, yet consistent, views of the target domain, promoting robust generalization to the target do- main. The next section details the overall model architecture and training process, which integrates these complementary masking principles.
d Ellpi, P.
(12)
where PJ.D and TT mentary masked images. Intended to encourage successful masked reconstruction for both masked views, we also de- fine a masked consistency learning loss:
D-D1-D the predictions for the comple-
4.3. Model Architecture and Training Strategy
The MaskTwins architecture consists of a shared encoder and segmentation head for both the source and target do- mains. To mitigate domain shift, we employ an Adaptive Instance Normalization (AdaIN) (Huang & Belongie, 2017)
ch = E[Ax Lce Cee 07) + (1 - A) x Lce (D. D.0 1))) ).
(13)

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
module in the shallow layers of the network, which aligns feature distributions between the two domains.
loss weight Aem 0.01, pseudo-label threshold 6 = 0.7. and random augmentation including flip, transpose, rotate, resize and elastic transformation. For synapse detection, the point annotations (3D coordinates) are transformed into voxel cubes with a size of x 3 x 3 to be used as the training target. We use a patch size b = 6, a loss weight Aam = 0.1 Empirically, we set the threshold 6pre = 0.75 for the pre- synapse, ōyost = 0.65 for the post-synapse by default. The experiments conducted on 8x RTX 3090 GPU.
During training, we apply the complementary masks to the target domain images and enforce consistency between the predictions of these masked versions. This encourages the model to learn invariant representations that generalize well to the target domain. Our training strategy integrates supervised learning on the source domain with self self-training and consistency regularization on the target domain.
The supervised loss on the source domain is defined as:
5.2. Natural Image Semantic Segmentation
Ceup El (v) y(]] EE| v, log(r()),
(16)
First, we compare MaskTwins with previous UDA meth- ods on SYNTHIA- Cityscapps in Table 1. It can be seen that MaskTwins outperforms the previously state-of-the-art method by a significant margin of +2.7 mloU and remains competitive in segmenting almost all classes, which verifies the effectiveness of the dual form of complementary masks on target images. Classes that most profit from our method are sidewalk, road, vegetation, bus, and rider. Particularly, sidewalk owns the lowest UDA performance over 13 cat- egories, meaning that it is the most difficult to adapt for previous methods. Here, contextual relationships seem to be crucial for achieving successful adaptation. However, we increase the IoU of the sidewalk by +19.6 from 50.5 to 70.1 IoU. Additionally, our performance improvement on road +4.8 from 91. 2 to 96.0 IoU, probably because of its strong correlation with sidewalk. For some classes, our method increases the performance by smaller margin causes minor reduction, probably because the small objectives lead MaskTwins to misunderstand the complementary masked regions. In Figure 2, we visualize the segmentation results and the comparison with previous strong methods HRDA (Hoyer et al., 2022b), MIC (Hoyer et al., 2023) and the ground truth. While previous methods are confused by illu- mination as well crossings and fail to distinguish sidewalk from road, MaskTwins enables more robust recognition of these categories. We can conclude that the complemen- tary masking significantly enhances semantic segmentation, particularly for large or complex objects, where it effec- tively preserves structure and enables accurate segmentation despite obstacles, enhancing visual comprehension.
where v. Jo [=(); ) is the source prediction of the network fe and y is the corresponding ground truth.
By integrating these components- complementary masking, consistency regularization, and self-training with a teacher model- MaskTwins effectively leverages the complemen- tary information from masked inputs, promoting robust fea- ture learning and improved generalization.
The overall loss function that encapsulates our training strat- egy is formulated as:
Liotal LS + CTu ++Aemhe F
(17)
where Suup is the supervised loss on the source domain, Cr is the masked consistency learning loss on the target domain, CT is the complementary masked loss, and Aem is the weight for the complementary masked loss. We summarize the pipeline of MaskTwins in Algorithm I in Appendix E.
5. Experiments
5.1. Implementation details
Datasets. To demonstrate the versatility of MaskTwins, we conduct experiments spanning six distinct datasets: SYN- THIA (Ros et al., 2016) and Cityscapes (Cordts et al., 2016) are natural datasets, VNC III (Gerhard al., 2013), Lucchi (Lucchi et al., 2013), MitoEM (Wei et al., 2020) and WASP- SYN (Li et al., 2024) are biological datasets. The details of the datasets and the task-specific implementation on these datasets can be found in Appendix F.
5.3. Mitochondria Semantic Segmentation
MaskTwins parameters. MaskTwins uses the square mask for 2D domain adaptation and the cube mask for 3D respectively. The complementary masks have equal loss weight and the same mask ratio r = 0.5. The mask patch size is fixed for each task, approximately 1/16 of the in- put size. For
We conduct quantitative comparison results of our approach with multiple UDA baselines on the Lucchi and MitoEM datasets to demonstrate the superiority of our approach. As listed in Table 2, MaskTwins achieves the new state- of-the-ar art results in all cases, which corroborates the ef- fectiveness of the proposed complementary masking strat- egy. Specifically, MaskTwwns enhances the IoU of VNC IlI=Lucchi(Subsell and Lucchi(Subse(2) to 75.0% and 78.6%, outperforming the of art methods by 3.2% and 3.2%. On the MitoEM dataset with larger structure
-iiysspps we use patch size b = 64, a loss weight Apm = 0.01, and common color aug- mentation following the parameters of DAFormer (Hoyer et al., 2022a) and HRDA (Hoyer et al., 2022b). For mito- chondria semantic segmentation, we use a patch size b 32,
5

---

MwskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Table 1: Comparison results with previous UDA methods on SYNTHIA-~Citycappe traffic light, "TS" for traffic sign, "Veg." for vegetation, "PR" for person. We present per-class IoU and mean IoU (mloU), averaged across 13 categories. The competitors include DAFormer (Hoyer al., 2022a), CAMix (Zhou al., 2022b), HRDA (Hoyer al., 2022b), MIC (Hoyer al., 2023), etc. More details are shown in Appendix D.
"SW" stands for sidewalk, "TL" " for
Method
Road
SW
Build
TL
TS
Veg.
Sky
PR
Rider
Car
Bus
Motor
Bike
mloU
SIBAN
82.5
24.0
79.4
16.5
12.7
79.2
82.8
58.3
18.0
79.3
25.3
17.6
25.9
46.3
DADA
89.2
44.8
81.4
8.6
11.1
81.8
84.0
54.7
19.3
79.7
40.7
14.0
38.8
49.8
BDL
86.0
46.7
80.3
14.1
11.6
79.2
81.3
54.1
27.9
73.7
42.2
25.7
45.3
51.4
APODA
86.4
41.3
79.3
22.6
17.3
80.3
81.6
56.9
21.0
84.1
49.1
24.6
45.7
53.1
FDA
79.3
35.0
73.2
19.9
24.0
61.7
82.6
61.4
31.1
83.9
40.8
38.4
51.1
52.5
CCM
79.6
36.4
80.6
22.4
14.9
81.8
77.4
56.8
25.9
80.7
45.3
29.9
52.0
52.9
LDR
85.1
44.5
81.0
16.4
15.2
80.1
84.8
59.4
31.9
73.2
41.0
32.6
44.7
53.1
CD-SAM
82.5
42.2
81.3
18.3
15.9
80.6
83.5
61.4
33.2
72.9
39.3
26.6
43.9
52.4
ASA
91.2
48.5
80.4
5.5
5.2
79.5
83.6
56.4
21.9
80.3
36.2
20.0
32.9
49.3
DAST
87.1
44.5
82.3
13.9
13.1
81.6
86.0
60.3
25.1
83.1
40.1
24.4
40.5
52.5
UncerDA
79.4
34.6
83.5
32.1
26.9
78.8
79.6
66.6
30.3
86.1
36.6
19.5
56.9
54.6
RPLR
81.5
36.7
78.6
20.7
23.6
79.1
83.4
57.6
30.4
78.5
38.3
24.7
48.4
52.4
UACR
85.5
42.5
83.0
20.9
25.5
82.5
88.0
63.2
31.8
86.5
41.2
25.9
50.7
55.9
DACS
80.6
25.1
81.9
22.7
24.0
83.7
90.8
67.6
38.3
82.9
38.9
28.5
47.6
54.8
ProDA
87.8
45.7
84.6
54.6
37.0
88.1
84.4
74.2
24.3
88.2
51.1
40.5
45.6
62.0
DAFormer
84.5
40.7
88.4
55.0
54.6
86.0
89.8
73.2
48.2
87.2
53.2
53.9
61.7
67.4
CAMiX
87.4
47.5
88.8
55.2
55.4
87.0
91.7
72.0
49.3
86.9
57.0
57.5
63.6
69.2
HRDA
85.2
47.7
88.8
65.7
60.9
85.3
92.9
79.4
52.8
89.0
64.7
63.9
64.9
72.4
MIC
86.6
50.5
89.3
66.7
63.4
87.1
94.6
81.0
58.9
90.1
61.9
67.1
64.3
74.0
Ours
96.0
70.1
89.5
66.8
62.1
89.1
94.3
81.5
59.7
90.5
66.6
67.7
63.6
76.7
Target Image
HRDA
MIC
Ours
Ground Truth
Figure 2: Qualitative segmentation results on SYNTHIA-~Citssana MaskTwins significantly improves the segmentation
results of classes such as sidewalk, road, bus and rider.
discrepancy, our method consistently has remarkable im- provements by +2.1% IoU and +1.3% IoU respectively, It is noticeable that the mitochondria in MitoEM-H exhibit denser and more intricate distributions compared to those in MitoHM-R, rendering the domain adaptation from MitoEM- to MitoEM-H more challenging than the reverse. Despite this, MaskTwins surpasses CAFA (Yin et al., 2023) by sig- nificant margin on the benchmark of MitoEM-R M-iM---MM-ttMM---- H. It demonstrates that the proposed strategy can strengthen the generalization capacity of the learned model and adapt it to the challenging and diverse target domain. In Figure 3, we
further qualitatively compare MaskTwins with other petitive methods including DAMT-Net (Peng et al., 2020), DA-VSN (Guan et al., 2021), DA-ISC (Huang et al., 2022b), and CAFA (Yin et al., 2023). The results highlighted by yellow boxes reveal that MaskTwims shows better adaptabil- ity while other methods fail to handle hard cases with large domain gap. By leveraging the complementary masked context, our method manages separate mitochondria cor- rectly from the background and delivers more fine- ine-grained results the target domain. This indicates that Mask is adept at extracting robust features of segmented objec-
6

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Table 2: Quantitative comparisons on VNC II--uucchi-Subsell (V2L1), VNC Ill--Lucchi-Sussee (V2L2), MitoEM- R-MMitEM-- (R2H) and HonoeMM---MittENe (H2R), metrics in %. "Oracle" denotes the model is trained on target with groundtruth labels, while "NoAdapt" represents the model pretrained on source is directly applied target for inference without any adaptation strategy. The results of Oracle, NoAdapt, UALR, DAMT-Net, DA- VSV and DA-ISC are adopted from Huang et al. (2022b).
Methods
V2L1
V2L2
R2H
H2R
IoU
F1
MCC
mAP|
IoU
F1
MCC mAP/
IoU
F1
MCC
mAP
loU
F1
MCC
mAP
Oracle
86.5
92.7
86.5
88.6
93.9
84.5
91.6
91.2
97.0
87.3
93.2
92.9
98.2
NoAdapt
40.3
57.3
40.3
44.3
61.3
39.6
56.8
59.2
74.6
61.9
76.5
76.8
88.5
Advent
59.7
74.8
73.3
78.9
70.7
82.8
81.8
90.5
69.6
82.0
81.3
89.7
74.6
85.4
84.8
93.5
UALR
57.0
72.5
71.2
80.2
65.2
78.8
77.7
87.2
72.2
83.8
83.2
90.7
75.9
86.3
85.5
92.6
DAMT-Net
60.0
74.7
60.0
68.7
81.3
73.0
84.4
83.7
92.1
75.4
86.0
85.7
94.8
DA-VSN
60.3
75.2
73.9
82.8
71.1
83.1
82.2
91.3
71.4
83.3
82.6
91.6
76.5
86.7
86.3
94.5
DA-ISC
68.7
81.3
80.5
89.5
74.3
85.2
84.5
92.4
74.8
85.6
84.9
92.6
79.4
88.5
88.3
96.8
CAFA
71.8
83.4
82.8
91.1
75,4
85.8
85,4
94.8
76.3
86.6
86.0
92.8
80.6
89.2
88.9
96.8
Ours
75.0
85.6
85.1
92.4
78.6
87.9
87.4
95.2
78.4
87.9
87.4
94.0
81.9
90.0
89.7
96.9
Image
DAMT-Net
DA-VSN
DA-ISC
CAFA
Ours
Ground Truth
Figure 3: Qualitative comparison of MaskTwins with previous methods on VNC Lu-chh Subset2 (row 1) and MitoEM- H-MMitEMMRR (row 2). The pixels in red and green denote the false-negative and false-positive results respectively.
tives, thereby achieving effective adaptation from the source domain to the target domain.
new challenge, we re-implement SSNS-Net (Huang et al., 2022a), AdaSyn (Chen al., 2024) and MIC (Hoyer et al., 2023) strictly following their experimental implementations and make fair comparison. Table 3 shows that MaskTwin achieves the highest Fi-score with an outstanding gain of 2.02% totally, 3.13% on post-synapse. Due to its high den- sity and the one-to-many synapse connectivity problem, post-synapses are more difficult to identify. Other meth- ods perform poorly on post-synapse detection with a sharp decline in the performance of detecting the post-synapses. However, MaskTwins can learn robust features and capture more post-synapses correctly with the help of complemen- tary masks. The results demonstrate that the dual-form complementary masking is effective even in the task of 3D synapse detection, showing remarkable capability in ex- tracting domain-invariant features across various domains and increases the robustness of network.
5.4. Synapse Detection
We also evaluate the effectiveness of our proposed method on 3D synapse detection. This task aims to pinpoint the positions of pre-synaptic and post-synaptic sites in the 3D space, as well as to determine the connectivity between them, specifically identifying the IDs of the pre-synapses to which the post-synapses are linked. To further understand the complexity, it's worth noting that synapses often appear in highly cluttered regions, making accurate identification even more challenging. For more vivid depiction of the detection outcomes, we visualize the 3D results of pre- and post-synapse detection in Appendix G.2. Following Chen et al. (2024), we convert the task of synapse detection into a segmentation task. Since there is few prior works on this
7

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Table 3: Comparison result on the WASPSYN Challenge. The Fi-score is the average of Pre and Plptt. The com- petitors are SSNS-Net (Huang et al., 2022a), AdaSyn (Chen al., 2024) and MIC (Hoyer et al., 2023).
Table 5: Ablation study of the patch size b and the mask
ratio r of MaskTwins on SYNTHIA -Cttyccpps with a input size of 1024. Specially, the mask ratio r indicates a complementary combination of [r, 1-rl. r].
(a) Mask Ratio.
(b) Patch Size.
Method
Fl pre
Fl post
Fl Fl-score
SSNS-Net
0.7201
0.3072
0.5137
Mask Ratio
mloU
Patch Size
mloU
AdaSyn
0.7846
0.3136
0.5491
0.1
72.0
32
76.2
MIC
0.7823
0.3599
0.5711
0.2
74.6
64
76.7
Mask MwskTrins(Ours)
0.7914
0.3912
0.5913
0.3
75.4
128
75.9
0.4
76.5
256
75.6
Table 4: Ablation study of the components and masking strategy of MaskTwins on SYNTHIA--Cityscpee includ- ing target-domain consistency learning loss (CL), comple- mentary masking strategy (CMask), random masking strat- egy (RMask) as an alternative, EMA teacher and AdaIN.
0.5
76.7
512
75.0
cally substantiate the proved theorems in Section 3, showing the effectiveness of complementary masking strategy pared to random one.
CL
CMask
RMask
EMA
AdaIN
mloU
53.7
Patch Size and Mask Ratio. Table 5 demonstrates the effect of the mask patch size b and mask ratio on SYNTHIA-
72.8
74.3
with a input size of 1024. We systematically alter the mask ratios and specifically ex- plore the combinations of [r, r] with a mask ratio {0.1, 0.2, 0.3, 0.4, 0.5} in complementary form. By gradually increasing the mask patch size, we observe that the best performance is achieved when b 64, i.e., 1/16 of the input size. Patches that are either larger smaller exhibit varying degrees of performance reduction. This is likely because patches that are too large may excessively cover the foreground while those that are too small tend to apply an overly dense masking, potentially hindering the complementary learning of contextual information. On the contrary. by concentrating
V
V
74.6
V
V
75.0
V
75.2
76.0
V
76.1
76.4
V
76.7
5.5. Ablation Study
Component Ablation. First, ablate each component of MaskTwins on SYNTHIA-CCiyscape in Table 4. When the target-domain information is lacked, the performance is only 53.7 mloU, since the source model trained with a single supervised loss has a poor ability of generalization. Com- pared to MIC, random masking yields a improvement of +1.2 mloU, which demonstrates the effectiveness of the dual- form masking consistency. Replacing the random masking strategy with the proposed complementary masking strategy, the complete MaskTwins achieves 76.7 mloU, which is +3.9 mIoU better than the domain adaptive baseline with con- sistency learning loss. On the other side, without the EMA teacher and AdaIN module, the performance only reduces by -0.6 mIoU and -0.3 mloU, respectively. Therefore, mask- ing strategy contributes most the improvements among other used components of Twnks And the complemen- tary masking can achieve significantly higher improvement by only changing the masking strategy, further validating the substantial performance boost brought about by the com- plementary masking and the total gain does not rely on the masking strategy itself. The experiment results unequivo-
context-rich areas using appropriate mask patch size, the model can better utilize the spatial relations within the image, leading to improved performance in unsupervised domain adaptation. For mask ratio, we observe that the performance gets better when increasing the mask ratio r to 0.5. Compared MIC, Mask- Twins consistently achieves significant improvements in a range of b between 32 and 256 and between 0.2 and 0.5. Only for a mask ratio r of 0.1, MaskTwins decreases the per- formance. The best performance is achieved when = 0.5 and b = 64, i.e., 1/16 of the input size. Therefore, we use these parameter settings in all experiments.
6. Conclusion
In this work, present novel perspective masked reconstruction by reinterpreting it as sparse signal recon- struction problem and theoretically prove the effectiveness of the dual form of complementary masks. Based on this thcoretical foundation, we propose MaskTwins, an effective framework that utilizes complementary masks to simulta-
8

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
neously enhance the robust feature extraction for domain- adaptive segmentation. Our MaskTwins has demonstrated remarkable superiority over the state-of-the-art methods across a diverse range of domain adaptation scenarios, span- ning from natural to biological imaging and from 2D to 3D modalities. For instance, MaskTwins respectively achieves significant performance improvements by +2.7% and +2.5% in IoU on SYNTHIA CCttsspppss and biological datasets. Since Tann'wwins performs masked image consistency with- out extra annotations, it offers flexible technique that can be seamlessly incorporated with other methods to further facilitate the learning of Nomain-invarrinn features, ensuring the cross- cross-domain knowledge adaptation process. In the fu- ture, we will continue to explore the potential MaskTwins in broader spectrum of visual recognition challenges, in- cluding but not limited domain-adaptive video segmenta- tion and image classification.
Brown, T. B. Language models are few-shot learners. arXiv
preprint arXiv:2005..4166: 2020.
Chen, M., Xue, H., and Cai, D. Domain adaptation for semantic segmentation with maximum squares loss. In ICCV, 2019.
Chen, Q., Huang. W., Zhang, Y., and Xiong, Z. Domain adaptive synapse detection with weak point annotations. In ISBI. IEEE, 2024.
Choi, J., Kim, T., and Kim, C. Self-ensembling with gan- based data augmentation for domain adaptation in se- mantic segmentation. In Proceedings of the IEEE/CVE international conference computer vision, pp. 6830 6840, 2019.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, The cityscapes dataset for semantic urban scene under- standing. In CVPR, 2016.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 624B2137 and 62021001.
Devlin, J. Bert:
Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint arXiv:1810004805, 2018.
Impact Statement
DeVries, T. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.0453 2017.
This work aims to advance the field of machine learning. While the research may have various societal implications, we deem it unnecessary to highlight specific ethical or social impacts herein.
Dong. X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, I.., Chen, D., Wen, F., and Yu, Bootstrapped masked autoencoders for vision bert pretraining. In ECCV, 2022.
References
Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., Yu, N., and Guo, B. Peco: Perceptual codebook for bert pre-training of vision transformers. In
Araslanov, N. and Roth, S. Self-supervised augmentation consistency for adapting semantic segmentation. Pro- ceedings of the IEEE/CVF conference computer vision and pallern recognition, pp. 15384-15394 2021.
AAAI, 2023.
Donoho, D. L. Compressed sensing. IEEE Transactions on Information Theory, 5284):1299 2006.
Bao, H., Dong, L., Piao, S., and Wei, F. BETT: BERT pre-training of image transformers. In International Con- ference on Learning Representations, 2022.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Gen-David, S., Blitzer, J., Crammer, K., and Pereira, F. Anal- ysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Gerhard, S., Funke, J., Martel, J.. Cardona. A., and Fetter, R. Segmented anisotropic sstem dataset of neural tissue. figshare, pp. 0-0, 2013.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F. and Vaughan, J. W. A theory of learning from different domains. Machine learning, 79:151-175, 2010.
Goodfellow, I., Pouget-Abadie J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Bermúdez-Chacón, R., Márquez-Neila, P., Salzmann, M., and Fua, P. A domain-adaptive two-stream u-net for electron microscopy image segmentation. In ISBI. IEEE, 2018.
Guan, D., Huang, J., Xiao, A., and Lu, S. Domain adaptive video segmentation via temporal consistency regulariza- tion. In ICCV, 2021.

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Lee, K., Zung, J., Li, P., Jain, V., and Seung, H. S. Super- human accuracy on the snemi3d connectomics challenge. arXiv preprint arXiv: 00120, 2017.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Li, G., Kang, G., Liu, W., Wei, Y., and Yang. Y. Content- consistent matching for domain adaptive semantic seg- mentation. In ECCV, 2020.
Hoffman, J., Wang, D., Yu, F., and Darrell, T. Fens in the wild: Pixel-level adversarial and constraint based adaptation. arXiv preprint urXiv:1612.02649, 2016.
Li, J., Zhou, K., Qian, S., Li, W., Duan, L., and Gao, S. Fea- ture c--rereeeetttion and reliable pseudo label retraining for cross-domain semantic segmentation. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 46 1694, 2022.
Hoyer, I.., Dai, D., and Van Gool, L.. Daformer: Improving network architectures and training strategies for domain- adaptive semantic segmentation. In CVPR, 2022a.
Li, Y., Yuan, L., and Vasconcelos, N. Bidirectional learn- ing for domain adaptation of semantic segmentation. In CVPR, 2019.
Hoyer, L., Dai, D., and Van Gool, L. Hrda: Context-aware high-resolution domain-adaptive semantic segmentation. In ECCV, 2022b.
Li, Y., Li, W., Chen, Q., Huang, W., Zou, Y., Xiao, X., Shi- nomiya, K., Gunn, P., Gupta, N., Polilov, A., et al. Wasp- syn: A challenge for domain adaptive synapse detection in microwasp brain connectomes. IEEE Transactions on Medical Imaging. 2024.
Hoyer, L., Dai, D., Wang, H., and Van Gool, L. Mic: Masked image consistency for context-enhanced domain adaptation. CVPR, 2023.
Huang, W., Chen, C.,, Xiong, Z., Zhang, Y., Chen, X., Sun, X., and Wu, F. Semi-supervised neuron segmentation via reinforced consistency learning. IEEE Transactions Medical Imaging, 41(11):30163300 2022a.
Liu, D., Zhang, D., Song, Y., Zhang, F,, O'Donnell, L., Huang, H., Chen, M., and Cai, W. Pdam: panoptic- level feature alignment framework for unsupervised do- main adaptive instance segmentation in microscopy im- ages. IEEE Transactions on Medical Imaging, 40(1): 154-165, 2020a.
Huang, W., Liu, X., Cheng, Z., Zhang, Y., and Xiong, Z. Do- main adaptive mitochondria segmentation via enforcing inter-section consistency. In MICCAI, 2022b.
Liu, H., Jiang, X., Li, X., Guo, A., Hu, Y., Jiang, D., and Ren, B. The devil is in the frequency: Geminated gestalt autoencoder for self-supervised visual pre-training. In AAAI, 2023.
Huang, X. and Belongie, S. Arbitrary style transfer in real- time with adaptive instance normalization. Proceed- ings of the IEEE International Conference on Computer Vision, 2017.
Liu, X., Han, Y.. Bai, S., Ge, Y., Wang, T., Han, X., Li, S., You, J., and Lu, J. Importance-aware semantic segmenta- tion in erf-driiiig with discrete wasserstein training. In AAAI, 2020b.
Jangg Z., Li, Y., Yang, C., Gao, P., Wang, Y., Tai, Y., and Wang, C. Prototypical contrast adaptation for domain adaptive semantic segmentation. In ECCV, pp. 36-54. Springer, 2022.
Long, M., Cao, Z., Wang, J., and Jordan, M. I. Condi- tional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Jin, Y., Wang, X., Long, M., and Wang, J. Minimum class confusion for versatile domain adaptation. ECCV, 2020.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4/2.44120 2014.
Lucchi, A., Li, Y., and Fua, P. Learning for structured prediction using approximate subgradient descent with working sets. CVPR, 2013.
Kong, X. and Zhang, X. Understanding masked image modeling via learning occlusion invariant feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Luo, Y., Liu, P., Guan, T., Yu, J., and Yang. Y. Significance- aware information bottleneck for domain adaptive seman- tie segmentation. ICCV, 2019.
Lee, D. et al. Pseudo-label: The simple and efficient sumi-supervised learning method for deep neural net- works. In ICML., 2013.
Luo, Y., Liu, P., Zheng, L., Guan, T., Yu, J., and Yang, Y. Category- level adversarial adaptation for semantic seg- mentation using purified features. IEEE Transactions on
10

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Puttern Analysis and Machine Intelligence, 44(8):3940- 3956, 2021.
Wang, H., Tang. Y., Wang, Y., Guo, J., Deng, Z.-H., and Han, K. Masked image modeling with local multi-scale reconstruction. In CVPR, 2023.
Matthews, B. W. Comparison of the predicted and observed secondary structure of 14 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442- 451, 1975.
Wang. Y., Peng, J., and Zhang, Z. Uncertainty pseudo label refinery for domain adaptive semantic segmentation. In ICCV, 2021.
Mei, K., Zhu, C., Zou, J., and Zhang, S. Instance adap- tive self-training for unsupervised domain adaptation. In ECCV, 2020.
Wei, C.,, Fan, H., Xie, S., Wu, C.-Y., Yuille, A., and Feicht- enhofer, C. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
Wei, D., Lin, Z., Tranco-Barranee D., Wendt, N., Liu, X., Yin, W., Huang, X., Gupta, A., Jang, W.-D., Wang, X., al. Mitoem dataset: Large-scale 3d mitochondria in- stance segmentation from em images, In MICCAI, 2020.
Melas-Kyriazi, L. and Manrai, A. K. Pixmatch: Unsu- pervised domain adaptation via pixelwise consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12435- 12445, 2021.
Wu, S., Chen, C., Xiong, Z., Chen, X., and Sun, X. Uncertainty-awan label rectification for domain adap- tive mitochondria segmentation. In MICCAI, 2021.
Peng. J., Yi, J., and Yuan, Z. Unsupervised mitochondria segmentation in em images via domain adaptive multi- task learning. IEEE Journal of Selected Topics in Signal Processing, 14(6):1199-1200) 2020.
Xie, E., Wang, W., Yu, 7., Anandkumar, A., Alvarez, J. M., and Luo, P. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34: 12077-10000 2021.
Peng, X., Usman, B., Kaushik, N., Hoffman, J., Wang, D., and Saenko, K. Visda: The visual domain adaptation challenge. arXiv preprint 1rXiv:1710.0000
2017.
Xie, J., Li, W., Zhan, X., Liu, Z., Ong, Y. and Loy, C. C. Masked frequency modeling for self-supervised visual pre-training, arXiv preprint 2xXiv:2006.07706, 2022a.
Rangwani, H., Aithal, S. K., Mishra, M., Jain, A., and Radhakrishnan, V. B. closer look at smoothness in domain adversarial training. In ICML, 2022.
Xie, Z.,, Zhang, Z.,, Cao, Y., Lin, Y., Bao, J., Yao, 7., Dai, Q., and Hu, H. Simmim: A simple framework for masked image modeling, In CVPR, 2022b.
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., and Lopez, A. M. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
Xu, T. Chen, W., Wang. P., Wang, F., Li, H., and Jin, R. Cdtrans: Cross-domain transformer for unsupervised domain adaptation. arXiv preprint arXiv:2109.061 2021.
Shin, U., Lee, K., Kweon, I. S., and Oh, J. Complementary random masking for rgb-thermal semantic segmentation. In ICRA, 2024.
Yang, J., An, W., Wang, S., Zhu, X., Yan, C., and Huang, J. Label-driven reconstruction for domain adaptation in semantic segmentation. In ECCV, 2020a.
Tarvainen, A. and Valpola, H. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Yang, J., Xu, R., Li, R., Qi, X., Shen, X., Li, G., and Lin, An adversarial perturbation oriented domain adaptation approach for semantic segmentation. In AAAI, 2020b.
Tranheden, W., Olsson, V., Pinto, J., and Svensson, L. Dacs: Domain adaptation via cross-domain mixed sampling. In WACV, 2021.
Yang, J., An, W., Yan, C., Zhao, P., and Huang, J. Context- aware domain adaptation in semantic segmentation. In WACV, 2021.
Tsai, Y.H., Hung, W.-C., Schulter, S., Sohn, K., Yang, M.- H., and Chandraker, M. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018.
Yang, J., Liu, J., Xu, N., and Huang, J. Tvt: Transferable vision transformer for unsupervised domain adaptation. In WACV, 2023.
Vu, T.-H., Jain, H., Bucher, M., Cord, M., and Pérez, Dada: Depth- domain adaptation in semantic seg- mentation. In ICCV, 2019.
Yang, L., Hoyer, L.,, Weber, M., Fischer, T., Dai, D., Leal- Taixé, L.., Pollefeys, M., Cremers, D., and Van Gool,

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Micdrop:
masking image and depth features via comple- mentary dropout for domain-adaptive semantic segmen- tation. In European Conference on Computer Vision, pp. 329-346. Springer, 2025.
Zhou, W., Wang, Y., Chu, J., Yang. J., Bai, X., and Xu, Y. Affinity space adaptation for semantic segmentation across domains. IEEE Transactions Image Processing, 2020.
Yang, Y. and Soatto, S. Fda: Fourier domain adaptation for semantic segmentation. In CVPR, 2020.
Zou, Y., Yu, Z., Kumar, B., and Wang, Unsupervised domain adaptation for semantic segmentation via class- balanced self-training. In ECCV, 2018.
Yin, D., Huang, W., Xiong, Z., and Chen, X. Class-award feature alignment for domain adaptative mitochondria segmentation. In MICCAI. Springer, 2023.
Zou, Y., Yu, Z., Liu, X., Kumar, B., and Wang. J. Confidence regularized self-training. In CVPR, 2019.
Yu, F., Zhang, M., Dong, H., Hu, S., Dong, B., and Zhang, L. Dast: Unsupervised domain adaptation in semantic segmentation based on discriminator attention and self- training. In AAAI, 2021.
Zhang, P., Zhang, B., Zhang, T., Chen, D., Wang, Y., and Wen, F. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmen- tation. In CVPR, 2021.
Zhang, Q., Zhang, J., Liu, W., and Tao, D. Category anchor- guided unsupervised domain adaptation for semantic seg- mentation. Advances in neural information processing systems, 32, 2019a.
Zhang, Y., Liu, T., Long, M., and Jordan, M. Bridging theory and algorithm for domain adaptation. ICMI., 2019b.
Zhao, D., Wang. S., Zang, Q. Quan, D., Ye, X., Yang, R., and Jiao, L. Learning pseudo-relations for cross-domain semantic segmentation. In ICCV, 2023.
Zheng, Z. and Yang, Y. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision, 12964):106-1122) 1120, 2021.
Zheng. Z. and Yang. Adaptive boosting for domain adap- tation: Toward robust predictions scene segmentation. IEEE Transactions on Image Processing, 31:5371-5388 2022.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:21 /1.0232, 2021.
Zhou, Q., Feng, Z., Gu, Q., Cheng, G., Lu, X., Shi, J., and Ma, L. Uncertainty-awar consistency regularization for cross domssnn semantic segmentation. Computer Vision and Image Understanding, 221: 2022a.
Zhou, Q. Feng, 7., Gu, Q. Pang. J., Cheng, G., Lu, X., Shi, J., and Ma. L. Context-aware mixup for domain adaptive semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 33(2):804- 817, 2022b.
12

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
A. Theory Proofs
A.1. Complementary Masking Theory: Mean and Variance Analysis
Definition 1:
Complementary Mask Let D E {0, 1]d be a random binary vector where each clement D, is independently drawn from Bernouli((0.5). The complementary mask is D, where is the vector of ones in I2
Definition 2: Random Masks Let D1, D2 E {0, 1 be independent random binary vectors where each element Dac (for
1, 2) is independently drawn from Bernoulli0055) These are the random masks.
A.2. Information Preservation Metric
Given a deterministic vector E Rd, we define masked versions of as:
For complementary masks:
21 = DD
T2 (1 - D) O
For random masks:
11 DI0x, 32 D2 02
where C denotes element-wise (Hadamard) product.
Define the information preservation (IP) metric as:
(x1, .2)
(P(11,2))
||0||²
A.3. Mean and Variance Computations
1. COMPLEMENTARY MASKS
Mean:
For complementary masks, note that for each coordinate d:
D:(1 - D2) = 0
because D, is either 0 or 1.
Therefore, the inner product:
(2), Σ D 0
Thus,
IP(21,02) = ||
and
E[I((11.22) =0
Variance:
Since I2))).I2) = 0 almost surely,
Var(IP((11,
A.3.2. RANDOM MASKS
Mean:
13

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
For random masks:
11 F22 =
D1: 2210
Since D1s, D2s independent Bernoulli(0.5,, have:
1
E B(D1,
2
Therefore,
1 B (x1,
||:||
and
E[I((1)
Variance:
Compute 2x:);:
(4)
2
3
(ar(D), =
16
Then,
3
3
Var((E1,22)
=
4
16
16
1=1
Thus,
3
Var(IP x2))
16 (1|ar 12)
Remark 2. Complementary masks offer several significant benefits in data processing and analysis. Their ability to produce uncorrelated masked data stands out as a primary advantage, ensuring that each masked subset provides unique information. The deterministic nature of these masks, characterized by zero variance, guarantees predictable outcomes, which crucial for reproducibility in research and applications. Complementary masks excel in efficient data partitioning. creating distinet subsets without redundancy, thus optimizing computational resources. From security and privacy perspective, these masks enhance data protection, neither mask alone reveals the complete information, adding a layer of confidentiality sensitive data. The consistency provided by complementary masks is particularly valuable in applicutions requiring deterministic results, ensuring that repeated analyses yield identical outcomes. This combination of features makes complemenaryy masks a powerful tool in various fields, from data science to cryptography, offering a balance of efficiency, security, and reliability.
Theorem 4 (Consistency Bound for Feature Learning). Consider general feature learning framework with the objective function:
L()) = F. [e (((())
where f Rd > R is the feature extraction function, e: R x R + R the loss function, and (x1, is a sample pair generated from input data x after applying masks or transformations.
Assume:
(a) The loss function is L-Lipschitz continuous with respect to both arguments, i.e., any a, b, c, E RR.,
|l(a, b) - E(c, < L (Iaa - c|12 + dl|2).
(b) The feature extraction function J is B-Lipschitz continuous (or B-smooth), i.e., for any x. y Rd.
(l)(x) - f(u)ll2 S B1| -Vl2.
14

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
(c) The inpul data x takes values in compact subset X R2, and sup || B. Then, for any & E (0, 1). with probability at least 1 - 8. the following holds:
(i) For complementary masks:
[CU) - (.) & L3B LLBBI
(4+ /
2loo((200)
where = (f(z1)), ((2x)) is the empirical risk computed n samples.
(ii) For random masks:
[C) - 2n0) LLBB (4+ 22log(2/6) + 2L33
Proof. We will prove the bounds for both complementary masks and random masks separately.
Case (i): Complementary Masks
Step 1: Define the Function Class
Let F = {a - (f(De)) - ((())) : f is B-Lipschitz), where D is a deterministic mask operator (for complementary masks).
Step 2: Bounding the Rademacher Complexity
Consider the empirical Rademacher complexity of F:
1 Ry (F) = E. sup - Σ ((U
SEF n f=1
where o = (aI,..., σn) are independent Rademacher random variables (i.e., PP = +1) +1) Pa(a = =11 1/2)).
Using the Lipschitz property and f. we have:
(F) S LE.
sup
σ (IIf(Dz:) [[f([[[ - -D).
71
< LE.
1
0s (I)f(Dz;)) f(0)l12 If((I- D)x,) - F(0))l2)
n
n
< 2L
|o 112
71
1
2L3 n
|| Eg. |0
2L3 -
||æe|le
EE, [1]
1
2L3 11
||a:|12
<
2L3B,
since |2 < B. However, to get a dependence on n, we consider the Rademacher complexity bound for Lipschitz functions, which gives:
2L3B
Ri, <
Vni
15

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Step 3: Apply Concentration Inequalities
By McDiarmid's inequality, since changing sample affects the empirical loss by most
2L3B
have for any > 0:
P (.) E [EC Ln +1) s 2 eexp
2nl
(T21,3B))
Setting t = L3B
2 2
we get with probability at least 6:
( + L log((/0)
PC) - 2. & 2n11
Step 4: Combine the Bounds
Using symmetrization and the bound on Rr, we have:
4L33
( 2n(ss]]
(F) 5
vn
Therefore, combining the above, we have:
22 22log(200
T.3BI/ / (4+ 11++2log(222 22log(2/8)
- LBB
LBB
Vn
n
Case (ii): Random Masks
Step 1: Modify the Function Class
Let Frand = fa + E (f f is 3-Lipschitz, R1, R2 are random masks).
Step 2: Bounding the Rademacher Complexity
Similarly, we consider:
n orl (f(R)
Rr (Frand)
Es, Ra
sup n
Again, using Lipschitz properties, have:
(Fand) S LEo Ri sup Σe σr (Il((Rr))) - f(0)ll2 + ( f(0)12)
Since is 3-Lipschitz and ||as||| S B, we have:
( - f(0)l22 S ( Ry,, -012--
Given that Ry is a random mask (e.g., a diagonal matrix with entries being Bernoulli random variables), we have:
d 1111112= 1|r.112.
E Ri [| 12]
Σ Bi(RI))
assuming each (Ri;) is independent and takes value with probability 1/d.
16

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Therefore,
ER. [1l()))))))) f(0)ll2) s SER, || s BV BR, R1,, [2 3 Vi
Similarly for Rzi.
Therefore,
B
R, (Frad) S 2L3 Vd
Step 3: Apply Concentration Inequalities
Following similar steps as in the complementary masks case, and accounting for the extra term due to random masks, we have:
I L3B
12g(2
- 2.))
4LBB
2LBB
n
n
Vd
Since
Vá 1 Vz for d I n, we can write:
I (CC) (C))-2((11 LBBV
( /2 2log(2/))
+2LBB
This completes the proof.
Theorem (Signal Recovery Guarantee). Let RRd be signal generated from the sparse linear model:
I Mz E.
where:
M E pdxn is known measurement matrix (dictionary),
I. a k-sparse vector (i.e., ||~/1o S k),
o~0,000 is additive Gaussian noise.
Suppose we have masking matrices R1, R2 Rdxd representing partial observations of x::
For complementary masks, Rı and R2 satisfy R R, = 0 and R R1 + Rí R2 Id. Le., they partition the indices of x without overlap and cover all entries.
For random masks, R₁ and R2 select entries independently at random.
Define the aggregated observation y E 1m2m as:
1/1
Riæ Ro
RR M R&M
RRI Ro2
Az 1
y2
where R2mxx is the effective measurement matrix, and 1 E R2m is the aggregated noise.
Assume that A satisfies the Restricted Isometry Property (RIP) of order with constant 82z 8 for some 8* <
17

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Let å be the solution the basis pursuit denoising problem:
= arg min ||u]] subject to wERn
l1 Aull2 S e,
where 2 1|n1|2-
Then, for any 8 > O, with probability at least 1 - 8.
|| - S Coy k (log(n/0)
where C > is a constant depending only on the RIP constant 82k-
Moreover, when Rí and R2 are complementary masks that together cover entries of x without overlap, and m = d/2. the recovery error achieves the bound:
h log((/6))
- - 2112 S V
d
where C1 a constant depending only on 62k-
Proof. We will establish an upper bound on the estimation error || - 2|12 under the given assumptions.
Step 1: Formulating the Observations
The observations are:
/1 = R1 = R1 + R1M2 + RIE,
1/2 R2 R2 +) R2M= RzC.
By stacking y1 and y/, we have:
1/ 1/2
R R22
R
y
2+
Az + 1.
R25
Ri M R2M
RIC RaC
where A
R 2mxx and
2m
Step 2: Recovering = via Basis Pursuit Denoising
We consider the optimization problem:
arg min ||u/] "ER"
subject ly Aull2 $ e,
with 2 ||/]|2-
Our goal is to bound ||| -2112-
Step 3: Applying Compressed Sensing Recovery Guarantees
Since satisfies the RIP of order 2k with constant 82g < 8", standard compressed sensing results (e.g., Candès et al. (2006)) imply that:
||ll
1| 21|2 S Co
Vm
where Co 0 depends only on d2x.
Step 4: Bounding |!7||2
The noise vector 7 consists of 2m components, each being either or zero. Since & ~ (0,00 each nonzero entry of is 10, o2).
Therefore, ||m| is the sum of 2m independent xi random variables, where xi denotes a chi-squared distribution with one degree of freedom.
18

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Using concentration inequalities for chi-squared distributions (see, e.g., Laurent & Massart (2000)), for any 0:
Pr [Inll 2mo² (1+ (122/=122+++++++202/0/
Se
Setting & = m log(n/6), we obtain:
1 + 2/ log(n/6) 2 2log(n/0) S
[1l12 2 2mo²
m
For sufficiently large m, the terms involving 1/m become negligible, and have, with probability least 1 8:
n ||ll2 - V m log (5)
where Cj > 0 is constant.
Step 5: Final Estimation Error Bound
Substituting the bound on |!vl]a into recovery guarantee:
Cia Clayma mlggn//))
Coy/ log (7))
2 - - S Co
vm
where CoC1.
To incorporate the sparsity k consider the number of possible supports of size which is (7) Applying a union bound over all supports, have:
Pr (12 [12-z10 =1|2 SCy ooo (7)
2 1 8.
Noting that log (+) < kloogn/k)), refine the bound:
_| =2112 S Ca k log
n
< C'ov
k klog(n/8)
where C" is a constant.
Step 6: Special Case with Complementary Masks
When R and Ra complementary and m d/2, substituting d/2 yields:
2k log(n//))
klo((n/0)
|| - 5 30llac''o C'ov'
Coov
d
d
Remark 3 (Advantages of Complementary Masks), Complementary masks offer significant advantages in compressive sensing applications, enhancing both the theoretical foundations and practical implementations. These masks maximize measurement utilization by covering entries of the signal x without overlap, ensuring optimal use of available information. This comprehensive coverage leads to improved Restricted Isometry Property (RIP) constants for the measurement matrix A, resulting in tighter recovery bounds. The noe-overlapping nature of complementary masks also plays crucial role in minimizing noise influence, it prevents noise accumulation and effectively reduces ||7]]22 A key benefit is the improved recovery accurucy, where the error bound scales inversely with the dimensionality d of x, leading to enhanced recovery performance. Furthermore, the structured nature of these masks contributes to algorithumic efficiency, facilitating faster and more effective computation in practical recovery algorithms. Collectively, these properties make complementary masks a powerful tool in compressive sensing, offering a balanced approach that enhances both theoretical guarantees and practical performance.
19

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
References
Candès, E. J., Romberg. J., & Tao, T. (2006). Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics, 59(8), 1207-1223.
Laurent, B., & Massart, (2000). Adaptive estimation a quadratic functional by model selection. Annals of Statistics, 28(5), 1302-1338.
B. Applications and Extensions
B.1. Self-Supervised Learning
The complementary masking theory can be directly applied to self-supervised learning tasks, particularly in contrastive learning frameworks. Here, we present a corollary that demonstrates how our theory can be used to analyze the performance of contrastive learning algorithms.
Corollary 6 (Contrastive Learning with Complementary Masks), Consider contrastive learning setup where positive pairs are generated using complementary masks (D, - D). Let So : IRd + R& be the encoder network parameterized by and let the contrastive loss be defined as:
((0)
-EE log
where sim(-.-) is the cosine similarity and a temperature parameter. Then, under the assumptions of Theorem 2, with probability least - 6:
L3B (V
log(1 (15)
[C(0) (0) 20)) SO
n
where (0) is the empirical loss on n samples. L is the Lipschitz constant of the loss function, 3 is the smoothness parameter of fo, and the bound on the input norm.
Proof. The proof follows directly from Theorem by observing that the contrastive loss is Lipschitz continuous with respect to the encoder outputs, and the encoder network is assumed to be 3-smooth. The key step is to apply the consistency bound complementary masks to the positive pair (Dx, (I D)z) the numerator of the contrastive loss.
This corollary provides a theoretical justification for using complementary masks in contrastive learning algorithms. It suggests that the generalization error such algorithms scales favorably with the number of samples and independent of the input dimension, which crucial for high-dimensional data such as images.
B.2. Extension to Multi-View Data
The complementary masking theory can be extended to scenarios where we have multiple views of the data, not just two. This extension is particularly relevant for multi-view learning and modal data analysis.
Theorem 7 (Multi-Vien Complementary Masking). Let x E IRd be the input data, and consider K complementary masks Dı,..., Dx such that I D; Define the multi-view information preservation metric as:
(wit xz))
MIP(II,..., K(K 1)
|
where v, = D,x. Then:
E[MIP(# æK)] K²
20

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
2. Var. rK)) v
K-1 K
Proof. (Sketch) The proof follows similar structure to that of Theorem 1, but requires careful accounting of the pairwise interactions between the K views. The key insight is that the complementary nature of the masks ensures that the expected overlap between any two views is 1/K2 of the total information.
This niet extension opens up possibilities for analyzing and designing algorithms that work with more than two views of the data, such as multi-view clustering or multi-moda fusion techniques.
C. Conclusion
The complementary masking theory presented this paper provides a rigorous framework for analyzing information preservation in masked data representations. The key advantages of complementary masks over random masks include:
1. Tighter generalization bounds in feature learning tasks.
2. More robust signal recovery guarantees, especially in the presence of strong signals.
3. Guaranteed preservation constant fraction of the original information.
These theoretical results have immediate implications for the design and analysis of self-supervised learning algorithms,, particularly in contrastive learning setups. They also provide insights into why certain masking strategies might outperform others in practice.
21

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
D. Extended Related Works on UDA
Adversarial learning. Hoffman et al. (2016) are the first to apply the adversarial approach for UDA on semantic segmentation encourage domain-invariant alignment globally. SIBAN (Luo al., 2019) employs a significance- aware information bottleneck (SIB) before the adversarial feature adaptation to extract latent representations in semantic segmentation tasks. FDA (Yang & Soatto, 2020) performs spectral transfer by swapping the low-frequency component of the spectrum. APODA (Yang et al., 2020b) explicitly trains Nomainninvariial classifier by generating and defensing against point-wise feature space adversarial perturbations, in order to adapt the representations of the tail classes or small objects for semantic segmentation. DAST (Yu et al., 2021) proposes self-training strategy which adaptively improves the decision boundary of the model for target domain and implicitly facilitates the extraction of
features.
Pseudo-label self-training. DADA (Vu et al., 2019) introduces a novel depth-aware adaptation scheme while BDL (Li et al., 2019) proposes a novel bidirectional learning framework for domain adaptation of segmentation. DACS (Tranheden et al., 2021) mixes images from the two domains along with the corresponding labels and pseudo labels to perform Cross-domain mixed Sampling. Some generative methods try to acquire target-like synthetic images by content-consisten matching (CCM) (Li et al., 2020) or label-driven reconstruction (LDR) (Yang al., 2020a). To improve the quality of pseudo labels, UncerDA (Wang al., 2021) provides uncertainty-aware pseudo label assignment strategy while RPLR (Li et al., 2022) retrains the networks using selected reliable pscudo labels. Many works focus on consistency regularization to capture contextual relations, such CD-SAM (Yang et al., 2021), UACR (Zhou et al., 2022a), CAMix (Zhou et al., 2022b), HRDA (Hoyer et al., 2022b) and MIC (Hoyer et al., 2023). Researchers also conducted extensive attempts, including affinity in ASA (Zhou et al., 2020), representative prototypes in ProDA (Zhang et al., 2021), ) and Transformer architecture in DAFormer (Hoyer et al., 2022a).
Theory for UDA. The theoretical works (Ben-David et al., 2006; 2010; Zhang et al., 2019b) provide fundamental insights into UDA, especially concerning domain discrepancy and theoretical bounds. Specificciff,llly. they study margin bounds for classification tasks at the distribution level, while we focus on segmentation tasks and the theory of Masked Image Modeling and compressed sensing the image level. We have analyzed the information preservation, generalization bounds and feature consistency to demonstrate the effectiveness of complementary masking. Zhang et al. (2019b) discuss generalization bounds based empirical Rademacher complexity, building upon the domain adaptation theories presented in previous works such as those by Ben-David et al. (2006; 2010). We preliminary observe that there exists deeper connections between these works and ours. Hopefully, we will make further thcoretical analysis in the future work.
E. MaskTwins Training Procedure
We provide the overall training procedure of MaskTwins for image segmentation in Algorithm 1.
Algorithm MaskTwins Algorithm
Input: Source domain Ds, Target domain Dr, student model fe, teacher model fo, the total iteration number N. 1: Initialize network parameter 0 with ImageNet pre-trained parameters, Initialize teacher network & randomly.
2: for iteration = 1 to N do
3: x =,,y² ~ Ds-
4:
xT ~ ~DI.
5:
p + )..(xS).
6:
8T + argmaxfo(x"
7:
XT. Xi Patch-wise complementary masking by Eq. 10 and 11.
8:
PD fo(x)), ((_.).pp.> + fo(x
9:
Liotal + Total loss by Eq. 17.
10:
Compute VoLioaal by baok-prooagttion.
11:
Perform stochastic gradient descent on 0.
12:
Update teacher network á with 0.
end for
14:
return f8-
22

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
F. Experimental Details
E.I Natural Image Semantic Segmentation
Following common UDA protocols (Tsai ct al., 2018; 8; Zhou et al., 2022b), we use the synthetic dataset SYNTHIA (Ros al., 2016) as the source domain, and the real dataset Cityscapes (Cordts al., 2016) as the target domain. SYNTHIA synthetic dataset composed of 9,400 annotated images with the resolution of 1280 x 960, while Cityscapes consists of 2,975 training and 500 validation real-world images.
We evaluate MaskTwins based on the HRDA (Hoyer et al., 2022b) architecture with MiT-B5 encoder (Xie et al., 2021) pretrained on ImageNet. To be specific, we follow the DAFormer (Hoyer et al., 2022a) self-training strategy and training parameters, i.e. AdamW (Loshchilov, 2017) with a learning rate of 6 x 10-6 for the encoder and 6 x 10-4 for the decoder, 40k training iterations, a batch size of 2, linear learning rate warmup, loss weight Ast 1, an EMA factor a = 0.999 and DACS (Tranheden al., 2021) data augmentation. Since the pseudo labels inevitably introduce noise, we set a quality threshold following MIC (Hoyer al., 2023). We set the pseudo-label box threshold ò 0.8 following and the quality threshold 0.968.
F.2 Mitochondria Semantic Segmentation
We evaluate the proposed method three challenging EM datasets for 2D domain adaptive mitochondria segmentation tasks: VNC III (Gerhard et al., 2013), Lucchi (Lucchi al., 2013) and MitoEM (Wei et al., 2020) dataset. VNC III consists of 20 sections of size 1024 x 1024. The training subset (Subsetl) and the test subset (Subset2) of Lucchi each contain 165 images, with resolution of 1024 x 768 pixels. MitoEM dataset can be divided into (itoEMM-R(Ra)) and MitoEM ((Hummmm)HHmma) Each volume contains 1000 images of size 4096 x 4096, with the first 500 images annotated. Following Huang et al. (2022b). four widely used metrics are used for evaluation, i.e., mean Average Precision (mAP), Fl score, Mattews Correlation Coefficient (MCC) (Matthews, 1975) and Intersection over Union (loU).
We use a five-stage U-Net following Huang al. (2022b) and Yin al. (2023). During training, we randomly crop the original EM section into 512 x 512 with random augmentation including flip, transpose, rotate, resize and elastic transformation. All models are trained for 200k iterations with a batch size of 2. We use the Adam optimizer (Kingma, 2014) with 31 = 0.9 and B3- 0.999. The learning rate is set at 1 x 10-4 and has a polynomial decay with power of 0.9,
E3. Synapse Detection
To further diversify the experiment settings, we study the 3D domain adaptive synapse detection task using the WASPSYN (Li et al., 2024) dataset. The WASPSYN dataset includes 14 image chunks from different brain regions of Megaphragma viggianii, and five of them have point annotations. Specifically, we take the first one as the source data, and the remaining four chunks are considered target data.
The experiments are performed based on 3D ResUNet following Lee et al. (2017). Considering the data are imaged with an set the kernel size for the initial embedding layer
isotropic voxel size, adopt isotropic 3D convolutions. Specifically, to be 5 5 x 5, whereas the convolutional layers subsequently utilize a default kernel size of 3 x 3 x 3. In the training process, we use a crop size of 96 x 96 96 with batch size of 4 and train for 200k iterations. We use an Adam optimizer with base learning rate of 0.0001 and a linear warming in the first 1000 iterations.
23

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
G. More Results
G.1. Classification Tasks
While our main focus is pixel-wise segmentation tasks, we extend our method to classification tasks to further validate its effectiveness.
We conduct additional experiments the VisDA-2017 dataset (Peng al., 2017), which consists of 280,000 synthetic and real images of 12 classes, with ResNet-101 (He et al., 2016) and VITB/16 (Dosovitskiy, 2020). For UDA training, we follow SDAT (Rangwani et al., 2022), which utilizes CDAN (Long al., 2018) with MCC (Jin et al., 2020) and smoothness enhancing loss. We use the same training parameters, i.e. SGD with a learning rate of 0.002, a batch size of 32, and smoothness parameter of 0.02. We use a patch size b=64, a mask ratio r=0.5, a loss weight Aem = 0.01.
As shown in Tables and 7, our method improves the UDA performance by +0.3 and +0.4 percent points when used with a ViT and ResNet network, respectively. The improvement is consistent over almost all classes.
Table 6: Image classification accuracy in % on VisDA-2017 for UDA with ViT-B/16. "Skth" stands for skateboard. The competitors include TVT (Yang et al., 2023), CDTrans (Xu et al., 2021), SDAT (Rangwani et al., 2022), and MIC (Hoyer et al., 2023). The results are adopted from Hoyer et al. (2023).
Method
Plane
Bicycle
Bus
Car
Horse
Knife
Motor
Person
Plant
Sktb
Train
Truck
Mean
TVT
92.9
85.6
77.5
60.5
93.6
98.2
89.3
76.4
93.6
92.0
91.7
55.7
83.9
CDTrans
90.5
77.5
93.6
97.9
86.9
62.8
88
SDAT
98.4
90.9
85.4
82.1
98.5
97.6
96.3
86.1
96.2
96.7
92.9
56.8
89.8
SDAT w/ MAE
97.1
88.4
80.9
75.3
95.4
97.9
94.3
85.5
95.8
91.0
93.0
65.4
88.4
MIC
99.0
93.3
86.5
87.6
98.9
99.0
97.2
89.8
98.9
98.9
96.5
68.0
92.8
Ours
99.1
95.0
86.6
89.0
98.8
99.3
96.8
88.3
98.8
99.1
97.2
69.7
93.1
Table Image classification accuracy in % on VisDA-2017 for UDA with ResNet-101. "Skth" stands for skateboard. The competitors include CDAN (Long et al., 2018), MCC (Jin et al., 2020), SDAT (Rangwani et al., 2022), and MIC (Hoyer al., 2023). The results are adopted from Hoyer al. (2023).
Method
Plane
Bicycle Bus
Car
Horse
Knife
Motor
Person
Plant
Sktb
Train
Truck
Mean
CDAN
85.2
66.9
83.0
50.8
84,2
74.9
88.1
74.5
83.4
76.0
81.9
38.0
73,9
MCC
88.1
80.3
80.5
71.5
90.1
93.2
85.0
71.6
89.4
73.8
85.0
36.9
78.8
SDAT
95.8
85.5
76.9
69.0
93.5
97.4
88.5
78.2
93.1
91.6
55.3
84.3
МIC
96.7
88.5
84.2
74.3
96.0
96.3
90.2
81.2
94.3
95.4
88.9
56.6
86.9
Ours
96.9
88.8
81.8
77.1
96.4
97.2
90.3
83.8
93.3
94.8
90.2
57.4
87.3
24

---

MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
G.2. Visualization Results
We visualize the segmentation results of MaskTwins and qualitatively compare with the state-of-art methods on -ityscapCsyyscppes in Figure 4 and mitochondria datasets in Figure 7. We also provide more visualization for synapse detection the WASPSYN dataset in Figure 5 and 6.
Image
HRDA
MIC
Ours
Ground Truth
Figure 4: More segmentation results on SYNTHIA-
Figure 5: Visualization of the volume in the WASPSYN dataset. Left to right: sections from X-Y, X-Z, and Y-Z plane.
400
100
300
300
100
100
300
300
200
200
100
2001
100
y
100
200
100 y
300
300
400
Figure 6: An example of visualization of the detection results of pre-synapse (left) and post-synapse (right). Dots and lines: magenta-tre true positive, yellow-fale false negative, and eyan-false positive.
25

---

MwskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation
Image
DAMT-Net
DA- VSN
DA-ISC
CAFA
Ours
Ground Truth
Figure 7: More segmentation results on VNC Lu-hhucch Subsetl (row I and 2), VNC III I Subset2 (row and 4), R-iDEM-R--MitEM-EE (row and 6) and H-MEMM--MMMEEM- (row 7 and 8). The pixels in red and green denote the felse-ne and calse-positive segmentation results respectively.
26


Reviews:
[{"review_id": "FphEQ1NjPx", "reviewer": "Reviewer_sqBy", "summary": "This paper explores the connection between Masked Image Modeling (MIM) and consistency regularization in Unsupervised Domain Adaptation (UDA). It reframes masked reconstruction as a sparse signal reconstruction problem and theoretically proves that complementary masks can effectively extract domain-agnostic features. Based on this insight, the authors propose MaskTwins, a UDA framework that integrates masked reconstruction into the main training pipeline. MaskTwins enforces consistency between predictions of images masked in complementary ways, uncovering intrinsic structural patterns across domains and enabling end-to-end domain generalization. Extensive experiments on natural and biological image segmentation demonstrate its superiority over baseline methods, highlighting its effectiveness in extracting domain-invariant features without separate pre-training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and comments. The main concerns are addressed below. \n>**W1:** Misunderstanding on Complementary Masking Techniques\n\nFirst, the ideas related to Masked Image Modeling (MIM) can be novel. For instance, the latest ECCV 2024 paper MICDrop [1] explored the masking of multi-modal information, indicating continuous innovation within the MIM field. And MIM serves as the foundation of our approach. Based on this, our work focus on the extension to domain adaptative tasks and the exploration of the upper bounds of performance using the proposed complementary masking strategy. Meanwhile, the improvements in our approach stem from both the dual-form and the framework. Moreover, we conducted a series of additional experiments specifically on the masks themselves to validate our theoretical analysis.\n\nWe highlight our theoretical contributions, which are an indispensable part of our work and are entirely lacking in existing works. Specifically, we have provided a theoretical foundation for masked reconstruction from a perspective of sparse signal reconstruction problem and have rigorously analyzed the properties of complementary masking from three aspects: information preservation, generalization bound, and feature consistency. For more theoretical details, please see the supplementary materials.\n\n---\n\n> **W2:** More ablation experiments\n\nIn Table 4, we focus on exploring the individual effects of each component by controlling for single variables. More importantly, we aim to compare with random masking to validate the effectiveness of the proposed complementary masking strategy and support our theoretical analysis. Still, to conduct more rigorous experiments, we additionally ablated complementary mask-only and the effect of EMA and AdaIN using random masking strategy.\n\n---\n\nTable R1. Supplementary ablation rxperiment results for Table 4. The last four rows are newly added experimental data.\n\n|  CL  | CMask | RMask | EMA  | AdaIN | mIoU |\n| :--: | :---: | :---: | :--: | :---: | :--: |\n|  -   |   -   |   -   |  -   |   -   | 53.7 |\n|  √   |   -   |   -   |  √   |   √   | 72.8 |\n|  √   |   -   |   √   |  √   |   √   | 75.2 |\n|  √   |   √   |   -   |  √   |   √   | 76.7 |\n|  √   |   √   |   -   |  -   |   √   | 76.1 |\n|  √   |   √   |   -   |  √   |   -   | 76.4 |\n|  √   |   √   |   -   |  -   |   -   | 76.0 |\n|  √   |   -   |   √   |  √   |   -   | 75.0 |\n|  √   |   -   |   √   |  -   |   √   | 74.6 |\n|  √   |   -   |   √   |  -   |   -   | 74.3 |\n---\n> **W3:** Revision of the formatting of tables\n\nThank you for pointing it out. We will reorganize the formatting of tables carefully in the revised paper.\n\n---\n\n**Reference**\n\n[1] Yang L, Hoyer L, Weber M, et al. Micdrop: masking image and depth features via complementary dropout for domain-adaptive semantic segmentation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 329-346.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contributions of this paper relate to the broader scientific literature by advancing the understanding of masked image modeling (MIM) in the context of unsupervised domain adaptation (UDA). It builds on prior works that connect MIM and consistency regularization but goes further by reframing masked reconstruction as a sparse signal problem and proving the effectiveness of complementary masks for domain-agnostic feature extraction. This theoretical grounding differentiates it from previous methods that treat masking superficially. The proposed MaskTwins framework integrates these insights directly into the training pipeline, offering a novel approach to domain generalization.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well-organized, with a clear and intuitive research motivation. It provides comprehensive theoretical analysis, and the proposed MaskTwins framework is validated through extensive experimental results.\n\nWeaknesses:\n1. Although the paper offers thorough theoretical analysis, the proposed MaskTwins framework lacks architectural innovation. The ideas related to Masked Image Modeling (MIM) are not novel.\n2. The ablation study in Table 4 is not comprehensive. Ablation experiments should control for single variables, but the current design lacks specificity.\n3. The formatting of Table 4 and Table 5 is inconsistent with other tables in the paper, such as the use of bold text with a gray background.", "other_comments_or_suggestions": "Some of the statements in this paper need refinement. For example, the first contribution in the introduction appears to be an overclaim, and the subsequent analysis lacks sufficient detail.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xoLmYT6SNk", "reviewer": "Reviewer_DCWw", "summary": "This paper introduces a complementary masking strategy for the semantic segmentation UDA task. Building on the existing masked image consistency (MIC) training paradigm, which relies on pseudo-labels from the teacher model, the authors propose a complementary masking loss to further enforce consistent predictions on complementary masked images. Additionally, the paper provides a theoretical analysis of the complementary masking strategy. Experiments demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and comments. The main concerns are addressed below. \n\n> **W1:** Misunderstanding on Complementary Masking Techniques\n\nThank you for your comments again. But we respectfully cannot agree with the justifications here, and would like to further clarify as below:\n\n- First, the proposed complementary masking is not limited to \"RGB images alone\"; it can also be applied to biological datasets that contain gray-scale images. \n\n- Meanwhile, we highlight our theoretical contributions, which are an indispensable part of our work and are entirely lacking in existing works. Specifically, we have provided a theoretical foundation for masked reconstruction from a perspective of sparse signal reconstruction problem and have rigorously analyzed the properties of complementary masking from three aspects: information preservation, generalization bound, and feature consistency.\n\n---\n\n> **Q1:** Explanation on the gain of random masking over MIC\n\nThe reason why random masking achieves a certain improvement compared to MIC (75.2 mIoU vs. 74.0 mIoU) is the addition of the dual-form masking consistency, while MIC only has a single masked image. Mask itself is indeed of great significance. However, the improvement brought about in this way is limited, and having more branches of masked images will also lead to the problem of redundancy. In our experiments, the performance of a single masked branch reached 74.0 mIoU, that of two branches improved to 75.2 mIoU and using three or more branches will not yield further gains.\n\nIn contrast, the complementary masking can achieve significantly higher improvement by only changing the masking strategy (76.7 mIoU vs. 75.2 mIoU). This further validates the substantial performance boost brought about by the complementary masking and the total gain does not rely on the masking strategy itself.\n\n---\n\n> **Q2:** The performance of complementary masking strategy in the feature space\n\nWe are highly intrigued by the ideas you proposed, and as a result, we conducted the following experiments in Table R1. Since applying masking during the decoder phase may significantly impact the already-extracted high-level semantic information and lead to a decline in model performance, we perform masking on the four layers of encoded features (where 1 represents the shallowest layer and 4 represents the deepest layer) respectively.\n\n\n---\nTable R1. The performance of complementary masking strategy in the feature space on SYNTHIA→Cityscapes.\n\n| Complementary Masking         | mIoU |\n| ----------------------------- | ---- |\n| image level                   | 76.7 |\n| feature level: 1 (shallowest) | 76.0 |\n| feature level: 2              | 74.6 |\n| feature level: 3              | 72.4 |\n| feature level: 4 (deepest)    | 67.7 |\n---\n\nIt is worth noting that the feature-level masking are mainly utilized in generative tasks, such as in MaskGit [1]. According to Assumption 1 in the main paper, latent-level masking may result in more information loss. Therefore, it is not suitable for comprehension tasks, for example, image segmentation. \n\nThe above results also confirm this. The deepest feature layer contains the most semantic information and applying complementary masking to this layer will result in the most substantial loss of valuable details, leading to the most significant drop in performance.\n\nRegarding the additional experiments on feature-level masking, we appreciate your interest and will include these results in the supplementary materials of the revised paper.\n\n---\n\n**Reference**\n\n[1] Chang H, Zhang H, Jiang L, et al. Maskgit: Masked generative image transformer[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 11315-11325.", "claims_and_evidence": "Yes, the paper mainly proposes a complementary masking strategy for input images and provides a theoretical analysis of this approach.", "methods_and_evaluation_criteria": "This paper mainly introduces a complementary masking loss to encourage consistent predictions on complementary masked images. The model architecture follows the baseline MIC, consisting of a student and an EMA teacher. The masked consistency learning loss also follows the MIC loss from the baseline MIC (Hoyer et al., 2023), which uses pseudo-labels of the complete target image from the teacher for supervision. Notably, complementary masking strategies have already been explored in existing multi-modal works, both at the input and feature levels (e.g., Shin et al., 2024; Yang et al., 2025). Therefore, the primary contributions of this paper are its theoretical analysis and experimental validation, which demonstrate the effectiveness of the complementary masking strategy for RGB images alone. However, the method lacks sufficient novelty and insight.", "theoretical_claims": "I have reviewed the theoretical claims.", "experimental_designs_or_analyses": "In Table 4, although the ablation study is not directly discussed against the baseline MIC, Table 1 shows that simply replacing complementary masking with random masking (Row 3 in Table 4) already outperforms MIC (75.2 mIoU vs. 74.0 mIoU). This indicates that the performance gains in this paper heavily rely on the masking strategy itself, even when using random masking. Therefore, while the paper reports state-of-the-art results, these improvements are less convincing. Moreover, the authors do not provide an explanation for why masking strategies, including random ones, lead to such significant gains.", "relation_to_broader_scientific_literature": "1. Model architecture: Follows the baseline MIC, consisting of a student and an EMA teacher.\n2. Masked consistency learning loss: Directly follows the MIC loss from the baseline (Hoyer et al., 2023), using pseudo-labels of the complete target image from the teacher for supervision.\n3. Complementary masking strategy: Already explored in existing multi-modal works at both input and feature levels (e.g., Shin et al., 2024; Yang et al., 2025).", "essential_references_not_discussed": "I think this paper has included enough related works to provide a clear understanding", "other_strengths_and_weaknesses": "Advantages:\n\n1.The paper is well-written and clear.\n2. The experiments are sufficient to evaluate the proposed method.\n3. The paper provides detailed theoretical analysis and proofs.\n\nDisadvantages:\n1. As mentioned earlier, the method largely builds on strategies from existing baselines, with only minor modifications to achieve good performance, but lacks novel or insightful contributions.  I have another key question: how this masking strategy would perform in feature space, as both input and feature masking have been studied in multi-modal tasks. Similar concerns apply to the theoretical analysis.", "other_comments_or_suggestions": "No other comments", "questions_for_authors": "As mentioned in previous comments:\n1. Explain why random masking alone can achieve state-of-the-art performance.\n2. Discuss how the complementary masking strategy would perform in the feature space in this single-modal RGB setting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nG34OqNQEF", "reviewer": "Reviewer_vim2", "summary": "In this paper, the authors introduce MaskTwins, a UDA framework that integrates masked reconstruction into the main training pipeline. They argue that existing UDA methods leveraging masked image modeling treat masking merely as a form of input deformation and lack theoretical analysis, resulting in a superficial understanding and underutilization of its potential for feature extraction and representation learning. To address this, the authors reframe masked reconstruction as a sparse signal reconstruction problem and theoretically demonstrate that the dual form of complementary masks enhances the extraction of domain-agnostic image features. Experimental results on both natural and biological images show that MaskTwins improves domain generalization by enforcing consistency between predictions from complementary masked images, uncovering intrinsic structural patterns that persist across domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and comments. The main concerns are addressed below. \n\n> **W1:** Application on more challenging segmentation datasets\n\nThank you for the concerns on the generalizability of the proposed method. \n\nOn the one hand, the SYNTHIA dataset we use is of a relatively large scale, boasting over 20,000 synthetic images with diverse urban scene representations. As for the \"SYNTHIA→Cityscapes\" task, it has been extensively studied in domain adaptation [1], with a domain gap between synthetic and real-world data. On the other hand, biological image segmentation is commonly acknowledged to exhibit strong domain shifts between various domains.\n\nStill, we agree that extending our method to more challenging segmentation dataset can further validate its effectiveness. To address this, we conducted additional experiments on the RIM-ONE-r3 dataset for fundus optic disc (OD) and cup (OC) segmentation.\n\n---\n\nTable R1. Quantitative results of OD and OC segmentation on the RIM-ONE-r3 dataset with a CBMT backbone, using only the proposed complementary masking strategy.\n\n|            |  $Dice_{OD}$↑  |   $ASSD_{OD}$↓    |  $Dice_{OC}$↑   |   $ASSD_{OC}$↓    |\n| ---------- | :--------------: | :-------------: | :--------------: | :-------------: |\n| EOAPNet |   92.61(±3.13)   |   6.67(±2.91)   |  74.59(±25.64)   |   8.74(±5.34)   |\n| CBMT[2] |   93.36(±4.07)   |   6.20(±4.79)   |  81.16(±14.71)   |   8.37(±6.99)   |\n| **Ours**   | **94.17(±2.48)** | **5.15(±2.14)** | **82.74(±9.13)** | **7.52(±6.33)** |\n\nThe results show that our method improves the performance of OD and OC segmentation by 0.81%(1.58%) Dice and 1.05(0.85) ASSD, respectively.\n\n---\n\n>**W2:** Connection with semi-supervised methods\n\nWhile we acknowledge the similarities between our method and semi-supervised image segmentation, they differ in task settings. Semi-supervised methods require labeled target data, whereas our focus is on unsupervised domain adaptation, especially under significant domain gaps. In the absence of labeled data, image-level masking plays a crucial role, and experiments focusing on this better validate our theoretical analysis. \n\nStill, we appreciate your interests and we will briefly discuss the connection with semi-supervised methods in the revised paper to strengthen the comprehensiveness and robustness of our research.\n\n---\n\n>**Q1:** Gain of complementary masks and term ratios in formulas\n\nIntuitively, the generalization bound for random masking includes an additional term compared to complementary masking. While this extra term complicates the bound, complementary masking significantly tightens the generalization bound, leading to better performance across various scenarios.\n\nDirectly quantifying this improvement from the formulas is complex due to multiple terms, but experimental results provide clarity. As shown in Table 1, MaskTwins outperforms MIC by +2.7 mIoU, demonstrating the effectiveness of our approach. Additionally, complementary masks contribute a +1.5 mIoU improvement over random masks (Table 4), confirming their value.\nRegarding the ratio of terms in the bounds, we observe that the additional term in random masking is influenced by the masking strategy. Complementary masking not only improves information preservation but also reduces generalization error variance, as stated in Theorem 4 of the supplementary material.\n\nFinally, \"SM\" does not appear in the paper, and we speculate the reviewer refers to the generalization bounds section, which discusses the benefits of complementary masking. The theory shows that complementary masking offers a tighter generalization bound compared to random masking.\n\n---\n\n> **Q2:** Statistical results \n\nWe provide the mean and standard deviation of the experiments as follows. The results show that the standard deviation of the metrics does not vary significantly with the change in the mask ratio.\n\n\nTable R2. Statistical results based on Table 5(a). \"±\" refers to the standard deviation over 3 random seeds.\n\n| Mask Ratio | mIoU       |\n| ---------- | ---------- |\n| 0.1        | 72.0(±0.2) |\n| 0.2        | 74.6(±0.3) |\n| 0.3        | 75.4(±0.2) |\n| 0.4        | 76.5(±0.2) |\n| 0.5        | 76.7(±0.2) |\n\n---\n\n> **Q3:** Question on architectures\n\nThe proposed complementary masking strategy certainly works on different architectures and we highlight that it does not rely on the network structure or additional internal components. Specifically, for natural and biological image segmentation, we use a Transformer network and a CNN network respectively, following the corresponding pipelines in [1]. The details of the architectures in the experiments can be found in Appendix F.\n\n**Reference**\n\n[1] Hoyer, Lukas, et al. \"MIC: Masked image consistency for context-enhanced domain adaptation.\" CVPR 2023.\n\n\n[2] Tang L, Li K, He C, et al. Source-free domain adaptive fundus image segmentation with class-balanced mean teacher, MICCAI 2023.", "claims_and_evidence": "Most claims are valid. However, the authors state that they demonstrate the superiority of the proposed approach through extensive experiments. I disagree, as the experiments were conducted on a few relatively small datasets, and the domain gap between the source and target domains appears limited. This raises concerns about the generalizability of the proposed method.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are generally reasonable. However, their effectiveness remains to be evaluated on larger datasets with more significant source-target domain gaps to better assess their real-world applicability.", "theoretical_claims": "I briefly reviewed the proofs but did not conduct a thorough verification, so I cannot confirm their correctness.", "experimental_designs_or_analyses": "I reviewed the experimental design and found it generally reasonable. However, the small dataset sizes and limited source-target domain gap raise concerns about the generalizability of the results. Further evaluation on larger and more diverse datasets is needed to confirm the method’s effectiveness.", "relation_to_broader_scientific_literature": "This work builds upon Hoyer et al.’s approach, which masks the target image and evaluates consistency between predictions from images with and without masking. It advances UDA by introducing a complementary masking strategy and providing a formal theoretical analysis to support the proposed method. The complementary masking strategy has been shown to improve domain generalization. Furthermore, the method closely aligns with semi-supervised image segmentation, differing mainly by incorporating an additional consistency loss that quantifies the discrepancy between predictions from masked target-domain images. While semi-supervised learning often assumes that labeled and unlabeled datasets originate from the same or similar domains, I argue that this assumption is not strictly necessary for semi-supervised methods to be effective. Therefore, the connection to semi-supervised image segmentation should be discussed more explicitly.", "essential_references_not_discussed": "Given the similarities between the proposed method and semi-supervised image segmentation, relevant works in semi-supervised image segmentation should be reviewed, and their connections and differences should be explicitly discussed.", "other_strengths_and_weaknesses": "In addition to domain adaptation methods, I think it would be better if some semi-supervised learning methods can be used for comparison.", "other_comments_or_suggestions": "I have no other comments or suggestions.", "questions_for_authors": "The paper is clearly written. I think I only have the following questions:\n\n1. Complementary masks provide tighter bounds, but how significant is the improvement? What is the ratio between the first and second terms in the bound shown in the SM for random masks, using typical values for the relevant symbols?\n\n2. The results indicate that segmentation performance improves as the mask ratio increases from 0.1 to 0.5. Are these differences statistically significant?\n\n3. Can the same conclusions hold across different architectures, such as CNNs, Transformers, and hybrid networks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxrieWqns1", "reviewer": "Reviewer_7ChN", "summary": "This paper tackles image segmentation within the unsupervised domain adaptation (UDA) framework. Rather than employing random masks, the authors develop dual-form complementary masked images to strengthen the generalization capabilities of their approach. They demonstrate that robust, domain-invariant features can be extracted through consistency learning applied to these masked images. The method's effectiveness is validated through comprehensive experiments across six distinct test cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and comments. The main concerns are addressed below. \n\n> **W1:** Complementary Mask-only framework\n\nThe proposed complementary masking strategy certainly works on the Complementary Mask-only framework and we highlight that it does not rely on the network structure or additional internal components. \n\nSpecifically, AdaIN is a classic component used in domain adaptation and EMA commonly serves for teacher-student structure. We adopt the EMA teacher model for pseudo-label generation, following common practices in UDA methods [1]. These components are simple and require no special hyper-parameter tuning. In fact, the corresponding contributions are quite limited according to Table 4.\n\nConsidering the misunderstanding caused by, we have supplemented Table 4 with the results of Complementary Mask-only framework as follows. The results show that remarkable gains can be achieved by using the complementary mask alone. We highly agree that this can better demonstrate the effectiveness of the proposed complementary mask framework, and we will include this in the revised paper.\n\n---\n\nTable R1. Supplementary ablation study based on Table 4.\n\n|  CL  | CMask | RMask | EMA  | AdaIN | mIoU |\n| :--: | :---: | :---: | :--: | :---: | :--: |\n|  √   |   √   |   -   |  -   |   -   | 76.0 |\n---\n\n> **W2:** Question on theorem\n\nThe assumption of sparse signal in Assumption 1 is a general background.  This assumption serves as the foundation for our theoretical analysis. Although we use the image (X) in the formula without decomposing it into the sparse signal (S) and the noise (N), it holds true completely in both the analysis of the generalization bound and that of the feature consistency. This connection is implicit and more importantly, intrinsic. In our analysis, the generalization bound focuses on the theoretical explanation of mask reconstruction, while feature consistency is applied to account for the setting of domain adaptation.\n\nMeanwhile, we explicitly use Assumption 1 in the proofs of Theorem 5 (Signal Recovery Guarantee) in the supplementary materials. Specifically, we reframe X as a signal generated from the sparse linear model, and apply Compressed Sensing Recovery Guarantees during the proofs.\n\n> **W3:** Application on more challenging segmentation dataset\n\nWhile our main experiments have comprehensively covered both natural images and medical images, we agree that extending our method to fundus segmentation tasks can further validate its effectiveness. To address this, we conducted additional experiments on the RIM-ONE-r3 dataset.\n\n---\n\nTable R2. Quantitative results of OD and OC segmentation on the RIM-ONE-r3 dataset with a CBMT backbone, using only the proposed complementary masking strategy.\n\n|            | $Dice_{OD}$↑  |  $ASSD_{OD}$↓   | $Dice_{OC}$↑  |  $ASSD_{OC}$↓   |\n| ---------- | :--------------: | :-------------: | :--------------: | :-------------: |\n| EOAPNet |   92.61(±3.13)   |   6.67(±2.91)   |  74.59(±25.64)   |   8.74(±5.34)   |\n| CBMT    |   93.36(±4.07)   |   6.20(±4.79)   |  81.16(±14.71)   |   8.37(±6.99)   |\n| **Ours**   | **94.17(±2.48)** | **5.15(±2.14)** | **82.74(±9.13)** | **7.52(±6.33)** |\n\nThe results show that our method improves the performance of OD and OC segmentation by 0.81%(1.58%) Dice and 1.05(0.85) ASSD, respectively. Regarding the additional experiments on fundus image segmentation, we appreciate your interest and we will include these results in the supplementary materials of the revised paper.\n\n---\n\n> **W4:** Feature visualizations\n\nAs suggested, we provide some feature visualizations compared with the random masking strategy to more comprehensively demonstrate the effectiveness of the proposed complementary masking strategy. We provide the anonymous links for online viewing.\n\nFor the direct comparison with random mask scheme, Figure R1 is: https://picx.zhimg.com/80/v2-c3ade1e071de958d79964bbb69c3564f.png. The input, feature and segmentation results are presented in dual-form. 'CMask' means using the proposed complementary masking strategy while 'RMask' means random masking.\n\nThis is the results of the model trained with complementary masking strategy: https://picx.zhimg.com/80/v2-b6a712296ffc45377c54284a2ffb029b.png (Figure R2, last layer) and https://picx.zhimg.com/80/v2-603fcc5b6d192413c8ec988c28dffbc8.png (Figure R3, middle layer); this is the rusults of using random masking: https://picx.zhimg.com/80/v2-655ba0ffed4cbe7ad9168ad1e5f66e44.png (Figure R4, last layer) and https://picx.zhimg.com/80/v2-2957e2cca93f1cb93c05c95020c298e4.png (Figure R5, middle layer).\n\n---\n\nOnce again, thanks for your time and comments. We hope these concerns can be adequately addressed through the above explanations.\n\n---\n\n**Reference**\n\n[1] Hoyer, Lukas, et al. \"MIC: Masked image consistency for context-enhanced domain adaptation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.", "claims_and_evidence": "To comprehensively understand the effectiveness of the proposed Complementary Mask, I recommend the authors can conduct experiments on some more challenging segmentation datasets. For example, the medical image segmentation has strong domain shifts across domains. The fundus segmentation tasks such as OC and OD segmentation is a good choice.", "methods_and_evaluation_criteria": "Since the overall framework of this paper relies on multiple component e.g., the AdaIN and the EMA-based pseudo labels, it seems to be over-complicated and is not elegant enough from a perspective of ML community. Meanwhile, according to the ablation study, it seems that the proposed Complementary Mask cannot work on Complementary Mask-only framework? This eliminates the effectiveness of the proposed Complementary Mask. I suggest the authors can report the result of a Complementary Mask-only framework to demonstrate the efficacy of the proposed Complementary Mask framework.", "theoretical_claims": "To be honest, although I try to understand the envisioned Theorems, I find that there is not a directly related connection between the sparse signal reconstruction in Assumption 1 and corresponding generalization bound as well as feature consistency. Therefore, I do not know how the actually sparse signal reconstruction theory is utilized? It seems that envision Theorems are not directly link to the claimed contribution 1.", "experimental_designs_or_analyses": "I appreciate nice numerical results, but it is also necessary for the reader to vividly understand why the proposed Complementary Mask works. I suggest some feature visualizations may be added to compare with random mask scheme.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- This paper introduces the Complementary Mask strategy based on random mask, the empirical experiments look good.\n- This paper is easy to follow and well-written.\n- The authors provide theoretical analysis to justify the proposed complementary mask related to the sparse signal reconstruction.\n\nWeakness:\n\n- Since the overall framework of this paper relies on multiple component e.g., the AdaIN and the EMA-based pseudo labels, it seems to be over-complicated and is not elegant enough from a perspective of ML community. Meanwhile, according to the ablation study, it seems that the proposed Complementary Mask cannot work on Complementary Mask-only framework? This eliminates the effectiveness of the proposed Complementary Mask. I suggest the authors can report the result of a Complementary Mask-only framework to demonstrate the efficacy of the proposed Complementary Mask framework. \n\n- To be honest, although I try to understand the envisioned Theorems, I find that there is not a directly related connection between the sparse signal reconstruction in Assumption 1 and corresponding generalization bound as well as feature consistency. Therefore, I do not know how the actually sparse signal reconstruction theory is utilized? It seems that envision Theorems are not directly link to the claimed contribution 1.\n\n- To comprehensively understand the effectiveness of the proposed Complementary Mask, I recommend the authors can conduct experiments on some more challenging segmentation datasets. For example, the medical image segmentation has strong domain shifts across domains. The fundus segmentation tasks such as OC and OD segmentation is a good choice.\n\n- I appreciate nice numerical results, but it is also necessary for the reader to vividly understand why the proposed Complementary Mask works. I suggest some feature visualizations may be added to compare with random mask scheme.\n\n=====================================================================================\nThanks for authors' responses that have addressed my concerns. I have raised my score.", "other_comments_or_suggestions": "Please see weakness", "questions_for_authors": "Please see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Dear authors,

Thank you for submitting draft for review. This draft has received 4 weak accept. One reviewer improved from weak reject to weak accept after rebuttal. However, none of the reviewer is interested in assigning accept rating.  One reviewer pointed out concern regarding whether it can handle considerable domain shift. Method itself is quite straightforward,  however,  notion that just use of complimentary masks can result in this considerable improvement is quiet interesting. We will encourage authors to include all the points discussed during rebuttal in the final copy, especially the updated ablation studies. 



regards
AC
Decision: Accept (poster)