CAKE: CAusal and collaborative proxy-tasKs lEarning for Semi-Supervised Domain Adaptation
Authors: Wenqiao Zhang
CHANGSHUO LIU
Can Cui
Beng Chin Ooi
Venue: Submitted to ICLR 2023 (2023)

Abstract: Semi-supervised domain adaptation (SSDA) adapts a learner to a new domain by effectively utilizing source domain data and a few labeled target samples. It is a practical yet under-investigated research topic. In this paper, we analyze the SSDA problem from two perspectives that have previously been overlooked, and correspondingly decompose it into two \emph{key subproblems}: \emph{robust domain adaptation (DA) learning} and \emph{maximal cross-domain data utilization}.  \textbf{(i)}  From a causal theoretical view, a robust DA model should distinguish the invariant ``concept'' (key clue to image label) from the nuisance of confounding factors across domains. To achieve this goal, we propose to generate \emph{concept-invariant samples} to enable the model to classify the samples through causal intervention, yielding improved generalization guarantees; \textbf{(ii)} Based on the robust DA theory, we aim to exploit the maximal utilization of rich source domain data and a few labeled target samples to boost SSDA further. Consequently, we propose a collaboratively debiasing learning framework that utilizes two complementary semi-supervised learning (SSL) classifiers to mutually exchange their unbiased knowledge, which helps unleash the potential of source and target domain training data, thereby producing more convincing pseudo-labels. Such obtained labels facilitate cross-domain feature alignment and duly improve the invariant concept learning. In our experimental study, we show that the proposed model significantly outperforms SOTA methods in terms of effectiveness and generalisability on SSDA datasets.

Under review as a conference paper at ICLR 2023
CAKE: CAUSAL AND COLLABORATIVE PROXY-TASKS LEARNING FOR SEMI- DOMAIN ADAPTA- TION
Anonymous authors Paper under double-blind review
ABSTRACT
cemi-supervised domain adaptation (SSDA) adapts learner to a new domain by effectively utilizing source domain data and a few labeled target samples. It is a practical yet under-investigated research topic. In this paper, we analyze the SSDA problem from two perspectives that have previously been overlooked, and correspondingly decompose it into two key subproblems: robust domain adaptation (DA) learning and maximal cross-domain data utilization. (i) From causal theoretical view, robust DA model should distinguish the invariant "concept" (key clue to image label) from the nuisance of confounding factors across domains. To achieve this goal, we propose to generate concept-invarica samples to enable the model to classify the samples through causal intervention, yielding improved generalization guarantees; (ii) Based on the robust DA theory, we aim to exploit the maximal utilization of rich source domain data and a few labeled target samples to boost SSDA further. Consequently, we propose a collaboratively debiasing learning framework that utilizes two complementary semi-supervised learning (SSL) classifiers to mutually exchange their unbiased knowledge, which helps unleash the potential of source and target domain training data, thereby producing more convincing pseudo-labels. Such obtained labels facilitate cross- domain feature alignment and duly improve the invariant concept learning. In our experimental study, we show that the proposed model significantly outperforms SOTA methods in terms of effectiveness and generalisability on SSDA datasets.
INTRODUCTION
Domain Adaptation (DA) aims to transfer training knowledge to the new domain (targer D DT)
using the labeled data available from the original domain (source D = =Ds), which can alleviate the poor generalization of learned deep neural networks when the data distribution significantly
deviates from the original domain Wang & Deng (2018); You et al. (2019); ; Tzeng et al. (2017). In the DA community, recent works Saito et al. (2019) have shown that the presence of few labeled data from the target domain can significantly boost the performance of deep learning- based models. This observation led to the formulation of Semi-Supervised Domain Adaptation (SSDA), which is a variant of Unsupervised Domain Adaptation (UDA) Venkateswara et al. (2017) to facilitate model training with rich labels from Ds and few labeled samples from Dy. For the fact that we can easily collect such additional labels on the target data in real-world applications, SSDA has the potential to render the adaptation problem more practical and promising in comparison to UDA.
Broadly. most contemporary approaches Ganin al. (2016); [Jiang et al. (2020): Kim & Kim (2020); ;
Yoon et al. (2022) handle the SSDA task based on two domain shift assumptions, where X' and D respectively denote the samples and their corresponding labels: (i) Covariate Shift, P(X Ds) * PD = DT); (ii) Conditional Shift, P()'XX'' D Ds) A P() DT). refers to the difference of conditional label distributions of cross-domain data. Intuitively, one straightforward solution for SSDA is to learn the common features to mitigate the domain shift issues. Further quantitative analyses, however, indicate that the model trained with supervision on few labeled target samples and labeled source data can just ensure partial cross-domain feature alignment Kim & Kim (2020). That is, it only aligns the features of labeled target samples and their correlated nearby samples with the corresponding feature clusters in the source domain.

---

Under review as a conference paper at ICLR 2023
Irter-doman
BSDA
Source Domain D,
Target Domain D2
dats
Invariant Concepts
Clansieer A
deg
eaterg to get 0022
domale
bo
Pottyle eliport D2) i P(Style 1D =D)
By Labelad Data
Domsin Ditribution Boundary
P(Canceppt P'Cunrepe dog
Dy Labsled Data
Classitiction Deciioo Boundary
D, Unlabeted Data
Invar ant Causal Factors
(a) Cross Domain Adaptation
(b) Class-wise Distribution
(c) Causal and collaborative tasks learning
Figure 1: (a) Four DA cases ("Clipart" + "Real"). (b) Class-wise distribution of source domain and target domain. (c) A simplified version that indicates how our proposed model facilitates the SSDA.
To systematically study the SSDA problem, we begin by asking two fundamental questions, Q1:What properties should robust DA model have? To answer this question, we first present DA example in Figure [a), which suggests that the image "style" in D = DDT is drastically different from the D = Ds. A classifier trained on the source domain may fail to predict correct labels even though the "ccncept" (e.g.. plane) is invariant with a similar outline. The truth is that the minimalist style features being invariant in "clipart" domain plays critical factor in the trained classifier, which may consequently downplay the concept features simply because they are not as invariant as style features. Importantly, such observation reveals the fundamental reason of the two domain shift assumptions, Le., P(Style = clipart D = Ds) # P(Style = real DD = Dr). Therefore, a robust DA model needs to distinguish the invariant concept features in X across domains from the changing style. Q2: How to maximally exploit the target domain supervision for robust SSDA? As discussed, supervised learning on the few target labels cannot guarantee the global cross-domain feature alignment. which hurts the model generalization for invariant learning. A commonly known approach in this few labeled setting, semi-supervised learning (SSL), uses a trained model on labeled data to predict convincing pseudo-labels for the unlabeled data. This approach relies on the ideal assumption that the labeled and unlabeled data have the same marginal distribution of label over classes to generate pseudo-labels. However, Figure [bb) indicates these distributions are different in both inter-domain and intra-domain. This may result in the imperfect label prediction that causes the well-known confirmation bias Arazo et al. (2020), affecting the model feature alignment capability. Further, in the SSDA setting, we have three sets of data, i.e., source domain data, labeled and unlabeled target domain data. One single model for SSDA may be hard generalize to the three sets with different label distributions. Thus, the premise of better utilization of labeled target samples is to mitigate undesirable bias and reasonably utilize the multiple sets. Summing up, these limitations call for reexamination of SSDA and its solutions.
To alleviate the aforementioned limitations, we propose a framework called CAusal collaborative proxy-tasKs |Earning (CAKE) which is illustrated in Figure ([).. In the first step, we formalize the DA task using a causal graph. Then leveraging causal tools, we identify the "style" as the confounder and derive the invariant concepts across domains. In the subsequent steps, we build two classifiers based on the invariant concept to utilize rich information from cross-domain data for better SSDA. In this way. CAKE explicitly decomposes the SSDA into two proxy subroutines, namely Invariant Concept Learning Proxy (ICL) and Collaboratively Debiasing Learning Proxy (CDL). In ICL, we identify the key to robust DA is that the underlying concepts are consistent across domains, and the confounder is the style that prevents the model from learning the invariant concept (C) for accurate DA. Therefore, a robust DA model should be an invariant predictor P()) D = DT) = &( R.D- Ds)) under causal interventions. To address the problem, we devise a causal factor generator (CFG) that can produce concept-invarinn
samples & with different style to facilitate the DA model to effectively learn the invariant concept. As such, our ICL may be regarded as an improved version of Invariant Risk Minimization (IRM) Arjovsky et al. (2019) for SSDA, which equips the model with the ability to learn the concept features that are invariant to styles. In CDL, with the invariant concept learning as the foundation, we aim to unleash the potential of three sets of cross domainn data for better SSDA. Specifically, we build two correlating and complementary based semi- superiseee learning (SSL) classifiers for Ds and Dy with self selff-enalizatio.. These two classifiers ensure that the mutual knowledge is exchanged expand the number of "labeled" samples
labuli-a
2

---

Under review as a conference paper at ICLR 2023
in the target domain, thereby bridging the feature distribution gap. Further, reduce the confirmation bias learned from respective labeled data, we adopt Inverse Propensity Weighting (IPW) Glynn & Quinn 2010) theory which aims to force the model to pay same attention to popular ones and tail ones in SSL models. Specifically, we use the prior knowledge of marginal distribution to adjust the optimization objective from P()IA))) to P(XI)) (Maximizing the probability of each
X with different y E )) for unbiased learning. Thus, the negative impact caused by label distribution shift can be mitigated. Consequently, the two subroutines mutually boost each other with respect their common goal for better SSDA.
(1) We formalize the DA problem using causality and propose the explicitly invariant concept learning paradigm for robust DA. (2) To unleash the power of cross- domain data, we develop a collaboratively debiasing learning framework that effectively reduces the domain gap to enforce invariant prediction. (3) We extensively evaluate the proposed CAKE. The empirical results show that it outperforms SOTA approaches on the commonly used benchmarks.
Our contributions are three-fold:
DOMAIN ADAPTATION THROUGH CAUSAL LENSES: FINDING THE DEVIL
We shall start by grounding the domain adaptation (DA) in a causal framework to illustrate the key challenges of cross-domain general- ization. As discussed in introduction, given data X' and their labels ), the main difficulty of DA is that the extracted representation from X' is no longer a strong visual cue for sample label in another domain. To study this issue in-depth, first make the following assumption:
Causal Relationship Spurious Relationship
D
X
y
SD
Se
e
Assumption 1 (Disentangled Variables). Data X can be disentan- gled into concept C, cross-domain style Se and intra-domain style S₁ variables which are mutually independent, i.e., X' = (C, Se, Sz), where C -LL Se IL Sz. Only concept C is relevant for the true label y of X. ie.. style changing is concept-preservic
Decomposition variables of X'
D
X
y
Under this assumption, we abstract the DA problem into a causal graph (Figure P In this figure, 'D represents the Domain (e.g., Ds or Dr). while Sz (e.g., different appearance of concept in same domain) and Se (e.g., different background of concept cross-domain) are the nuisance variables that confound changing is irrelevant for true label y. C is the invariant concept which contains directly causal relationships with Y. Therefore, the causal graph reveals the fundamental reasons for distinguishing issues across domains, i.e. the cross/intra-oomann style serves as the confounding variables that influence the X - y.
5c
e
n
Decomposition variables of
V. The absence of any style
Figure 2: Causal graph of DA.
P(YCC, D = Ds) = P(VCC, D = Dr) and P(Y|S, D Ds) & P()SS, D = DT) (1)
P()|&, D = + + P(VXX Dr). VS S2)
The "devil" for DA problem could be style confounders Se and Sz in that they prevent the model from
learning the concept-invarian causality X + From the causal theoretical view, such confounding effect can be eliminated by statistical learning with causal intervention Pearl et al. (2000) Putting all these observations together, we now state the main theorem of the paper.
Theorem 1 (Causal Intervention). Under the causal graph in Figure 2 and Assumption 1, we can conclude that under this causal model, performing interventions on Se and St does not change the P(Y)X)) Thus, in DA problem, the causal effect P()]do(X)) D= DDDT) can be computed as:
P())do(X), D = =Dr) P(y|do (c)) D DDD
ΣE>
Σ
Σ
Σ
Σ
Disentangled Variables
DE((s,DD) SemiSc iz~Ss
P(VCC, S7, D)P(C Sc, SI D) ~ Σ ( X
While this assumption may be in all settings, we believe that the single image classification can be approximated by this assumption. More discussion about this assumption is in the appendix.
P(())do((2) D = Dr) uses the do-operator Glymour et al. (2016). Given random variables X, )) we write P()' = y X =)) indicate the probability of Y y when intervene and set X' to he zr.
3

---

Under review as a conference paper at ICLR 2023
where &' are the invariant causal factors with the same concepts of X but contain different cross/intra- domain styles, Le., invariant concept-aware samples. Realistically. & is often large set due the multiple style combinations. This may block the model's computational efficiency according to Eq. 2 and hard to obtain such numerous causal factors. However, it is non-trivial to personally determine the & size to study the deconfounded effect. We employ compromise solution that significantly reduces the X size small number for causal intervention.
CAKE: CAUSAL AND COLLABORATIVE PROXY-TASKS LEARNING
This section describes the CAKE for Supi--Suerviied Domain Adaptation (SSDA) based on the studied causal and collaborative learning. We shall present each module and its training strategy.
3.1 PROBLEM FORMULATION
In the problem of SSDA, we have access to a set of labeled samples S = ((x1) Val 2 source domain Ds. And the goal of SSDA is adapt learner a target domain Dr, of which the training set consists of two sets of data: a set of unlabeled data Ta = {(z J.V. and a small labeled have Ni Nv, and Ní < N, We solve the problem by decomposing the SSDA task into two proxy subroutines: Invariant Concept I Learning (ICL) and Collaboratively Debiasing Learning (CDL). Such subroutines are designed seek robust learner M(-;6) () which performs well on test data from the target domain:
i.i.d from
set Ti (( Vet
(i) (8) y Typically,
M(: (Oz, Oc)) (Z, Fú, (S,, Tup Oz) Mc((Tpl(S) Tú, Tt, Tu, Tw)); Oc)
Learner CAKE
ICL Proxy Subroutine
CDL Proxy Subroutine
where Mr and Mc indicate the ICL model parameterized by θ₂ and the CDL model parameterized by Oc respectively. In ICI. proxy, Mr(:; θr) learns the causal factors (81, Ti, TT)) for Ds and D in unsupervised learning paradigm, aiming to generate the invariant causal factors and use Eq. 2 to remove the confounding effect. CDL aspect, we construct two pseudo labeling-
SSL techniques: ()) + Tu and (Ti, TT) - Tu. aiming at utilizing all the training data possible to bridge the feature discrepancy under the premise of invariant concept learning.
3.2 INVARIANT CONCEPT LEARNING PROXY
As we discussed Section 2 the key to robust DA to eliminate the spurious correlations between styles (Se and S7) and label y. To tackle this problem, we propose an approximate solution to kindly remove the confounding effect induced by Sc and Sr. In detail, we develop the two invariant causal factor generators that can produce the causal factors X with C. Next, we propose the Invariant Concept Learning (ICL) loss function, which forces the backbone (e.g., ResNet-34 He al. (0166) to focus on learning concepts that are invariant across a set of domains.
3.2.1 INVARIANT CAUSAL FACTOR GENERATOR
Achieving the invariant concept-aware X is challenging due to the fact that supervised signals are missing or expensive to obtain. Thus, we resort to the unsupervised learning paradigm. designing two causal factor generators (-=(+)= (-) (cross-domain) and Cf (-) (intra-domain) to achieve X for Ds and DD without the reliance on the supervised signals. Take D = Ds as an example, the invariant causal factors of S, is given by S = ($), S ( cf) w.r.t Se and Sy:
Cross-domain Causal Factor. S are generated by N₂ GAN-based techniques Creswell al. (2018), enabling the source concept to be preserved during the cross-domain conversion process. By considering the huge domain discrepancy, we optimize the style transfer loss as follows:
min max Lk (r)) Pe [log DJ (x+) + log(1 - (** G5, Dj
+ Lb, (Xat, Fz; 0r) + Ch (Tat, Nei OF)], k- argmin C.((;Or), (iEl,-,NN))
4

---

Under review as a conference paper at ICLR 2023
where |::|] represents the union of two imputs, D, is the discriminator distinguish the original source of the latent vector if from Dy. Caye and Lide are the cycle and identity loss Zhu et al. (2017). Gh 4thh Cd" Through min-max adversarial training, the domain style-changing samples are obtained.
Intra-domain Causal Factor. We utilize the image augmentations intra-domain style interven- tions, e.g., modifying color temperature. brightness, and sharpness. We randomly adjust these image properties as our mapping function to change the intra-domain style for Ds with invariant concept.
Thus, the invariant causal factors S ( are produced. Correspondingly, for the target domain, Ti and Tu are also obtained in the generating learning strategy.
3.2.2 ICL OPTIMIZATION OBJECTIVE
After obtaining a of invariant concept-aware samples St for source domain Ds. the goal of the proposed ICI. can thus be formulated as the following optimization problem:
min Lial: (02, θ+) 09
04), 05) Ao (+(())
Σ ((((s.). ( + di H (( (na)), ((ff((x)))) (5)
s.l. 0 arg min
Sar~Si (E(C.2)
where 04 and O2 are learnable parameters for the backbone and classifier, respectively, ((ūx;; O the backbone extracting feature from a. Xir is the trade-off parameter and d(.) is the euclidean distance between two imputs. Lets (Q) (4), ) Vat; 09) is the cross- enssspy loss for classification. To further access the concept-invariant learning effect, we develop the invariant regularization loss (rr((; 0%) through a regularizer. We feed the S₁, St- S into the backbone network and explicitly enforcing them have invariant prediction, i.e., KL(P() S)), S;) P(Y))) S Such regular- ization is converted to an entropy minimization process McLachlan (1975), which encourages the classifier to focus on the domain-invarian invariant concept and downplay the domain- varriant style. The key idea of ICI similarly corresponds to the principle of invariant risk minimization (IRM) which aims to model the data representation for invariant predictor learning. More diseussion about IRM and ICL in the appendix.
3.3 COLLABORATIVELY DEBIASING LEARNING PROXY
After invariant concept-aware samples generation, obtain the S₁, T. and F.. Next, we will elaborate on how to utilize the advantages the extra supervised signals of target domain data Ti over the UDA setting. We introduce the Collaboratively Debiasing Learning framework (CDL) based on the robust DA setting with causal intervention. Specifically, we construct two SSL models: (c Og) w.r. (S₁, S and T,, and M3 w.zr (Ti fi and T,, as two complementary models with the same network architecture, which can cooperatively and mutually produce the pseudo-labels for each other to optimize the parameters Chen al. (20T)); Qiao et al. (2018). For instance, pseudo-label of Ita ~ To from M 02) is given by:
ÿ arg max P P 0%) Ta), where Ma b6) if P(j O%) Tr (6)
where and T. are the predefined threshold for pseudo-label selection. We will further elaborate on the two components in CDL, namely, a debiasing mechanism and self-penalizztion technique. Without loss of generality, we describe the components using one of the SSL models Ma(:; (3).
3.3.1 CONFIRMATION BIAS ELIMINATING MECHANISM
The ultimate objective of most SSL frameworks is minimize a risk, defined as the expectation of particular loss function over a labeled data distribution (X² )) S, Van Engelen & Hoos 2020).
Note that any distance measure distributions can be used in place of the Kullback-Ieeible (KT.) diver- gence Van Erven & Harremos (201)
5

---

Under review as a conference paper at ICLR 2023
Therefore, the optimization problem generally becomes finding Ø that minimizes the SSL risk.
min (:; 2 = I8 (a. [Ent, ((rat) Ma)): 0 + TT Pu Enbu (Ttm: {
s.b. θ arg max
Σ (S.156))~-
(yat 620l
og
where X is the fixed scalar hyperparameter denoting the relative weight of the unlabeled loss. Ent, (.) and Ent((-) (-) are the cross-entres eross-entropy loss function for labeled data S and unlabeled data Tw.
Proposition 1 (Origin of Confirmation Bias). SSL methods estimate the model parameters O₂ via maximum likelihood estimation according labeled data (X. St. Thus, the confirmation bias Be in SSL methods is generated from the fully observed instances, namely labeled data.
Under this proposition, the unbiased SSL learner should be impartial for less popular data (e.g., tail samples X)) and popular ones (e.g., head samples X2),, Le., P(2(,))) = P(X,)) y). Inspired by the inverse propensity weighting Glynn & Quinn 2010 theory, get the unbiased theorem for SSL.
Theorem 2 (Unbiased SSL Label Propagator). The optimization parameter 06 for SSL model should be taken same attention for all the labeled data, i.e., lurn maximizing Exmes, (orallalWa);; O (Complete proof in Appendix.).
0 arg max
Σ
logP(yallaat O5: = arg max
Σ za~S:
Iog θā) Sr (Nst, Mst)
0% (5.1:9a)55
6
(8)
Σ
Piyal (log) 2 2 - logP 62))
(
where (.Pw(.) is the Inverse Probability Weighting score. This formula can be understood as using the prior knowledge of marginal distribution P(); O to adjust the optimization objectives for unbiased learning. To make practical use of this Eq. 20 we estimate P(Y; θr B..t) in each mini-batch training for error backpropagation at iteration with batch size B,. It is noteworthy that we use a distribution moving strategy over all the iterations reduce the high- variance estimation between time adjacent epochs. With the gradual removal of bias from the training process, the performance gap between classes also shrinks, and both popular and rare classes can fairly treated.
3.3.2 SELF-PENALIZATION OF INDIVIDUAL CLASSIFIER
We also design eeff-eenalization that encourages the SSL model to produce more convincing pscudo-labels for exchanging peer classifier knowledge. Here, the negative pseudo-label indicates the most confident label (top-1 label) predicted by the network with confidence lower than the threshold r,. Since the negative pseudo-label is unlikely to be a correct label, we need to increase the probability values of all other classes except for this negative pseudo-label. Therefore, we optimize the output probability corresponding to the negative pseudo-label to be close to zero. The objective of self self-penalization defined as follows:
min (: 02) E (= (max ((yyu Fiui 02) < <T)))= ytallog(1 P(ytu (ar.: θ2)] (9) 0%
Such self- penalization is able to encourage the model to generate more faithful pscudo-labels with high-confidence score, and hence improve the data utilization for better invariant learning.
EXPERIMENTS
4.1 DATASET AND SETTING
Benchmark Datasets. DomainNe is originally multi-source domain adaptation benchmark.
Following Saito et al. (019 in its use for SSDA evaluation, we only select 4 domains, which are Real, Clipart, Painting, and Sketch (abbr. R. C. P and S), each of which contains images of 126 categories. Office -rrimee--me Venkateswara et al. (017)) benchmark contains 65 classes, with 12 adaptation scenarios constructed from 4 domains (i.e., R: Real world, C: Clipart, A: Art. P: Product). Office (Saenko et al. (2010) is a relatively small dataset contains three domains including DSLR, Webcam and Amazon (abbr. D, W and A) with 31 classes.

---

Under review as a conference paper at ICLR 2023
Table 1: (ccuracy(%) comparison on DomainNet under the settings of 3-shot using Resnet34 as backbone networks. A larger score indicates better performance. Acronym of cach model can be found in Section 4.1 We color each row as the best second best and third best
Method
R to C
to
too
to S
S
to S
to R
Mean Accuracy
63.6
60.8
74.5
61.2
DANN Ganin et al. 2016
59.8
62.8
59.6
55.4
59.9
54,9
72.2
60.7
MME Saito et al 2019
72.2
69.7
71.7
61.8
66.8
61.9
78.5
68.9
APE
Kim & Kim
20/11
76.6
72.1
76.7
63.1
66.1
67.8
79.4
71.7
SSSD Yoon al. 2022
75.9
72.1
75.1
64.4
70.0
66.7
80.3
72.1
DECOTA Yang
2021
75.2
78.7
72.7
81.5
75.6
CDAC Li ct ml
202
79.6
75.1
79.3
69.9
73.4
72.5
81.9
76.0
PACL
2020
79.0
77.3
79.4
70.6
74.6
71.6
82.4
76.4
Baseline
75.4
71.8
74.2
65.9
70.3
70.2
78.8
72.4
CAKE (Ours)
83.3
77.6
79.1
72.2
73.0
74.5
83.2
77.6
Implementation Details. We employ the ResNet- 34 He et al. (0016) and VGG-16 Simonyan & Zisserman (2014) as the backbone model on Doma i inet and Office-Homme respectively. We train CAKE with SGD BBottou (2010) optimizer all experiments. Besides, we use an identical set of hyperparameters (B=24, M,=0.9, 7=05, 7 =0 X,=1, \==1, =0,= 00p=0.1).
The causal factor generator C Cyyyleen Zhu al. 2017 and Cf (>)IImaee Augmentation,
Mc(:; Me)=Mixmatch Berthelot et al. (019 across all datasets.
Comparison of Methods. For quantifying the efficacy of the proposed framework, we compare CAKE with previous SOTA SSDA approaches, including MME Saito et (2019), DANN Ganin et al. (2016), BiAT Jiang et al. (2020). APE Kim & Kim (2020), DECOTA Yang et al. (002))
CDAC et al. (022b) and SSSD Yoon al. (2022). More details of baselines are the appendix.
4.2 EXPERIMENTAL RESULTS AND ANALYSES
Comparison with SOTA Methods. Table and (in appendix) summarize the quantitative three- shot results of our framework and baselines on Doma inNet and Office-Home. The one-shot results and analysis are in the supplementary material. In general, irrespective of the adaptation scenario, CAKE achieves the best performance on almost all the metrics SOTA on the two datasets. In particular, CAKE outperforms other baselines in terms of Mean Accuracy by large margin [DomainNed) 1.2% 16.4%, Office- REfice-Home: 3.3% ~ 9.7% and Office: 3.8% 12.0%) for SSDA task. Notably, our baseline, a simplified variant of CAKE without causal intervention and debiasing operation also obtained comparable results compared with SOTA (-3.6%). These results both benefit from the carefully designed ICL and CDL proxy subroutines that demonstrate the superiority and generalizability of our proposed model.
Individual Effectiveness of Each Component. We conduct ablation study illustrate the effect of each component in Table which indicates the following: Causal Inference is critical to boost SSDA (Row vs. Row 6), which significantly contributes 2.4% and 1.9% improvement Domain and Office Home, respectively. Mcanwhile, Row indicates that it suffers from noticeable performance degradation without the bias-removed mechanism (Row 1) (-1.3% and-1.8%) Furthermore, the results of Row 3 and Row 4 severally show the performance improvement of the Invariant Regularization (L,r) and Self- self-penalization (C.p). Summing up, We can observe that the improvement of using either module alone is distinguishable. Combining all the superior components, our CAKE exhibits steady improvement over the baselines.
Maximally Cross-domain Data Utilization. Here, we evaluate the effectiveness of data utilization (a) and (b) show the comparison between CAKE and baseline with respect to the top-l-accuracy, accuracy and number pseudo-labels on Domai inNet (Real - Clipart).
of the proposed method. Figure 3
Subscript and T represent the learned trained on source domain Ds or target domain Dy. During the learning iteration, we observe that the accuracy of CAKE increases much faster and smoother
B, Mo. J.r and T Tmax refer to batch size, momentum, learning rate and max iteration in SGD optimizer. The Mz and Me are orthogonal other advanced style changing and SSL methods boost SSDA further.
7

---

Under review as a conference paper at ICLR 2023
CUB Paaletsn
No.
Nurbec
1000
102 1400 01.66
110 3000
C1 M M AT C4
Remtion
(8) Teet Accuracy
(b) Paeudo-labels
(c) Threshold Value
(d) Class-wise Accuracy
Figure 3: Analysis of cross-domain data utilization and debiasing mechanism of CAKE. (a) and (b) depict the top-l-accuracy and correct pseudo-labels of CAKE and baseline within the first 200K iterations. (c) Cake's ss sensitivity to pseudo-label threshold т. (d) demonstrates the class- accuracy for head and tail classes in dataset produced by CAKE (w/o)/(//) confirmation bias.
than baseline, and outperforms baseline by a large margin of accuracy. CAKE also produces more convincing pseudo-labels than baseline. These pscudo-labels can assist the SSDA in performing global domain alignment to decrease the intra-domain discrepancy for robust invariant concept learning. Apart from learning visualization, we also investigate the CAKE's sensitivity to the confidence threshold for assigning pseudo-labels. Figure (c) empirically provides an appropriate threshold, i.e., r=0.5, either increasing or decreasing this value results in a performance decay. What's 's more, we conducted the cooperation vs. solo ablation that verifies the power of collaborative learning in Table 3 The detached SSL. model performs worse, demonstrating that training two correlated models achieves better adaptation as opposed only aligning one of them, because collaborative learning allows both models to learn common knowledge from different domains that in turn facilitates invariant learning, The aforementioned observation and analysis verify the effectiveness of CAKE in being able to deeply mine the po- tential of cross- donain data, thereby achieving the SSDA improvement.
Table 2:
Ablation study that showcases the impact of individual module.
Effect of Confirmation Bias Eliminating. To build insights on the unbiased SSL in CAKE, we perform
Method
DomainNet
Office
IWP Score
75.3
73.6
Invariant Regularization
76.7
74.9
an in-depth analysis of the bias-climinating mecha- nism in Figure d). In this experiment, domly select 10 classes (5 head and 5 tail). The re- sults suggest that CAKE and its variant CAKE (w/
Self penalization
77.0
74.8
ran-
Causal Intervention
75.2
CAKE (Ours)
77.6
75.4
bias) obtain a comparable performance on the head class. However, CAKE (w/ bias) fails to maintain the consistent superiority on the tail class while our approach does. (e.g..
tail class 69, CAKE: 46.0% CAKE (w/ bias) : 36.2%). This phenomenon is reason- able since CAKE maintains unbiasedness to each class-wise sample by maximizing P(X)). As the labeled/unlabeled data share the same class distribution, the accuracy of the tail class can be improved. In contrast, CAKE (w/ bias) focuses more on the head class, which results in an unbalanced performance for all categories. These results empirically verified our theoretical analysis and the robustness of the debiasing mechanism, which provides a reliable solution that guarantees the mutual data knowledge to be exchanged from source and target aspects.
Number of Invariant Causal Factors. Figure 4 (a) reports the SSDA results of different numbers of Invariant Causal Factors (ICFs) X' (2 x .2)) on DomainNet. Across all scenes, the best performance is usually achieved with Ng
Table Results of cooperation vs. solo.
Method
Domain
Office-Home
M
70.6
67.4
2, except for P - C. This ablation proves the ICFs of X can be learned from set of limited style-changing samples. Appropriately using these ICFs to conduct the deconfounded operation can effectively improve the SSDA performance.
M
68.3
65.8
CAKE (Ours)
77.6
75.4
Grad-CAM Results Causal Intervention. We systematically present the explicit benefits of the invariant concept learning (ICL). Figure 4bb) visualizes the most influential part in prediction generated from Grad- Trad-CAM Selvaraju et al. (207)). It's rather clear to see that CAKE appropriately captures the invariant part of the concept while CAKE CI) failed. We also analyze the reason why CAKE performs better in these cases. For instance, the concept C="celling fan" has a complicated background, i.e., style "luctettered". Without causal intervention, CAKE (w/o ICL.) tends to focus
8

---

Under review as a conference paper at ICLR 2023
R-- D
S-P Ra5
PaR
C-5
76-
Value of N
hruge
CAKE (wbo
CAKE
(a)/Accurccy Different Ni
(b) Class Maps
(G)
Figure 4: In-depth analysis of CAKE. (a) is the plot of invariant causal factor number Ni, against %6). (b) Grad-CAM results of CAKE and CAKE(W/O ICL). (c) I-SNE plot of features.
on the irrelevant information of style S= "cluttered", therefore predicting the wrong class. On the contrary, CAKE attend to the vital image regions by learning the invariant concept C= "celling fan" through the deconfounded mechanism.
Cross-domain Feature Alignment. We employ the I-SNE Van der Maaten & Hinton (2008) to visualize the feature alignment before/after training of adaptation scenarios D== "Rea" and Dy="Clipart" on Doma NNet. We randomly select 1000 samples (50 samples per class). Our invariant concept learning focuses on making Xss and Xy alike. It can be observed that as the model optimization progresses, e.g., C="teapot", the target features gradually converge toward target cluster cores. Each cluster in the target domain also gradually moves closer to its corresponding source cluster cores, showing a cluster-wise feature alignment effect. This provides an intuitive explanation of how our CAKE alleviates the domain shift issue.
RELATED WORK
Semi-supervised Domain Adaptation. Semi-superrie domain adaptation (SSDA) Saito et al. (2019); Qin et al. (2020); Jiang et al. (2020); Li & Hospedales (2020); Kim & Kim 2020 Li et al. : Yoon et al. (2022) address the domain adaptation problem where some target labels are available. However, these techniques mainly rely the domain shift assumptions of Covariate Shift and Conditional Shift to conduct SSDA. Such assumptions present intuitive solutions but lack a solid theoretical explanation for the effectiveness of SSDA, which hinders their further development. Thus we develop the CAKE, which decomposes the SSDA as two proxy subroutines with causal theoretical support and reveals the fundamental reason of the two domain shift assumptions.
Qin
(2021b):
Invariant Risk Minimization. Recently, the notion of invariant prediction has emerged as an impor- tant operational concept in the machine learning field, called IRM Rosenfeld al. (2020); (Arjossyy et al. 2019). IRM proposes to use group structure to delineate between different environments where the aim is to minimize the classification loss while also ensuring that the conditional variance of the prediction function within each group remains small. In DA, this idea can be studied by learning classifiers that are robust against domain shifts Li et al. (2021a) but still has the Covariate Shift issue. Therefore, we propose the CAKE that enforces the model to learn the local disentangled cooriiat-concepts rather than the global eerariant-fcaturee across domains, thus facilitating the SSDA.
Causality in DA. There some causality study in DA community. Glynn & Quinn (2010) considered domain adaptation where both the distribution of the covariate and the conditional distribution of the target given the covariate change across domains. Gong al. (2016) consider the target data causes the covariate, and an appropriate solution is to find conditional transferable components whose conditional distribution given the target is invariant after proper location-scale transformations, and estimate the target distribution of the target domain. Different from the two causal DA handle the DA task that only deals with the Conditional Shift issue, we also consider the Covariate Shift, which presents a improved IRM view for SSDA.
6 CONCLUSION
We first propose causal framework to pinpoint the causal effect of disentangled style variables, and theoretically explain what characteristics should a robust domain adaptation model have. We next discuss the maximal training data utilization and present collaboratively debiasing learning framework to make use of the training data to boost SSDA effectively. We believe that CAKE serves complement existing literature and provides new insights to the domain adaptation community.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015. 2015.
Eric Arazo, Diego Ortego, Paul Albert, Noel Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprini arXiv:1907.02893 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiiv1905.000 2019.
Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. arXiv preprint arXiv:1812.03253, 2018.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pp. 177-186. Springer, 2010.
Minmin Chen, Kilian Q Weinberger, and John Blitzer. Co-training for domain adaptation. Advances in neural information processing systems, 24, 2011.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1): 53 65, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Densanaadversarial training of neural networks. The journal of machine learning research, 17(1):009-20300 2016.
Madelyn Glymour, Judea Pearl, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Adam N Glynn and Kevin Quinn. introduction to the augmented inverse propensity weighted estimator. Political analysis, 18(1):36-56, 2010.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf.
Domain adaptation with conditional transferable components. In International conference machine learning, pp. 2839-2848 PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. arXiv preprint arXiv:1812.002230. 2018.
Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Cross-domain weakly- supervised object detection through progressive domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5001-5009, 2018.
Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li. Bidirectional adversarial training for semi-supervised domain adaptation. In IJCAI, pp. 934-940, 2020.
Taekyung Kim and Changick Kim. Attract, perturb, and explore: Learning feature alignment network for cemi-super domain adaptation. In European conference on computer vision, pp. 591-607. Springer, 2020.
10

---

Under review as a conference paper at ICLR 2023
Bo Li, Yezhen Wang, Shanghang Zhang, Dongsheng Li, Kurt Keutzer, Trevor Darrell, and Han Zhao. Learning invariant representations and risks for supe-vupee domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1104-1113, 202 la.
Da Li and Timothy Hospedales. Online meta-learning for multi-source and semi-supervised domain adaptation. In European Conference on Computer Vision, pp. 382-403. Springer, 2020.
Jichang Li, Guanbin Li, Yemin Shi, and Yizhou Yu. Cross-domain adaptive clustering for semi- supervised domain adaptation. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2505-255 2514,, 2021b.
Kai Li, Chang Liu, Handong Zhao, Yulun Zhang, and Yun Fu. Semi-supervised domain adaptation with prototypical alignment and consistency learning. 2020.
Tao Li, Shenghuo Zhu, and Mitsunori Ogihara. Using discriminant analysis for multi-class clas- sification: experimental investigation. Knowledge and information systems, 10(4):453-472, 2006.
Geoffrey J McLachlan. Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation discriminant analysis. Journal of the American Statistical Association, 70 (350):365-36)) 1975.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUmiiverittPPress 19: 2, 2000.
Michael Puthawala, Konik Kothari, Matti Lassas, Ivan Dokmanić, and Maarten de Hoop. Globally injective relu networks. arXiv preprint 200.:20000.844444 2020.
Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan Yuille. Deep co-training for semi- supervised image recognition. In Proceedings of the european conference computer vision (eccr), pp. 135-152, 2018.
Can Qin, Lichen Wang, Qianqian Ma, Yu Yin, Huan Wang, and Yun Fu. Opposite structure learning for semi-supervised domain adaptation. arXiv preprint arXiv:2002.02545. 3(4):6, 2020.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. arXiv preprint arXiv:2010.0576), 2020.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226. Springer, 2010.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8050-8058, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 61 018-626, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.153: 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
11

---

Under review as a conference paper at ICLR 2023
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-she. Journal of machine learning research, 9(11), 2008.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning, Machine Learning, 109(2):373-440, 2020.
Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions Information Theory, 60(7):3797-380) 2014.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5018-5027,
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2015.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey, Neurocomputing, 312: 135-153, 2018.
Edwin Williams. Representation theory. MIT Press, 2002.
Luyu Yang, Yan Wang, Mingfei Gao, Abhinav Shrivastava, Kilian Q Weinberger, Wei-Lun Chao, and Ser-Nam Lim. Deep co-training with task decomposition for semi-supervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8906-8916, 2021.
Jeongbeen Yoon, Dahyun Kang, and Minsu Cho. Semi-supervised domain adaptation via sample-to- sample self-distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1978-1987, 2022.
Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Universal domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2720-2729 2019.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired toaggeto--maae translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2023
This is the Appendix for "CAKE: CAusal and collaborative proxy-tasKs tasks lEarning for Semi-Supervised
Domain Adaptation". Table summarizes the abbreviations and the symbols used in the main paper.
Table 4: Abbreviations and symbols used in the main paper.
Abbreviation/Symbol
Meaning
Abbreviation
DA
Domain Adaptation
UDA
Unsupervised Domain Adaptation
SSDA
Semi-supervised Domain Adaptation
SSL
Semi-supervised Learning
CAKE
Causal and collaborative proxy-tasks learning
ICL
Invariant Concept Learning
CDL
Collaboratively Debiasing Learning
ICF
Invariant Causal Factor
ICFG
Invariant Causal Factor Generator
IRM
Invariant Risk Minimization
Symbol in Theory
Ds
Source Domain
DT
Target Domain
X
Image Data
Y
True Label
Concept
Se
Cross-domain Style
Sz
Intra-domain Style
Invariant Causal Factors
2 Ng
The Number of Causal Factors
Symbol in Algorithm
S
Source Domain Data
T
Labeled Target Domain Data
Tu
Unlabeled Target Domain Data
Mz
ICL Proxy Subroutine
Mc
CDL Proxy Subroutine
Oz
Parameters of Mz Parameters of Mc
Oc
S
Causal factors for Ds
t, T
Causal Factors for Df
Liel
Loss of Invariant Concept Learning
Codl
Loss of Collaboratively Debiasing Learning Invariant Regularization Loss
Lin
Cap
Self- Loss
This appendix organized as follows:
Section 7 provides the proof about the disentangled X causal intervention, invariant risk
minimization, unbiased eliminating mechanism and further discussion of Assumption 1.
Section g provides the method details of proposed CAKE.
Section rrppors more experimental settings of datasets, baselines, implementation details and training process of CAKE.
13

---

Under review as a conference paper at ICLR 2023
Section
10
shows the additional experiments
DomainNet
and
Office -emooee Venkateswara et al. a (2017) to verify the effectiveness of CAKE.
Section
T lists the limitations of this paper.
PROOF AND DERIVATION
This section derives the disentangled causal intervention and the inverse probability weighting theory for the confirmation of unbiased eliminating mechanism.
7.1 DISENTANGLED X' CAUSAL INTERVENTION
We will first provide a brief introduction to the preliminaries of disentangled causal intervention. Real-world observations, according to physicists, are the result of a mix of independent physical rules. This also applies to causal inference Higgins et al. (2018), i.e., the laws are denoted as disentangled generative factors, such shape, color, and position. Let the X' represent the image data, each X' can be disentangled into concept C, cross-domain style Se and intra-domain style Sr variables which are mutually independent, i.e. a triplet X' (C, Sz). where C TT Se 1L Sz. Correspondingly, the invariant causal factors of X are given by & style mapping results of X), where E (Sc; ST) are different from X. Only concept C is relevant for the true label y of X, i.e., style changing is concept- preserving. In other words, there is set of independent causal mechanisms 6z: S - X, &, generating images from S. To study how can we get the X, we leverage the assumption of disentangled variables based on Higgins' definition of disentangled representation Higgins et al. (2018). We state the definition as follows:
Definition (Group Action on Disentangled Variables). Let G be the group acting on S, i.e, 9- E S S transforms s E S, e.g., group element of "turn domain style, real to clipart" changing the semantic from "real" to "clipart". Suppose there is a direct product decomposition 91 x 92 x... x 9g and S Si S2 X ... Sy. where 9i acts on S₃ respectively. A feature representation is disentangled there exists group G acting on X such that:
Theorem 3 ( Decomposable X.) There is a decomposition = x N2 x ... X X. such that each X; is fixed by the action of all 9ý. where j # i and affected only by 9r, e.g., changing the "domain style" semantic in S does not affect the "concept" vector in X.
P(X) = concept gi X.) = P(X, = concept|.t (x;)
(10)
Theorem 4 (Equivariant Semantic Changing.) 4g G. Vs S, 8) f(s),
e.g., the feature of the changed cross-domain style semantic: "real" to "clipart" in S, is equivalent directly change the style vector in X' from "real" to "clipar".
P(g s|S) = P(g s|x²
(11)
Under this Theorem 3, the disentangled representations are obtained by our Assumption 1, i.e. X = Sc, ST). Compared to the previous definition of feature representation which is a static mapping, the disentangled representation in Definition 1 is dynamic as it explicilly incorporates group representation Williams (2002). which is a homomorphism from group to group actions on a space, e.g.. 9 > X' X, and it is common to use the feature space X. What's more, Theorem 4 indicates that performing group action of semantic changing (e.g., style changing) g on S is equivariant for S and X. Thus, X' can be obtained by performing different g E G on X.
Next, we can introduce the causality that allows computing how an outcome would have changed, had some variables taken different values, referred to as a causal intervention. As a prerequisite. i should be calculated following the three steps of computing principles Pearl & Mackenzic (2018):
In abduction, all the invariant causal factors, i.e., (& $1, X -22 inferred from X through P(X) (()
. .xx) are
In action, & G; is drawn from P( = -::DD Ds) or p(g = -:IDD =D7), while the values of other & are fixed.
14

---

Under review as a conference paper at ICLR 2023
In prediction, the modified (& f2, X 82, ... X 2xk) is fed to the generative process
P(X X) obtain the output.
More details can be found in Glymour et al. (2016). Based on the computing principles, we consider 9 as an embedded function Besserve et al. (2018). 8). i.e. a continuous injective function with continuous inversion, which generally holds for convolution-ben based networks shown in Puthawala et al.) (2020). X' are obtained through the generative process 9 X + X. X with the invariant concept with X' can be regarded as the causal factors to take a causal theoretical view for the domain adaptation problem.
Proof of the Sufficient Condition. Suppose that the representation is fully disentangled w.r.f. G. By Definition 1, there exists subspace X; E X' affected only by the action of gi E G. This part aims to prove the following sufficient condition: if g: intervenes S,, the invariant causal factors(ICTs)) are faithful when the L. is smallest or the jth image augmentation have not change the concept. For a sample zg from D Ds, let 9 (xs) S (S1, S2. Sy.). We modify style by changing Di G drawn from P(g. D DT) (cross-domain style) or P(g: 'D Ds) (intra-domain style). Denote the modified style as S (S1 x S2 x Sx). Denote the sample with style S as sts. Given Ii intervenes S corresponds to a counterfactual outcome when S, is set S, through intervention. Now as 9 (xx) = S. using the counterfactual consistency rule, we have gi(zs) = æs. As &g is faithful with the Counterfactual Faithfulness theorem Pearl et al. (2000), we prove that (x())) also faithful, i.e., the smallest L.,, or the 7t image augmentation have not change the concept.
7.2 INVARIANT RISK MINIMIZATION
In seminal work, Arjovsky et al. (2019) consider the question that data are collected from multiple envrionments with different distributions where spurious correlations are due to dataset biases. This part of spurious correlation will confuse model to build predictions on unrelated correlations rather than true causal relations. IRM estimates invariant and causal variables from multiple environments by regularizing on predictors to find data represenation matching for all environments.
Let X' be the image space, Z and y represent the be feature space and classification output space (e.g.. the set of all probabilities of belonging to each class), the feature extractor backbone $ X' - 2 and the classifier w y. Let & be set of training environments, where each e E & is set of images. Mathematically, IRM phrase these goals as the constrained optimization problem:
min 40,01
(:: +A VAAI/Vmleee [Vmjm==1.0 R
(w (w-))l 1
(12)
REELT ERM Term
Invariant Risk
where (:; b) is the empirical classification risk (ERM) in the environment e, w 1.0 is scalar and fixed "dumm" classifier, the gradient norm penalty is used to measure the optimality of the dummy classifier at each environment e, and À [0, +oo) is a regularizer balaneing between predictive power, and the invariance of the predictor 1- (()).
In DA problem, IRM can be regarded as the classic ERM term (e.g., classification loss) plus the invariant risk (e.g., discrepancy between conditional distributions over the features) Li et al. (2021a). The invariant risk consider about the Conditional Shift assumption (P())xX D Ds) 7 P(Y|, D = Ds)) to learn the data representation, thereby learning the invariant predictor. However, IRM may not be the true savior for the DA task which still has two issues:
Covariate Shift. As we discussed in section I the marginal feature distributions are different across domain, i.e., P( Ds) * P(X = Dr). In SSDA task, there are only few labeled samples in Dy, hard to measure the gap of marginal feature distribution between two domains. Thus, an alternative is consider the Covariate Shift that reduce the features discrepancy across domains. However, IRM is not sufficient to tackle this issue.
Spurious Correlation. The learned global representation of image still has noise style information rather than the fine-grained concept. Using such global representation may leave confused style information in feature space, resulting in inaccurate prediction.
Different from the IRM in DA, ICL tackled the aforementioned issues at two points: 1) represent- ing images from the source domain Ds to target domain Dy by invariant causal factor generator; 2) eliminating the spurious correlation of style and label by statistical learning with causal intervention.
15

---

Under review as a conference paper at ICLR 2023
In summary, our proposed CAKE not only addressed the two domain shift issue, but also enforced the model to learn the disentangled invariant concepts across domains, which can boost the SSDA reasonably.
7.3 INVERSE PROBABILITY WEIGHTING THEORY
The main content of this paper indicates the unbiased SSL learner should be impartial for less popular
data (e.g., tail samples Xg) and popular ones (e.g., head samples 24), i.e. P(Xx)))) = P(2, )}). Inspired by the inverse propensity weightingGGlynn Glynn & Quinn (2010) theory. which introduces weight for each training sample via its propensity score, which reflects how likely the label is observed (e.g. its popularity). In this way. IPW makes up a pseudo- balandedd dataset by duplicating each labeled data inversely proportional to its propensity- less popular samples should draw the same attention as the popular ones more balanced imputation. Thus, take D Ds as an example, the optimization parameter @g for SSL model Må should be taken same attention for all the labeled data. In general SSL training, Og can be optimized as follows:
O = arg max 0
Σ
log DygP(Walaaar) 0g)
(13)
(MatsMat)~-S.
P(ust; 2/)
arg max 0%
Σ
logP(xxllybl' 0z) p Q+)
(14)
(Fat I-
arg max 6g
Σ
log 0²) P(y:: +
(15)
log logP(zxilVeti 6g) + log logP((stt 02)
arg max 6g
Σ
log logP(xalyw::0))
(16)
log (ogP((al110 |Mat O+:)
(
log P(yat P1 log CoggP(y.t 62)
= arg max
Σ
logP(xxalle:: Og)
P( P(:)))))))))
(17)
(Bat.Mat)~-Si
log
P(yn:bg)
log P
+ log (2)
arg max og
Σ
log P(salaaa og)
P(
(18)
log P(unlla))))))
P(
logP((yslla 221 6:)
arg max og
Σ
log (I og)- logP (a: Q₂) log Plyat Qg)
(19)
log Dygal θg)
+ S.Pw(.)= LogP((yrl 2g) - logP 0g)
(20)
where SIPW (.) is the Inverse Probability Weighting score, P( Qg) is the sampling probability an empirical distribution which is thus constant. Thus, from (4) to (5) and (7) to (8) in Eq. 13 P P Qg)
can be ignored. Therefore, we can turn maximizing arg max 6
Σ
log |ra:: θ2) to maxi-
mizing arg max o using the prior knowledge of marginal distribution P(Y: θg) to adjust the optimization objectives for unbiased learning. Thus, the SSL model MM is unbiased to the class-wise sample by maximizing
Σ logP yar θ2) according Eq.
13 This formula can be understood as
Σ
log ((re( Bar; O2), thereby climinating the undesirable confirmation bias.
(S
7.4 FURTHER DISCUSSION OF ASSUMPTION
We would like to clarify the further explanation of our proposed assumption (Disentangled Variables). Besides, we also give other intuition of this assumption for other tasks.
16

---

Under review as a conference paper at ICLR 2023
ICL Subroutine
CDL Subroutine
Intra- domain
Cross-domain
Causal Factors
Causal Factors
Source domain
Target domain
X
Discriminator
M G5)
3:2(: o5)
Target Domain
Leerned on Labeled Data
Learmed on Labeled Data
CycleGAN Ggg
Causal
Kor
Factors
Blas Ellminating
Target Domain
Gre
62
Unbissed SSL Label Propagator
Unbiased SBL Labal Propagator
Cros-domain ICFC
Discriminator
Exp
Ler Lg Li
Lp Exp
Casual Inference
Intra-domain
Cross-domain Factors
P(r]do(X)) D.) PCT D.)
Figure 5: Overview of CAKE. In ICL proxy, .Mz(; Oz) first learns the causal factors for Ds and DJ unsupervised learning paradigm, aiming to generate the invariant causal factors and use causal intervention to remove the confounding effect. In CDL aspect, labeling-based SSL techniques: (i) - Tu and (Ti, Fi) - Tu,, aiming utilizing all the training data possible to bridge the feature discrepancy under the premise of invariant "concept" learning.
construct two pseudo
Single classification task. We have already noted that this assumption may not be true in all settings, but we believe that many image settings can be approximated. For example, in most of the single classification tasks, a given image (e.g., a dog on the lawn) only has one label. The annotators tend to focus on the most important region (the dog) which can be regarded as the concept to give the label. The dog with other colors or appearance can regarded the intra-domain style. The dog in other domain (e.g., clipart domain) with different backgrounds can refer to the cross-domain style. other words, and are confounders that interfere the model to predict the true label given the image.
Other complex vision tasks. For these tasks, e.g. multi-label task Li et al. 2006). visual question answer Antol al. (2015), visual captioning Vinyals al. 2013), this assumption may not be applicable. For instance, an image describes "a dog and cat on the lawn". When the image is classified extraneous factor called object confounder. Nevertheless, we also investigated this scenario in and the results systematically show the robustness of the proposed CAKE.
dog. besides the style confounders the cat is also an
METHOD DETAILS
This section presents the method details of the proposed CAKE. Figure. iillustrats the overview of our CAKE framework that contains two proxy subroutines: Invariant Concept Learning (ICL) and Collaboratively Debiasing Learning (CDL). Next, we will elaborate the details of cross-domain style transfer and bias eliminating mechanism practice.
8.1
CROSS-DOMAIN STYLE TRANSFER
The invariant concept samples with cross-domain style transferring is generated from CycleGANZZi al. 2017 Here we provide the architectures of generator and discriminator in Table
17

---

Under review as a conference paper at ICLR 2023
Table 5: Abbreviations and the symbols used in the main paper.
Generator
Discriminator
ReflectionPad (3)
Conv2D (4,64)
Conv2D (7,64)
LeakyReLU (0.2)
InstanceNorm ReLU
Conv2D (4,
Conv2D (3,128)
InstanceNorm
InstanceNorm ReLUx 2
LeakyReLU (0.2)
Conv2D (3,256)
Conv2D (4,256) InstanceNorm
InstanceNorm ReLU
ResNetBlock x 2
LeakyRel.U (0.2)
InstanceNorm ReLU
Conv2D (4,512)
TrnnSssasDD (3,64)
InstanceNorm
InstanceNorm ReLU
LeakyReLU (0.2)
ReflectionPad (3)
Conv2D (4,1)
Conv2D (7,3)
Tanh
Loss of Cross-domain Invariant Causal Factor Generator. The CycleGAN technique transforms the images to enable the preservation of source concepts during the cross-domain conversion process. Take Ds Dy example, develop Ng CycleGANs, each of them consisting of three-loss parts to conduct the cross-domain style transfer.
arg min Ch; = arg min (Nado Lyddo + Reye Elye + Xiat Lide)s k k= 8,
argmin C (eI,,.,,
(21)
8,
where C Cbye and Ch are adversarial loss, cycle loss and identity loss, respectively. Aado- Aeye and didt correspond to their trade-off parameters. Specifically, the Lad Lh and L can be calculated as follows:
Cadv
E [ogD((x)) [T,(TT..TT)
+ E
[log(1 - (C
(22)
2.
C
E
|| (Gk (x=))
-alll +
[|| (x+)) x:Il]
(23)
BarS
(T..TT)
Ch
[ll G (Iat) ) Ieilll Ja
E
||I (xx) 2z |l]
(24)
(T.
where ||-|| denote the LI norm. is the discriminator to distinguish the origin source of the latent vector if from 'D_ Gk, and G correspond to the kth Ds - Df and Df - Ds cross-domain invariant causal factor generator (ICFG), respectively.
8.2 INTRA-DOMAIN STYLE TRANSFER
For the intra-domain style changing factors, we utilize the data augmentations as intra-domain style interventions,e.g color temperature and sharpness according the cross-domain style changing samples. The code of the intra-domain style transfer can be found in our online project. Thus, the invariant causal factors S = ($), $ are produced. Correspondingly, for the target domain, T, and T, obtained in the generating learning strategy.
18

---

Under review as a conference paper at ICLR 2023
8.3 INVARIANT CONCEPT LEARNING
After obtaining a set of invariant concept-aware samples Si for source domain Ds, we design the ICL loss function which has two aspects follows:
min Lio(: (0%, 07)) Lots Air (ir(; 05
(25)
min Lid (: 8%
Σ
E d(((aa),
((C)) ((C + di d( (r.)), ((C()(
Ru'Si GE(C.))
where 0% and O are learnable parameters for the backhone and classifier, respectively. a +%
is the backbone extracting feature from F,t. Air is the trade-off parameter and d(.) is the euclidean distance between two inputs. ( O4), Mati 0%) is the cross-entropy loss for classification. To further access the concept-invariant learning effect, we develop the invariant regularization loss Lir(: 04) through regularizer. Such regularization is converted to an entropy minimization process McLachlan (1975), which encourages the classifier to focus the and downplay the domain-variant style.
concept
8.4 MONFON BIAS ELIMINATING MECHANISM
According complete proof of IPW theory in section 7.3, we aim find the optimal θ that logP(xxlluet' O to implement the debiasing SSL model learning.
maximizing Sm
Σ
log DogP(yallzar O SIPw(.) (-)
- arg max
Σ
logP Mar; θ0) arg max 0
Σ
(26)
(ya.22))
To make practical use of this Eq. 20 we estimate O t) in each mini-batch training for error backpropagation iteration / with batch size B.. It noteworthy that we use distribution moving strategy over all the iterations to reduce the high-variance estimation between time adjacent epochs. The details below:
Σ (Mo (f 0g Batt B.) + (1 (a)) PQy B.,
Ply (0) θ B.,t) t)
(27)
where B, is the batch size and M, is momentum coefficient, P(.) is the re-estimated prior. With the gradual removal of bias from the training process, the performance gap between classes also shrinks, and both popular and rare classes can be fairly treated.
8.5 OBJECTIVE FUNCTION OF CDL
The full objective function of CDL has three parts, supervised loss L.(: unsupervised loss L. (55 Oc) and self-penalization loss (sp(: Oc):
min (:ull: Oc) [E[A. (= Qc) + Au Ca(::0c) Cep Oe 10), 0%) (28) Bc
where Ae, du and Aep denote the pre-defined hyper-parametto
EXPERIMENTAL SETTINGS
Implementation Details, Our work can be checked the Anonymous Link [
Algorithm and
presents the pseudocode of training and inference process of CAKE. We use the PyTchh Paszke et al.)
(201)) deep learning framework to conduct all our experiments on 8x V100 GPUs and 8x 2080Ti
GPUs. We employ the R4sNet-34 He et al. (2016) and VGG-16 VGG-166Simonyan & Zisserman (2014) (We also report the results of ResNet-34 on office-Home as the backbone model on Doma nNet
hhtp://aihhu.cctdMM/////
19

---

Under review as a conference paper at ICLR 2023
Table 6: Complete list of hyper-parameters.
Hyperparameters
Notation
Value
Labeled Data Batch-size
B
24
Confidence Threshold
T
0.5
Momentum
M.
0.9
Learning Rate
Ly
0.001
Max Iteration
T max
20000
HP of Supervised Loss
X
HP of Unsupervised Loss
Xu
HP of Invariant Regularization Loss
Xir
0.1
HP of Belf-peenaizzttoon Loss
Asp
0.1
and Officee office-Home Home respectively. We train CAKE with a standard stochastic gradient descent (SGD) Bottou (010) optimizer all experiments. We follow Saito et al. (2019) to replace the last linear layer with a K K-way cosine classifier (e.g., K = 126 for DomainNet) et) and train it at a fixed temperature (0.05 in all our setting). Besides, we use an identical set of hyperparameters (B=24, across all datasets. We utilize the Mixmatch Berthelot et al. 2019 as the semi- -emi-superiised learning model, the basic loss function for CAKE consists of two cross-entropy loss terms: supervised loss L. applied to labeled data and unsupervised loss L..
M,=0.9, Ly, ==0.5, mma0==0,0000
For quantifying the efficacy of the proposed framework, we compare CAKE with previous SOTA SSDA approaches, including MME Saito et al. (2019). DANN Ganin et al. (2016), BiAT Jiang et al. (2020). APE Kim & Kim (020). DECOTA Yang et al. (021).
Comparison of Methods.
ELP nouu et al. (2018), CDAC Li al. (2021b) and SSSD Yoon et al. (2022) We also present simplified version of CAKE as the baseline.
We compare the results of CAKE with wide range of baselines, including early works and recent SOTA models on this task:
Baseline is simplified version of CAKE without causal intervention, bias eliminating mechanism, invariant regularization and self-penalization.
MME Saito et al. (2019) first proposed solve SSDA by aligning the features from both domains by means of ad adversaiial learning.
DANN Ganin et al. (2016) augmented the model with few standard layers and new gradient reversal layer based on the features that cannot be discriminated between the source and target domains.
ELP Inoue al. (2018) designed framework with domain transfer and pseudo labeling to generate instance- levell annotations for the target domain.
BiAT Jiang al. (2020) devised bidirectional strategy with an adaptive adversarial model and an entropy-pente penalized virtual adversarial model to guide the direction of generating adversarial examples.
APE Kim & Kim (2020) addressed the intra-domain discrepancy issue via attraction, perturbation, and exploration schemas.
DECOTA Yang al. (2021) decomposed SSDA into SSL and UDA problem two models to bridge the gap and exchange expertise between the source and target domains.
CDAC al. (2021b) developed an adversarial adaptive clustering loss to guide the model training towards grouping the features of unlabeled target data into clusters and further performing cluster-wise feature alignment across domains.
"htps://ww.hhmmamtt hww.manthwyorrooffsoooooomomeeme html
B, Mo. L, and Tmaz refer to batch size, momentum, learning rate and max iteration in SGD optimizer. http://hhhh.oo/m////////aaaaaaaa
20

---

Under review as a conference paper at ICLR 2023
Algorithm 1: CAKE: Causal Multi-proxy Subroutine Learning Framework
Input: Training data St from source Domain Ds. Ti and Tu from target domain Dy pre-trained classifiers Mc(:; O and Mc(:; O)) with parameters 0g and & respectively;
Output: Invariant causal factors of St Ti. Tu. fine-tuned classifier MMc(;
Initialization: Randomly initialize the parameters { N₂ k=1 of Cross-domain ICFGs (ttt
N₂
repeat
Cross-domain ICFG model in CAKE have not trained then repeat
Randomly sample a minibatch;
Train cross domain ICFGs {Gst)imn with unpaired S, Ti and Ta with cross-domain style transfer loss Let;
Update {+% 1 using Eq
10
until Convergence;
11 until Convergence;
Image augmentations for St, Ti and Tu:
13 Generate Š, = ( St J = (T, Te', Fu = (Ta, T.
14 Initialize Mc(:; @+) and Mc(;
repeat
Randomly sample minibatch from (St, St and (Ti, Tr);
Eliminate confirmation bias B, using Eq.
17
Obtain pseudo-labels using Eq. 6
18
19
Calculate CDL loss of L., L, and Dppi
Calculate Po)) D = Ds) and P()))d((2), D Dr) for Mc(; and Mc(:;
29
using Eq. Eq.2
21
Calculate ICL loss of Lets and Lir using Eg 25
Update parameters Og and Ø%:
22
23
until Convergence:
24 return Classifiers Mc(::0g) and Mc(;
Algorithm 2: CAKE Inference
Input: Test data T, from target domain Dy, fine-tuned classifier Mc(-99) with parameters O:;
Output: Predicted label 0t:
Sample Ztu ~ Tw:
Sample invariant causal factors { of Trui
2N
Measure deconfounded effect using Eq. 2
return you f argmax P()]do(X'' = Itu). = Dr).
SSSD Yoon et al. (2022) exploited the rich sample-to-sample relations using self celf-distillation fill the domain gap, thus facilitating the adaptation.
IRM Arjovsky et al. (201)) estimates nonlinear, invariant, causal predictors from multiple training environments, to enable -iittf-diitiibuttin generalization.
LIRR al, (22 simultaneously learning invariant representations and risks across domains, leading to bound minimization algorithm for SSDA.
PACL et al (020) uses a data augmentation-baat technique to produce highly perturbed images mitigate overfitting. The consistency alignment module incorporated into the framework, which enforces consistency regularization on the classifier.
21

---

Under review as a conference paper at ICLR 2023
Table 7: Accuracy on Off seffice-iee ce-Home (%) for three-shot setting with domains, using VGG-16, A larger score second best and third best
indicates hetter performance. We color each row as the best
R
P
C
S+T
09.5
78.6
63.6
72.7
47.2
55.9
69.4
47.5
73.1
69.7
56.2
70.1
62.9
DANN Ganin
77.9
MME Satto ml.
0019
$6.9
82.9
63.7
76.7
53.6
59.2
75.7
$1.9
75.3
72.9
61.1
76.1
67.6
59.3
75.0
73.7
75.1
66.5
DBC 3.21
83.9
67.7
17.3
57.7
60.7
78.0
74.3
63.2
78.4
69.3
EL. Inone et al.
2018
57.1
83.2
67.0
76.3
53.9
99.3
75.9
55.1
76.3
73.5
61.9
76.1
68.0
82.7
59.8
CAKR
87.0
70.8
81.6
Table 8: Accuracy( ()
comparison on Office under the settings of I-shot and 3-shot using Alexnet as backbone
networks.
W A
MEAN
Method
one-shot
three-shot
one-shot
three-shot
one-shot
three-shot
S+T
50.4
61.2
50.0
62.4
50.2
61.8
DANN
Ganin et al.
2016
57.0
64.4
54.5
65.2
55.8
64.8
MME
Saito ct al.
2019
57.2
67.3
55.8
67.8
56.5
67.6
BiAT
Jiang al.
2020
57.9
68.2
54.6
68.5
56.3
68.4
APE
Kim & Kim
2020
67.6
69.0
68.3
CDAC
Li et al.
2021b
63.4
70.1
62.8
70.0
63.1
70.0
CAKE(Ours)
66.9
74.5
66.0
73.1
66.5
73.8
10 ADDITIONAL EXPERIMENTAL RESULTS
We conducted the additional experiments on two datasets at different aspects (i.e., one-shot setting, larger shot Learning t-SNE and Crad visualization of invariant causal factors.) to verify the strength of CAKE.
One-shot Setting. We report the comparison with baselines in the one-shot setting Doma nNet. in Table and and
Of ce-Home Table 13 CAKE outperforms the SOTA methods by 1.8% and 2.3% on DomainNet (ResNet-34) and Office Home (VGG-16), respectively. The performance of CAKE for one-shot learning is better than the three-shot setting, which suggests the almost best accuracy obtained (except for P-RR DomainNes A A-C and C-+A Office (Efice-Home.), As shown later, we also employ the Resnet-34 as backbone to compare the SOTA CDAC (2021b) in Table [5| CDAC accuracy much lower compared with our CAKE on & shot setting. These observations demonstrate the robustness and generalizability of proposed CAKE once again.
Table 9: Accuracy(%) comparison on Domai nNet the settings one-shot using Resnet34 backbone networks.
Method
R to C
to
Pt C
Cooo
to P
R 8o
to R
Mean Accuracy
DANN Ganin al.
2016
58.2
61.4
56.3
52.8
57.4
52.1
70,3
58.4
MME Saito ot 2019
70.0
67.7
69.0
56.3
64.8
61.0
76,1
66.4
APE Kim & Kim
2020
70.4
70.8
72.9
56.7
61.5
63.0
76.6
67.6
SSSD Yoon et al.
2023
73.3
68.9
73.4
60.8
68.2
65.1
79.5
69.9
DECOTA A
Yang
2021
79.1
74.9
72,0
69.7
79.6
CDAC
bb2
77.4
74.2
75.5
67.6
71.0
69.2
80.4
73.6
PACI
75.3
Basclinc
75.3
71.2
73.4
65.1
68.9
69.5
77.6
71.5
CAKE (Ours)
80.6
76.9
78.3
69.5
72.3
71.9
80.3
75.7
22

---

Under review as a conference paper at ICLR 2023
Table 10: Accuracy(%) comparison of UDA and SSDA of proposed CAKE on Doma i inNet under the settings of three-shot using Resnet34 as backhone networks.
Method
UDA
SSDA
R C
to P
P to C
C
t P
R S
P to R
Mean Accuracy
CAKE w/ CDL
77.2
73.0
74.8
66.0
69.8
67.6
79.2
72.5
CAKE w/o ICL
75.1
77.8
68.2
71.7
72.4
80.9
75.2
CAKE w/o IPW
80.1
74.6
78.1
69.0
72.0
72.5
80.8
75.3
CAKE (Ours)
83.3
77.6
79.1
72.2
73.0
74.5
83.2
77.6
Table 11: Accuracy(%) comparison of one-stage CAKE and baselines on DomainNet under the settings of 3-shot using Resnet34 as backbone networks.
Method
R to C
R to P
P to C
Co S
S to P
R to S
P to R
Mean Accuracy
CDAC Li al.
2021b
79.6
75.1
79.3
69.9
73.4
72.5
81.9
76.0
PACL Li et al.
2020
79.0
77.3
79.4
70.6
74.6
71.6
82.4
76.4
CAKE (w/o) Se
79.0
77.3
79.4
70.6
74.6
71.6
82.4
76.4
CAKE (Ours)
83.3
77.6
79.1
72.2
73.0
83.2
77.6
UDA Setting. To further evaluate the effectiveness of the proposed ICL proxy, tested variety of ablation models: (1) CAKE w/o CDL, (2) CAKE w/o ICL (3) CAKE w/o IPW. From T3 could observe that the CAKE with UDA setting also obtains comparable results compared with SSDA models. This table also suggests the ICL and IPW are both usefuul to boost SSDA performance, which contributes to the accuracy of 2.3 and 2.2. These observations further demonstrate the robustness and generalizability of the proposed ICL and IWP once again.
Table Accuracy on fffice-Home one-
and three shot using ResNet-34.
Method
One-shot
Three-shot
CDAC ei 2021b
70.6
74.2
CAKE (Ours)
72.8
75.4
Analysis of Class Imbalance. Table 12 summarizes the comparison of generated pseudo- labels of CAKE and CAKE (w/o IWP) on Doma n.et. As reported in Table 12, under the imbalanced labeled and unla- beled data, Our CAKE generate more pseudo- labels with higher accuracy, both in tail class data and head class data. In contrast, CAKE w/o IWP generates few correct pseudo-labels, especially for the tail class data. These results empirically verified the robustness of the debiasing mechanism that can generate a more accurate and balanced result. Such mechanism provides a reliable solution that guaran- tees the mutual data knowledge exchanging from source and target aspects.
Table 16: Results on Doma nNet (R -> C) at 10, 20, 50-shot setting, using ResNet-34.
Method
10
20
50
Mean
S+T
69.1
72.4
77.5
73.0
DANN
66.2
68.0
71.1
68.4
ENT
77.9
80.0
83.0
80.3
77.0
80.9
CAKE (Ours)
83,3
84.0
86.2
84.5
Larger Shot Learning. We provide 10,20,50-shot SSDA results on DomainNet (R C) in Table 16 We randomly select and add additional samples per class from the target domain to the target labeled pool. The implementation details are the same as those of 1,3-shot. From this table, CAKE's performance improved along with more shots and can outperform baselines from 10- to 50-shot settings, which maintains remarkable results consistently.
As pointed out in Sec., IRM is is not sufficient to ensure reduced discrepancy across domains. To validate our point, we report the experimental results IRM vs. ICL (Table 14 and further analyses shed light on the point. According Table ICL-based CAKE outperform IRM- based approaches by a large margin of mean accuracy Domainnett:n +22.3% and LIRR:20.9% of f i ce Home, IRM: + +103% and LIRR +7.5%). IRM asserts that it is critical minimize the
IRM ICL.
23

---

Under review as a conference paper at ICLR 2023
Table 12: Comparison of generated pseudo-labels of CAKE and CAKE (w/ IWP) on DomainNet (R to C).
Classi
Correct Pls Class)
Class)
Class)
(wl IWP)
15,20
1,5122
12414
2999
2424
CAKE
15326
13912
12866
2999
2493
2398
Table 13: Accuracy on Office- (%) for onc-shot setting with domains, using VGG-16.
Method
R
R to
R A
Pt
P
Ptto A
A to
to C
to R
C.to R
C.o
Cto P
MRAN
S+I
39.5
75.3
61.2
71.6
37.0
52.0
63.6
37.5
69.5
64.5
51.4
65.9
57.4
DANN Garin al
2016
52.0
75.7
62.7
72.7
45.9
51.3
64.3
44.4
68.9
64.2
52.3
65.3
60.0
2019
56.0
71.3
DECOTA
3221
47.2
80.3
64.6
75.5
47.2
56.6
71.1
42.5
73.1
71.0
57.8
72.9
63.3
ELP Inone En et al.
2013
49.2
79.7
65.5
75.3
46.7
56.3
09.0
46.1
72.4
68.2
67.4
71.6
63.1
79.7
64.8
56.3
72.4
CAKR (Oues)
82.4
65.9
76.4
58.1
73.7
74.6
65.6
discrepancy between conditional distributions over the features. However, IRM is not sufficient to ensure reduced feature discrepancy across domains and it makes the model rely more on spurious correlations (style -> label). Instead, ICL not only considers the invariant risk, but also models the invariant concept by eliminating the confoundin effect of spurious correlations, which further acknowledges the importance of ICL.
1-SNE and Grad-CAM visualization of ICFs. To further assess the impact of invariant concept learning, we randomly select two class ( class A :"helicopter", class B "camel" in Doma i nNet dataset, R-> C scenario) samples with their invariant causal factors (ICFs) and use t-SNE to visualize their embeddings in 2D space. Overall, as shown in Figure g CAKE clusters instances with their ICF' apparently classified into two classes. The ICFs preserve the invariant class-wise concept semantic with different styles that helps the SSDA model distinguish different classes. What's more, we visualize four examples with ICFs by Grad-CAM Selvaraju et al. (2017) to further examine the invariant concept learning. Figure 6 shows CAKE learns the invariant concept and attends to the similar image pixels in these samples same concept with different styles), e.g., foreground object shape semantics. For instance, in the "Castle" example, the styles of the four ICFs are drastically different from the original "Castle" image. Nevertheless, benefitting from the causal interventions, our CAKE can distinguish the invariant "concept" features across domains and ignores the changing of the "style", which yields improved generalization guarantees.
Investigation for the Multi-object Scenario. Fig- ure I systtmaticallyy shows the robustness of our pro- posed invariant concept learning when there multi- ple objects in an image. For instance, in the first case (Real Clipart) with dog and cat, our CAKE attend to the most influential pixels on one concept when the classification result is "dog/cat". However, CAKE (w/o ICL) focuses on the two animals, even if one of them is the obviously extraneous factor. The truth is that the irrelevant animal plays critical con- text in the prediction process for the trained classifier CAKE (w/o ICL.) In other words, this extraneous animal is part of the invariant context about the key concept, which may downplay the key concept fea- tures simply. In contrast, our CAKE trained from complex and changeable context with the invariant concept, so it can attend to the appropriate and unique concept even if there are two objects in the image. This provides more reliable explanation of the proposed assumption once again.
Cuso
Figure 8: t-SNE visualization of two class samples (DomainNest R + C) with their ICFs.
24

---

Under review as a conference paper at ICLR 2023
Table 14: Accuracy (%) comparison (higher means better) of IRM method and CAKE on OfficeHome and
Doma nnet datasets.
Method
R to C
to R
C to S
Mean Accuracy
A to R
R to
P to C
Mean Accuracy
IRM Arjovsky 2019
60.7
67.8
$2.3
60.27
74.7
79.7
59.0
71.3
LIRR Li 2021a
62.7
69.4
54.1
76.1
83.6
74.1
Baseline
CAKE
85.9
83.6
79.6
83.0
84.5
88.3
72.2
81.6
Real Clipart
Teapot
1st ICF
2nd ICF
3rd ICF
4th ICF
Helicopter
1st
2nd ICF
ICF
4th
Clipart = Real
Coffee
1st ICF
2nd ICF
3rd ICF
4th ICF
Castle
1st ICF
nd ICF
3rd ICF
4th ICF
Figure 6: Class Activation Maps (CAMs) of four cases which correctly predicted by CAKE Doma nNet dataset (R -> and C + R scenes).
11 LIMITATIONS
One limitation of the proposed CAKE is the training time of invariant causal factor generation. i.e., CAKE training. the training duration of cross-domain ICFG, which is responsible for generating the cross-domain style transfer samples, is long in comparison to As shown in Table 17] generating these ICFs of cross-domain style changing samples requires around 12 days, mainly due to the large-scale of the benchmark dataset of Doma nNet. Although unfavorable, this paper focuses on the importance of the causal inference for the SSDA task. We believe that introducing the causal theoretical view into SSDA can provide new insights to the domain adaptation community. Moreover, we found that the ICFs of cross- cross-domain style samples both preserved the invariant concept shape with different styles. We believe that an alternative way only to use the simple image augmentation methods (e.g., image transformation of brightness, temperature and sharpness) to generate ICFs to measure the deconfounded effect. As shown Table, II] compared with SOTA methods CDAC et al. (2021b), PACL Li al. (2020) and our CAKE, obtained a comparable results that further verifies the validity of our method.
25

---

Under review as a conference paper at ICLR 2023
1 ICF
2nd ICF
3rd ICF
4th ICF
Castle
Intra-domain Causal Factors
Cross-domain Causal Factors
Figure 7: An example of generated intra-domain ICFs and cross-domain ICFs.
Real
Clipart
Dog & Cat
CAKE (w/o ICL)
CAKE
Dog & Cat
CAKE (w/o ICL)
CAKE
Clipart
Real
Dog & (w/o ICL)
CAKE
Dog
CAKE (w/o ICL)
Figure 9: Class Activation Maps (CAMs) of multi-object cases which predicted by CAKE (w/o ICI.) and CAKE (R - C and C - R scenes).
Table 17: Space-time complexity of CAKE
DomsainNet
Offic-Homee
Cross-domain Training
days
days
GPU Memory Cross-domaile ICRG)
Around 49GB
Around 49GB
CAKE Training
Around hours
Around 2 houns
Memocy (CDL.)
STMB
26


Reviews:
[{"review_id": "tRvul8d5iC", "reviewer": "Reviewer_HFEB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a causal framework to formalize SSDA, and theoretically explain what characteristics should a robust domain adaptation model have. They also discuss the maximal training data utilization and present a collaboratively debiasing learning framework to make use of the training data.", "summary_of_the_review": "Please refer to Strength And Weaknesses.", "strength_and_weaknesses": "Pros:\n* This paper understands the semi-supervised DA from the perspective of causality, which is novel and significant.\n\nCons:\n* This paper always tells the readers what they have done, but not their reasons. For example, in Intro, they adopt the IPW to develop the equitable pseudo-label propagator, and claim that the negative impact caused by label distribution shift can be mitigated. I think there exists a obvious gap between this technique and the label distribution shift. This type of gaps should be further clarified in the revised version.\n\n* Some previous works (e.g., [1] [2]) have studied the DA problem through the lens of causality. However, I do not find the necessary discussions about the similarities and differences between CAKE and previous works.\n\n* They use the CycleGAN to realize the cross-domain causal generation. However, the related analysis about why the images generated by CycleGAN are under causal interventions is missing. Actually, they use the CycleGAN to implicitly make the disentanglement and generation, and they do not learn the generative factors explicitly. Thus, clarifying this mechanism is very important.\n\n\nQuestions:\n* Intuitively, I cannot distinguish the cross-domain style variables and intra-domain style variables in an image. If possible, please give a specific example to explain these two types of variables.\n\n* They claim that the two types of style variables serve as the confounders. Based on the definition of confounder, it is the common causes of $x$ and $y$. However, in Figure 2, $S_I$ and $S_C$ are only the cause of $x$. Besides, they state that there exists spurious correlation between $S_I$, $S_C$ and $C$ for several times, but I cannot find this in Figure 2. I am still confused about this issue.\n\n* Some typos. For example, in Eq.1 and Eq. 2, $P(Y|do(x),D=D_T)$ and $P(Y|do(x);D=D_T)$ appear simultaneously.\n\n* In Theorem 1, I think $D=D_T$ indicates that the value of $D$ has been fixed. Why does this term, $\\sum_{D\\in\\{D_S,D_T\\}}$ , appear in the decomposition of $P(Y|do(x),D=D_T)$?\n\n[1] Kun Zhang et al. Domain adaptation under target and conditional shift. ICML, 2013.\n\n[2] Mingming Gong et al. Domain Adaptation with Conditional Transferable Components. ICML, 2016.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S6_or_JZq86", "reviewer": "Reviewer_dEun", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Focusing on semi-supervised domain adaptation, this paper considered two key points: robust domain adaptation learning and maximal cross-domain data utilization. Based on this,  a collaborative debiasing learning framework is proposed, which utilizes two complementary semi-supervised learning classifiers to mutually exchange their unbiased knowledge.", "summary_of_the_review": "The general idea of this paper is easy to understand and some theoretic and experimental analyses could support their claims. However, whether is it really applicable or acceptable for research is hard to estimate.", "strength_and_weaknesses": "Strength:\n- The causal view of SSL DA sounds good.\n- The proposed method is novel.\n\nWeaknesses:\n- Some typos in the paper.\n- The ICT module is in an unsupervised manner, which is not very matched to the semi-supervised setting.\n- The descriptions of the proposed method are not very clear. More details should be added in the supplementary.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J8m40tl5YM", "reviewer": "Reviewer_sxkD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the SSDA problem from the perspective of causality learning. It involves two step for learning, one using GAN and data augmentation to generate intermediate data and second using two semi-supervised learner to do cross-supervision for the aim of debiasing. The effectiveness of the proposed method is demonstrated on two datasets.  ", "summary_of_the_review": "This paper wraps some common techniques in some professional, appealing and smart way. The technical contribution is not enough and the evaluation is not sufficient and convincing.  I would vote for a clear rejection. ", "strength_and_weaknesses": "## Strength\n1. The proposed method is technically sound. \n2. Experiments verify the effectiveness of the proposed method. \n3. The paper is well written. It presents the proposed method in an appealing and professional way with nice figures, formulas, proofs, and proper  language.   \n\n\n## Weakness\n1. The novelty is limited. While the paper tactically presents the proposed method in an appealing way, by linking to the causality learning theory and using propositions, theorems, and proofs, the techniques in essence are quite common in this field. The so-called invariant causal factor generator is just using a GAN model to transfer labeled source image to the style of the target domain for the cross-domain causal factor, and for intra-domain causal factor using stochastic data augmentation which was used by several existing SSDA methods [1,2,3]. The collaboratively debiasing technique, i.e., using one-classier's output to supervised another classifier was also proposed before to address different problems, even for domain adaptation [4] (which is the only one I can recall at this moment, but there should be more). Linking DA to causality learning theory looks new to this problem, but it more like reiterating a well-acknowledged view in a different way that a domain can be decoupled into a domain-shared factors and domain-invariant factors. Besides, after using almost one-page trying to link DA to the causality theory, the paper concludes with the remarks that \"it is non-trivial to personally determine ...\" and \"we employ a compromise solution\", which make it doubtful for the motivation of introducing the causality stuff. In addition, SSDA is highly practical problem; it extends from UDA by adding some label supervision from the target domain to boost generalization performance. Personally I do not think many of the theorems and proofs presented in this paper are of big meaning for this practical problem. At last, the meaning of SSDA as an individual research problem is how to utilize the few labeled target samples, otherwise, SSDA can be simply reduced to a speicial case of UDA with an enlarged labeled sample set. The only technique in the proposed method that is specifically for SSDA is using two semi-supervised learners to do cross-supervision; using semi-supervised learner to learn from a mixture of labeled and unlabeled data brings little insights to help address the SSDA problem.  \n\n2. The performance is not convincing. The authors claim they \"**extensively** evaluate their method\" and \"**significantly** outperforms SOTA methods\", which I do not agree. First of all, previous SSDA methods [1,2,3 and other cited methods] usually conduct extensive experiments to verify the effectiveness and advantages over existing methods. The evaluation in this paper is far less than the previous methods, in terms of both datasets employed and backbone types. Second, previous SSDA methods usually train in one-step and can then can be deployed. The proposed method requires one extra step to train an external GAN model to generate intermediate data. The training process is more complicated, which might be ok if the results are significantly better. But as can be seen in the tables that the proposed method is only slightly better, which makes it questionable for the practical value. Last, there are other published works (e.g., [1,2]) which report better results on the employed benchmark than what are shown in the table; these works are not cited, discussed and        compared. \n\n\n- [1] Cross-domain adaptive clustering for semi-supervised domain adaptation, CVPR'21\n- [2]  Semi-Supervised Domain Adaptation with Prototypical Alignment and Consistency Learning, ICCV'21.\n- [3]  Clda: Contrastive learning for semi-supervised domain adaptation, NeurIPS'21.\n- [4] CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation, ICLR'22", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject