Improving Non-Transferable Representation Learning by Harnessing Content and Style
Authors: Ziming Hong
Zhenyi Wang
Li Shen
Yu Yao
Zhuo Huang
Shiming Chen
Chuanwu Yang
Mingming Gong
Tongliang Liu
Venue: ICLR 2024 spotlight (2024)
Keywords: non-transferable representation learning
domain adaptation
transfer learning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Non-transferable learning (NTL) aims to restrict the generalization of models toward the target domain(s). To this end, existing works learn non-transferable representations by reducing statistical dependence between the source and target domain. However, such statistical methods essentially neglect to distinguish between *styles* and *contents*, leading them to inadvertently fit (i) spurious correlation between *styles* and *labels*, and (ii) fake independence between *contents* and *labels*. Consequently, their performance will be limited when natural distribution shifts occur or malicious intervention is imposed. In this paper, we propose a novel method (dubbed as H-NTL) to understand and advance the NTL problem by introducing a causal model to separately model *content* and *style* as two latent factors, based on which we disentangle and harness them as guidances for learning non-transferable representations with intrinsically causal relationships. Speciﬁcally, to avoid fitting spurious correlation and fake independence, we propose a variational inference framework to disentangle the naturally mixed *content factors* and *style factors* under our causal model. Subsequently, based on dual-path knowledge distillation, we harness the disentangled two *factors* as guidances for non-transferable representation learning: (i) we constraint the source domain representations to fit *content factors* (which are the intrinsic cause of *labels*), and (ii) we enforce that the target domain representations fit *style factors* which barely can predict labels. As a result, the learned feature representations follow optimal untransferability toward the target domain and minimal negative influence on the source domain, thus enabling better NTL performance. Empirically, the proposed H-NTL signiﬁcantly outperforms competing methods by a large margin.

Published as a conference paper at ICLR 2024
IMPROVING NON-TRANSFERABLI REPRESENTATION LEARNING BY HARNESSING CONTENT AND STYLE
Ziming Hong",
Zhenyi Wang" Li Shen Yu Yao 4.5 Zhuo Huang"
Shiming Chen
Chuanwu Yang"
Mingming Gong 5.7
Tongliang Liu"
1+
"Sydney AI Centre, The University of Sydney UUniversity of Maryland, College Park IJD Explore Academy
*Caregie Mellon University "Mohamed bin Zayed University of Artificial Intelligence
"Huzzhong University of Science and Technology 'The University of Mclhourne
ABSTRACT
Non-transferable learning (NTL) aims to restrict the generalization of models toward the target domain(s). To this end, existing works learn non-transferable representations by reducing statistical dependence between the source and target domain. However, such statistical methods essentially neglect to distinguish be- tween styles and contents, leading them to inadvertently fit (i) spurious correlation between styles and labels, and (ii) fake independence between contents and labels. Consequently, their performance will be limited when natural distribution shifts oc- cur or malicious intervention is imposed. In this paper. we propose a novel method (dubbed H-NTL) to understand and advance the NTL problem by introducing a causal model to separately model content and style as two latent factors, based on which we disentangle and harness them as guidances for learning non- con-transferable representations with intrinsically causal relationships. Specifically, to avoid fitting spurious correlation and fake independence, we propose a variational inference framework to disentangle the naturally mixed content factors and style factors under our causal model. Subsequently, based on dual-path knowledge distilla- tion, we harness the disentangled two factors guidances for non- conttransferable representation learning: (i) we constraint the source domain representations to fit content factors factors (which are the intrinsic cause of labels), and (ii) we enforce that the target domain representations fit style factors which barely can predict labels. As a result, the learned feature representations follow optimal untransferability toward the target domain and minimal negative influence on the source domain, thus enabling better NTL performance. Empirically, the proposed H-NTL significantly outperforms competing methods by large margin.
INTRODUCTION
Non-transferable learning (NTL) (Wang et al., 2022b) was proposed as core technology in intellec- tual property (IP) protection (Zhang et al., 2018: Chakraborty et al., 2020; Le Merrer et al., 2020; Wang et al., 2024) and controllable artificial intelligence (Zhu et al., 2023; Yang et al., 2023; Li et al., 2023). The critical task in NTL is to learn feature representations (i.e., non-transferable representa- tions) that can maintain source domain performance but cannot perform well on an observed target domain (targct-specified NTL task) or any unobserved target domains (source-only NTL task).
Existing NTL methods try to resist the transferability of learned representations by reducing statistical dependence between source domains and target domains (Wang et al., 2022b; Zhu et al., 2023; Zeng & Lu, 2022). In general, they impose relaxation regularization terms standard supervised learning: (i) relaxing statistical dependence between "target domain representations" and "target domain labels", (ii) relaxing statistical dependence between "target domain representations" and "source domain representations". However, since such statistical methods are essentially difficult to distinguish the naturally entangled contents and styles in both source and target domain data, they casily inadvertently fit (i) spurious correlation between styles and labels, and (ii) fake independence between contents and labels.
For intuitive explanation, as shown in Fig. I (a), we introduce an NTL example that aims at banning knowledge transferring from the wildlife park (source domain) to the zoo (target domain). We first
CCorrespondence Tongliang Liu langliang.lu@@sydeeyccom

---

Published as a conference paper at ICLR 2024
Raturre
fatuue
donain
domain
cousal
sparions correlation
Eake
"'ndependence
domain
shared
target domain
sgsle
Eabade "Ahmap"
fabeds
"shoepp"
(a)
(h)
(C)
An exmaple: a wildlife park uses camel and sheep images captured in the wild to train a classification model which cannot be used by the zoo. (a) Source and target domain share the same content factors (i.e.. "camel" and "sheep") but different factors (i.e.. "natural wild" versus "artificial z00"). (b) Ideal NTL
Figure I:
factory
mechanism. (c) Existing statistical methods fit spurious correlation and fake independence.
show the ideal NTL mechanism in line with human consciousness and then the practical situation methods. The ideal and human-like NTL mechanism is shown in Fig. 1 (b). An NTL model should learn representations of camel and sheep in the source domain (i.e., contents) to implement correct recognition. To limit knowledge transferring, the NTL model should mine the difference between two domains, which naturally and reasonably leads to learning 200 environment in the target domain (i.e., styles). Such target-specified styles barely contain predictive information and negatively impact source domain recognition. However, in practical situation, the ideal NTL is destroyed by nuisance factors (Zhang et al., 2022: 2020) (e.g.. desert and grassland, which belong to styles and are co-occurrent with camel and sheep in the wild, respectively). As shown in Fig. 1 (c), since lacking the ability to distinguish the naturally entangled contents and styles, a statistical-basee method will be misled to learn desert/grasslane in the source domain and camel/sheep in the target domain to satisfy the non-transferable requirements. Indeed, knowledge cannot be transferred to the target domain in this case. However, the NTL model (i) fits the spurious correlation from styles labels in the source domain (i.e., desert label "camel". grassland to label "sheep") "' and (ii) models the independence from contents to labels in the target domain (they should have causal relationships, and thus, we call as fake independence). Consequently, the learned non-transe co-transfferabe representations are fragile and sensitive to small distribution shifts (from a training set to testing set). Thus, both the source domain performance maintenance and target domain degradation will be limited. More seriously, when malicious attackers impose interventions (e.g., domain masquerade attack), such ttaistical-basse methods may not work well since they only fit the superficially statistical relationships in the training data'.
in using cesssiiaa-deeenddnne-rrraaae
In this paper, we propose novel method to address the aforementioned problems by Harnessing content and style in Mon-Transferabee representation Learning, dubbed as H-NTL. We use a causal model (Peters et al., 2017: Li & Chu, 2023) to formalize the data generation process and mathemati- cally understand the non-transferable mechanism with optimal untransferability. In order to avoid be- ing misled to fit spurious correlation and fake independence, we distinguish the unobservable and nat- urally mixed contents and styles by separately modeling them in our causal model. As shown in Fig. 2, we decompose an instance X into two latent factors: (i) content factor C, which corresponds to the intrinsic, class-related information that is a cause of the label Y in both the souce and the target do- main (e-g.. bird, airplane, deer, etc.), and (ii) style factor S, where all other factors except the content factor C' can belong style factors (such as image background or artificial watermark (Wang et al., 2022b)) and
S
D
is the cause of domain D² The content factor C' and the factor S are causes of the instance X. Besides, we assume the co-occurrence be- tween contents and styles (which leads to the problems of fitting spurious correlation and fake independence in existing NTL methods) is caused by latent confounders, and thus, we model it by a statistical dependence" between the style factor S and the content factor C' in our causal model (i.e., the dashed line in Fig. 2) (Von Kügglgeen et al., 2021). Based on the causal model, we propose a novel variational inference framework to disentangle the content factor C' and factor S from observable data by maximizing an approximated evidence lower-bound (ELBO) (Blei et al., 2017; Yao et al., 2021) of the joint distribution P(X,Y.D)
X
Figure 2: A causal model to reveal the data generation process in NTL. The gray- shaded variables are observ- able and those unshaded are unobservable. The dashed line indicates the statistical dependence caused by la- tent confounders.
'In Appendix A, we present a toy experiment to demonstrate the vulnerability of statistical-based methods. We further discuss its rationality in NTL and consider another direction (i.e.. causes S) in Appendix E.2. Detailedd discussion and assumption of the dependence hetween S and C' is provided in Appendix E.
2

---

Published as a conference paper at ICLR 2024
Subsequently, by following the ideal (human-like) NTL mechanism, we harness the disentangled content factor C' and style factor S as guidances for non- non-transferable representation learning. Based on knowledge distillation (Gou et al., 2021). we treat the NTL model frut student network and train it by two distillation paths: (i) in the source domain path, we constraint the learned representation to fit content factor C' (which is the intrinsic cause of label Y), and (ii) in the target domain path, we enforce the learned representation to fit style factor S which barely can predict labels", Therefore, the non-transferable representations learned by Juu simultaneously approach optimal untransferabillty toward the target domain and minimal negative influence on source domain recognition, thus enabling better NTL performance. Our contribution can be summarized as follows:
We identify the problems of fitting (i) spurious correlation between styles and labels and (ii) fake in- dependence between contents and labels in NTL, which are caused by the statistical dependence between contents and styles. Such problems lead to the learned nonttransferable representations in existing methods become fragile and sensitive to small distribution shifts.
To address the existing problems, we distinguish the unobservable and naturally mixed contents and styles by separately modeling them latent factors in causal model. Further, we explore causal-driven non-transferable representations with the optimal untransferability, based on which, we propose a simple yet effective method (dubbed as H-NTL) to disentangle and harness the content factor and style factor for non- non-transferable representation learning.
We conduct experiments to evaluate the proposed H-NTL. method both target-specific and source-only NTL tasks. Empirical results across various settings show the superiority of H-NTL.
2 RELATED WORK
Non-transferable learning. NTL aims to restrict the generalizability of learned models toward cer- tain domain (target-specified NTL) or all other domains (source-only NTL), which will be widely applied in intellectual property protection (i.e.. "ownership verification" and "applicability authoriza- tion") and controllable artificial intelligence (Zeng & Lu. 2022; Wang et al., 2022b; Zhu et al., 2023). The most related research fields to the NTL are domain adaptation (DA) (Garg et al., 2022; Saito et al., 2020; Huang et al., 2021; Oza et al., 2023) and domain generalization (DG) (Rame et al., 2022; Zhou et al., 2022; Huang et al., 2023b;a). Specifically, the target-specified NTL is opposite to DA, which aims to enhance the performance on a certain target domain. Similarly, source-only NTL can be seen as an anti-task to single-domain DG. Existing NTL methods try resist transferability by reducing statistical dependence between representations on source and target domains. Wang et al. (2022b) first propose the NTL task. They design framework that adds two statistical dependence relaxation terms on standard supervised learning: (i) maximizing the Kullback-L (KI.) divergence between target domain representations and labels, and (ii) maximizing the maximum mean discrepancy (MMD) be- tween the distribution of source and target domain representations. Zeng & Lu (2022) aim at NTL in natural language processing. They use MMD as well as an auxiliary domain classifier to enlarge the distance of representations from different domains. Further, Zhu et al. (2023) enhance the NTL performance by a domain domain-weighted MMD strategy. However, such statistical based methods are easily trapped into fitting spurious correlation and fake independence, which leads to the limitation of both source domain performance maintenance and target domain degradation.
Causal-inspires representation learning. Despite the success of statistical learning, they always provide rather superficial description of limited reality, which means that such success only holds when the experimental conditions are fixed (Schölkopf et al., 2021). In practice, statistical methods are easy fit spurious correlation from image background to class labels (Zhang al., 2022: Lv et al., 2022; Ye et al., 2023), which leads to serious performance degradation when facing non-stationary data distribution. Such shortcoming is always reflected in the adversarial vulnerability (Yu et al., 2022; Zhang et al., 2022) and poor generalizability toward unseen domains (Kong et al., 2022; Wang et al., 2022a). Causal-inspired representation learning is a promising approach to address these concerns. It aims to learn representations that are invariant under other changing causal factors, leading to improved performance in the presence of domain shifts (Huang et al., 2022; Mitrovic et al., 2021). Specifically, mausal-inspires DA (Kong et al., 2022; Jiang & Veitch, 2022) and DG (Lu et al., 2021; Liu et al., 2021; Chen et al., 2022; Sheth et al., 2022; Chen et al., 2023a) focus on learning invariant representations across different domains to obtain better generalizability. On the contrary, our causal-inspired NTL method aims at learning representations without generalizability in a certain domain(s), which means that the feature representations are "variant" between different domains.
We assume weak relation between S and C' in the target domain, which will be empirically verified.
3

---

Published as a conference paper at ICLR 2024
3 METHOD
As shown in Fig. 3. we proposed a H-NTIL method to handle the non-transferable representation learning problem. In Section 3.1, we first illustrate the ideal tratssaanfeeeeee mechanism based on our causal model. In Section 3.2, we design a variational inference framework to disentangle the unobservable content factor C' and style factor S. In Section 3.3, we introduce dual-path knowledge distillation to harness the disentangled factors as guidances to teach a student network to learn ideal non-transferable representations. In Section 3.4, we illustrate the overall training process of H-NTL for target-specified and source-only NTL tasks, respectively.
Notation. As shown in Fig. 2, we assume that an instance X in domain with label Y has two unobservable cause variables: content factor C' and style factor S Particularly, X, Y, and D are observable variables, and we can access samples from their real joint distribution, namely source domain data D. = { v. Ji=1 and target domain data D; = {zi, v' We merge these two training sets to whole dataset 2> and represent it D D. D. {x;, y/, 1=) N,+,, where y/ is the content label of sample 2; with index i, and d; is the domain label. The goal of NTL is to train a classification model fnut= X - Y to degrade performance maintain performance on source domain D.. Without loss of generality, the faxt can be splited as a feature extractor fe and classifier feis (i.e., fnut fel fe).
target domain D, and simultaneously
3.1 CAUSAL-DRIVEN NON-TR NON-TRANSFERABL REPRESENTATION
In this section, to explore the ideal cronfraanfeeeblle mechanism in line with human consciousness (i.e., following intrinsic causal relationship), we provide a causal view to understanding the non- transferable representation learning. Based on the causal model in Fig. 2, the data generation process from unobservable variables to observable variables can be formalized as:
X = C,), D ca), Y = f,,,,,
(1)
where e represents the noise term of the corresponding variable, and they are independent of each other and usually can be ignored. Each represents the corresponding data generation function. Based on Eq. (1), we discuss the non-transferable representations on source/target domain, respectively.
Source domain representations. In the source domain, we start from the stable human cognition which relies on the ability of causal reasoning (Zhang et al., 2020; Schölkopf et al., 2021). In order to maintain high classification performance on the source domain, the NTL model fau should follow the intrinsic causal relationship from content factor C' to the label Y in Eq. (1), Specifically, the feature extractor fo in fau should model the latent content factors (i.e., fe(X.) C), and the classifier module fata is required to establish mapping from content factors to labels, i.e., fos (C) = Y.
Target domain representations. The next, and more important matter we focus on representations on the target domain. The crucial problem in non-transferable mechanisms is how to implement untransferability toward the target domain and simultaneously minimize harmful effects on source domain performance (i.c., optimal untransferability). For this purpose, we first explicitly define the properties of optimal untransferability as follows:
Definition 1 (Optimal Untransferabilityy. Let N denote a kind of representations across source domain D2 and target domain Dr. where N₂ and N, represent source and target domain feaain representations, respectively. Let Y denote the classification labels. If N satisfy:
N, and N₂ Y.
(2)
we say that the representations N have the optimal untransferabilliy from source domain D, to target domain Dr where 1L means the statistical independence.
In Definition 1, the two independent terms simultaneously ensure that (i) the target domain represen- tations contain no information to predict labels (i.e., N₂ IL Y) which equals to the untransferability toward the target domain, and (ii) no harmful effect toward source domain prediction (i.e., N₂ IL N.).
Remark 1. We consider the source domain representations mentioned above, and follow the data generation process in Eq. (1). By learning representations N to achieve N, C. on the source domain and N, S on the target domain, the N approximately satisfy optimal untransferability.
"We use subscript a and i to denote the variahle on the source and target domain, respectively. For example, X, and C, mean source domain instance and target domain content factor, respectively.
4

---

Published as a conference paper at ICLR 2024
Engoder fe.
C
Classifier Plr,
Y
X
Encoder 8o-
C2
Classifier At:
Decoder Pu,
X,
Encoder lo
St
Knowledge Distillation
Engoder So.
Classifier Pa,
D
NTL model Iust
(a)
(b)
Figure 3: The proposed H-NTL. (a) Disentanglement VAE, (b) Dual- patl-pathhhhh knowledge distillation.
Intuitively, Remark provides a principle for target domain representation learning, that is, fitting the style factors of the target domain (i.e., fX,) S2). Under the weak-relation assumption" 4, the style factor barely can predict labels although the statistical dependence is existing between C' and S. Thus, by fitting style factors of the target domain, the optimal untransferability toward the target domain can be approximately satisfied under the causal-driven representations in the source domain.
3.2 DISENTANGLEMENT UNDER THE CAUSAL MODEL.
Although the causal-insping understanding of the con-transferable mechanism provides a promising direction for practicing this task, both of the content factor C and the style factor S are unobservable and are naturally entangled in instance X. To leverage them for non-transferable representation learning, we have to infer them with the source and target domain datasets D. DD sampled from the joint distribution P(X, Y. D). In order to simultaneously infer content factor C and style factor S from sampled data D, Dr, according to the causal model in Fig. 2, factorize the joint distribution P(X, Y, C,S, D) under the Markov condition (Pearl et al., 2000; Yao et al., 2023) as follows:
Y, D, P(C, PYY|C P(DIC, S)PXICC))
(3)
where we the C and S variable set with the joint distribution P(C, due to the statistical dependence caused by latent confounders. The Eq. (3) motivates us to extend variational auto-encoder (VAE) (Kingma & Welling, 2013) to infer latent factors, shown in Fig. 3 (a). Specifically, separately infer C' and S from the input X by using two encoder modules 1o. and dc. to model the posterior distributions 4s. (CIX) and go. (S| X) with learnable parameters &. and Φ.,, respectively. In addition, we use a decoder Po_ to model the distribution Po., Po,, (XIC, parameterized by 0z. Moreover, we approximate P(Y S) by assuming that C should contain sufficient information about Y (Yao et al., 2021; Lu et al., 2018), i.e., P(Y|C, = P(YIC), and such assumption is also applicable to P(DIC, S). Therefore, we use two classifiers (Po, and po,)) ) to model the distributions Pe, (DIS) and pe, (YIC) with learnable parameter 0g and 0g- Totally, the VAE can presented as a set of network modules: (Go, (o, Po Po.)
To learn the parameters (dc. d.. 0z fy. 02), we follow the variational inference framework (Blei et al., 2017) to maximize the evidence lower-bound bound (ELBO) from the sampled observed data (x. 3/ d) E D. The ELBO( y, d) is derived as follows (the detailed derivation is shown in Appendix B):
ELBO(z, y, d) KL(9s. (clx))lp(()) - KKLgg, (s|=)]lp(S)) (e-ggs (clax) [log Po, (ylc)
+ +Eg-qs (s|=;) [log Po. (d[s)] Ex-ns, (c (s|x) [log Po. (#|c, s)]
(4)
where KL(-Il-) represents the Hellback-Leiblers (KL) divergence. The detailed derivation of Eq. (4) is
shown in Appendix B. The first two terms represent the KL divergence between the estimated factors and their prior distributon, i.e., multivariate Gaussian distribution (N0 I) (Kingma & Welling, 2013).
The third and fourth terms in Eq. (4) are the approximated lig-liielsooods of class label prediction from the estimated content factor C' and domain label prediction from the estimated style factor S, respectively. The last term can be seen as a reconstruction loss from the estimated latent variables C' and S to the input data X. We train the VAE by maximizing ELBO(x; x. y, d) and thus obtain two encoders & (x) and q,, (x)) for disentangling and estimating unobservable content factor C and style factor S, respectively.
Intuitively, by maximizing ELBO(2, y, d) in Eq. (4), we enforces images with the same label, regardless of their domain, have similar learned content factors C. This ensures that the learned C captures class-related information, and disentangle other style factors from contents. At the same time, we enforces images from different domains to have distinct learned style factors S, while images from the same domain share similar style factors. Thus, the VAE will learn the style factor S that captures the stylistic differences across different domains.
5

---

Published as a conference paper at ICLR 2024
3.3 DUAL-PPAI KNOWLEDGE DISTILLATION
After the disentanglement of content factor C and style factor S, we leverage them as guidances for subsequent non- transferable representation learning. shown Fig. (b), in order to satisfy requirements source and target domains mentioned in Section 3. we train the NTL network Jru in dual-path knowledge distillation paradigm where the Jau is seen as a student network.
Path for source domain. In the source domain, we follow the causal direction from content factor to label, i.c., C' - Y. means that the NTL model fnct = fets o fc should learn content factors through the feature extractor fe. and use the classifier fets to establish mapping from feature to label, i.., (C) = Y. The loss of the source domain path can be represented as follows:
arc Lkd
L2(Sne(( (3,,, a De. (āp. (2r))),
(5)
where L2 is the mean square error loss and F, is source domain image. In Eq. (5), the learning tar- get is the output logits of the classifier Pa_. with the estimated content do. (2.) serving as input.
Path for the target domain. In the target domain, the aim is to learn representations of factors which barely can predict labels. We present the distillation loss of the target domain path as follows:
Ligh (xz)) Po_ (âs, (xx)));
(6)
where ze is an image sampled from the target domain. Here the learning target is the classifier po. prediction logits of the estimated style factor ds, (x+).
Total knowledge distillation object. To satisfy both requirements for the source and the target
domain, we combine Lid and L to the overall knowledge distillation optimization object Lia
Lid = L + Catt
where A is a hyperparameter to balance the two loss terms. By minimizing the Eq. (7), the repre- sentations learned by the student model fau will follow the optimal untransferability to the target domain as well as minimal negative influence toward the source domain. Intuitively, in such way, given a new source domain image, fru aims to predict its content factor, which is directly related to classification. Given a new target domain image, fiet aims predict its style factor, which is designed to be the image style that differs across domains and is useless for classification.
3.4 OVERALL TRAINING PROCESS OF H-NTL
Algorithm 1 Train target-specified H-NTL
We present the overall training process of H-NTL in this section. Specifically, the NTL task is categorized into target-specified NTL and source-only NTL ac- cording to whether the target domain is known.
1: Training set DD = {Ir, Vs. d:) No*N, Dis- entanglement epoch Edit. KD epoch End- VAE modules (āa, 90,, :oo Poy, .Po.) and the NTL network fnu--
Target-specified NTL task. As shown in Algo- rithm 1, we use both the accessible sampled source domain data D, and target domain data D, to train the VAE and NTL network. For the first Edis epochs, we train the VAE modules through maximizing ELBO
2:
for := 1 to Edis do
3: Train {âo, 96. Pe.,, Po,, pest by maxi-
mizing ELBÖ in Eq. (4);
4:
end for
5: for i= 1 to FiRd do
6: Train fwu through minimizing Eq. (7);
in Eq. (4). For the later Fed epochs, we train the NTL network fnd through minimizing Eq. (7), and partic-
7: end for
ularly, we freeze all the parameters of VAE in this stage to extract the content factor C2 from the coming source domain data x, and style factor S, from the target domain data Zt.
Source-only NTL task. In the source-only task, we only access the sampled source domain data D.. Without loss of generality, we represent the source domain as un-distribution (ID) data and other domains with distribution shifts (including the unseen target domain) as out-of-distribution (OOD) data. Source-only NTL task focuses on degrading the recognition performance for those OOD data with the same contents but different styles (Wang et al., 2022b). Unlike the domain augmentation method proposed in Wang et al. (2022b) where generative adversarial network (GAN) is used to synthesize OOD data and make the source-only NTL feasible, we achieve this goal by imposing interventions on the style factor (i.e., do(S)) (Von Kügelgen et al., 2021; Mitrovic et al., 2021). Due to the independence of mechanisms (Peters et al., 2017), do(S) do not impact PCYIC). which means that manipulating the value of does not change the content factor C. Thus, the OOD data obtained by interventions meet the requirements in source-only NTL. scenarios. In practice

---

Published as a conference paper at ICLR 2024
Table I: Comparison on target-specified NTI. tasks. For each table cell, the first line shows averaged accuracy Acc (%) with standard deviation, and the second line shows accuracy drop (hchind 1) and relative drop A % (in brackets) compared to supervised learning (SL). The hest results (except SL. of "sounce/targer domain perfor- mance" and "performance difference between domains" are highlighted in "underline" and "bold", respectively.
Source Target
Img Size
SL
INTL (Wang et 2022b)
H-NTL (Ours)
Source
Target
Source
Target
Source
Target
94,30 10.79
97.47 10.40
88.33 1:125
8.47 12.29
23.10 10.96
9,90 10.60
MM-MT
32
1.997 (6.33%)
(91.3158)
1.20 (1.27%)
87.57 (89.54%)
87.47 10.55
50.33 15.32
81.33 1: 66
10.27 10.81
88.13 11.43
9.23 11.35
SN-SD
32
6.144 (7.02%)
40.06 (79.59%)
-0,66 (-0.75%)
41.11
98.23 10.06
55.30 13.00
89.50 66
9.20 :1.01
97.13 10.35
10.97 10,83
SD-MT
32
1.7.3 (8.89%)
46,10 (83.86%)
1.10 (1.12%)
1 44.33 (80.16%)
81.10 20.20
61.60 11:32
79.77 1:132
49.60 1:430
80.63 10.84
28,10 1:4.6
1.33 (1.64%)
12,00 (19.48%)
0.47 (0.58%)
J 33,50 (54.38%)
CID-S10
86.57 #0.38
67.60 +0.95
86.53 +1.61
9.93 +0.90
87. =10.26
9.63 +1.50
64
0.04 (0.059)
57.67 (85.
-1.03 (-1.19%) %
57.97 (85.75%)
89.67 +0.80
22,43 #3.56
88.47 +0.21
8.27 #0.81
91.73 +1.12
8.07 +0.71
32
(1.34))
L-2006 (-2.30 %
14.36 (64.02%)
VT-VV
64
93.40 +0.70
93.33 +0.35
8,47 +0.92
94.60 +00.4
+0.92
0:07 (0.07%)
27.13 (76.215%)
1 1-1.30 (-1.28 %
27.40 (76.97%)
65.57 +1.27
23.57 +1.01
62.80 +1
15.90 +0.20
65.60 +0061
5,50 +0.98
32
2,7.7 (4.228)
4 7.67 (32.548)
1-1M13 (-0.05 %
J 18.07 (76.6%))
OP-OC
64
75.60
+1.47
31.63 +1.02
74.13 +0.06
22.33 +0.40
76.43 +0.91
6.73 +0.67
4 1.44 (1.94%)
9.9.0 (29.44))
(-1.0%)
24.9 (78.72%))
considering the fact that we cannot obtain naturally entangled content factor C and style factor S from a single source domain and subsequently manipulate the style factor value, we perform image style augmentation (Sohn et al., 2020; Berthelot et al., 2019; Cubuk et al., 2020) on source domain data. Image style augmentations (c.g., blurring, sharpness, solarize) do not influence the contents but significantly change the image styles, thus satisfying our intervention aims Refer to Appendix C.4 for detailed style augmentations. We see all augmented images as the target domain, and then we use the accessible source domain and the target domain to train the H-NTL in the same way as uurget-specifent NTL. The full algorithm of source-only H-NTL is shown in Appendix C.5.
EXPERIMENTS
We conduct a series of experiments to evaluate the effectiveness of our H-NTL, and compare it with supervised learning (SL) and the NTL method proposed by Wang et al. (2022b) (tNTL and sNTL )) Our experiments involve three basic digit tasks and three challenging tasks on real-world datasets. The digit tasks contain three random-seled seleeted pairs from four digit datasets: MNIST (MT) (Deng, 2012). MNIST-M (MM) (Ganin et al., 2016), SVHN (SN) (Netzer et al., 2011) and SYN-D (SD) (Roy et al., 2018). For challenging tasks, we involve CIFARIO to STL/OO (Coates et al., 2011) 1) (C10-$10), VisDA (Peng et al., 2017) (VT-VV) and OfficeHome (Venkateswara et al., 2017) (OP-OC). We resize images to 32 x 32 on digit tasks and both 32 32 and 64 x 64 for challenging tasks. We follow the backbones in (Wang et al., 2022b) to conduct all experiments. To evaluate, we present top-1 accuracy (Ace) on both source and target domain. Results are reported as average with standard deviation over three independent runs. Additionally, we calculate the averaged accuracy drop (A Acc.. - Acc,ut) and relative drop (A% Accc. on source/target domain, respectively. More implementation details (e.g., datasets, network architecture, training details) are given in Appendix C.
4.1 EXPERIMENTS ON TARGET-SPECIFIED NTL TASK
Comparison with baselines. We compare the proposed H-NTL with baselines (SL and (NTL), and the results are shown in Table 1. H-NTL outperforms tNTL regarding the difference of source-targe domain performance on all datasets. For MM-MT and MT, tNTL has better target domain performance degradation (91. (01.31% % and 83.36%) compared to H-NTI (89.84% and 80.16%) but tNTL also impacts more source domain performance, with the relative drop A% achieving 6.33%
'In order to avoid confusion hetween the method name and the task name, we here use tNTL. and sNTL. to separately denote their method for target-specified NTI. task and source-only NTI. task.
7

---

Published as a conference paper at ICLR 2024
Table 2: Comparison source-only NTI. tasks. For each table cell, the first line shows averaged accuracy Ace (%) with standard deviation, and the second line shows accuracy drop A (hehind 1) and relative drop A% (in brackets) compared to supervised learning (SL). The best results (except SL) of ssource/target domain perfor- mance" and "performance difference between domains" are highlighted in "underline" and "bold", respectively.
Source Target
Img Size
sNTL (Wang al., 2022b)
H-NTL (Ours)
Source
Target
Source
Target
Source
Target
94.30 10.79
97:47 10:40
89.67 11:24
22.93 12.37
92.10 10.85
13.43 1341
MM-MT
32
(4.11%)
74.54 (76.4796
2.30 (2.33%)
$4.04 (86.22%)
87.47 10:55
50.35 15.32
86,20 10.40
18.60 11.59
86.70 12.26
10.70 1147
SN-SD
32
1.227 (1.45%)
31.73 (63.04%)
0.77 (0.58%)
39.63 (78.74%)
98.23 10.06
55.30 13.00
96,27
14.40
95.17
10.47 10.96
SD-MT
32
1 1.96 (2.00%)
(73.993)))))
3.06 (3.12%)
44.83 (S1.0%%)
81.10 10200
61.60 11:32
83,13 10.81
63.50 10.75
76.30 12:98
52.70 11.90
32
2.00 (-2.50%)
11.90 (-3.08%)
4.50 (5.92%)
8.90 (14.44%))
CIO-510
86.57 40.38
67.60 +0.5
84.67 +0.99
47.23 +3.99
86.63 +2.08
9.80 11.11
64
1.90 (2.195)
20.37 (30.138))
-0.06 (-8.07%)
57.80 (85.50%)
89.67 40.0
22.43 #3.56
91.70 +0.70
17.53 +1.46
91.63 +1.05
9.27 +1.30
32
1-0003 (-2.26/6)
4.900 (21.85))
-1.96 (-2.19%)
13.16 (58667))
VV-
93.40 40.30
35.60 +1.56
91,93 +0.93
18.37 +1.00
94.93 +0.57
7,80 +0.35
64
1 1.44 (1.575)
17.22 (48, (48.4096)
1-1.53 (-1.64%)
27.80 (78.099)) %
65.57 +1.27
23.57
60.73 +0.45
19.13 +0.50
63.73 +0.12
14.43 +0.95
32
1 4.84 (7.38%)
4.44 (18.849))
1.34 %))
9.14 735
OP-VOC
75.60 +147
31.63 +100
71.20 +0.87
26.97 40.78
71.63 +1.84
16.00 +2.55
64
4.40 8282)
(14.73%)
3.97 (5.25%)
15.63 (49.42%)
(NNMpi wHNNLOo st-Nilupe)
30
(%)
Acoray
60
60
40
40
20
40
100
50
96
100
Pazch Value
Pasdi Value
(a) C10 + C10 w/ Patch
(b) S10 - S10 w/ Patch
Figure 4: Results of target-specified NTL on watermarked data, where the "(sre)" and "(tgt)" behind each method name denote the accuracy on source and target domain, respectively.
and 8.89% respectively. For challenging tasks with low-resolution images and more categories (e.g.. OP and C10-$10 with 32 32 image size), thanks the disentanglement of content factor and style factor and dual-path knowledge distillation, H-NTL can still Iearn effective non- non-transferable representation and achieve source domain maintenance and target domain degradation, while (NTL fails in both aspects. Overall, these results demonstrate the effectiveness of the proposed H-NTL in target-specified NTL.
Target-specified NTL on watermark data. We also conduct target-specified NTL experiments on watermark data, in which we use pixel-level mask patches as watermarks and add them to the original source domain data to form target domain (Wang et al., 2022b). To test the ability of learning non-transferable representations, we conduct experiments under different patch values. The value reflects the degree of distribution shifts from the original source domain to the target domain with human-added watermarks. It is more difficult for model to learn non-transferable representations with the patch value decreasing. The results on C10 and $10 are shown in Fig. 4. On either C10 or S10, the target domain performance degradation of INTL is gradually limited, with the patch value decreasing. In contrast, H-NTL is slightly affected (S10) or even not affected (C10) by the patch value dropping. This is because the disentanglement and dual-path knowledge distillation in H-NTL can better distinguish and guide the fnu to content factor on the source domain and style factor on the target domain, even only slight distribution shifts existing between source and target domain.
4.2 EXPERIMENTS ON SOURCE-ONLY NTL TASK
The results of source-only NTL tasks are shown in Table 2. For all datasets, the proposed H-NTL outperforms sNTL (Wang et al., 2022b) on the evaluation metric of difference on source-tare target domain performance, which reflects the overall advantages of the proposed H-NTL as well as the
8

---

Published as a conference paper at ICLR 2024
Table 3: Ablation Studies. For target-specified NTL. and source-only NTI,, the best results of "difference on caurce-target domain performance" are highlighted in "bold".
C10-$110
VV-VV
OP-OC
Task
Method
Source
Target
Source
Target
Source
Target
SL
86.57
67.60 20.55
93.40 =0.70
35.60 11.56
75.60 =1.47
31,63 #1.02
H-NIL
10.80
8.87 +0.50
8.17 +0.25
1.17
target-peciheed
w/o C
75.77 (87.529%)
4. 57.885 (85.5598)
84.53 (30.509%)
27,43 (77.05%8)
4 77567 (97.45%)
30.46 (96.309)
H-NIL
87.90 +0,44
72,40 +1.30
94.80 -0.36
36.20 +0.53
77.60 +1.82
39.10 +1.14
wAO S
1 (-1.559))
1-,,00 (-7.10%)
1 [-1.40 (-1.509))
40.60 (-1.098)
1-2000 1-2.00 (-2.658)
4-7.47 (-23.6256)
H-NIL
87.60 +0.26
9.63 +1.50
94.60 -0.44
8.20 +0.92
76.43 +0,91
6.73 +0.67
(full)
(-119%)
57.97 (85.75%)
-1.20 (-1.88%)
27.40 (76.97%)
-0.3 (-1.10%)
(78.72%)
H-NIL
10.80 +1.01
9.87 +1.21
9.17 +0.67
8.60 +1.45
2.40 +0.17
1.40 40.52
w/o C
75.27 (87.528))
57.75 (85,4098)
84.23 (90,
27,00
7,7220 (96.83%)
30.23 (95.5755)
SOHrce-omb
H-NIL
87.93
66.07 +2.38
95.77
24.23
+0.42
31.60
w/o S
1-1.3 (-157%)
1 (..362)))%)
1 1-2.37 (-2.64%)
11,37 (31.98))
1-1.33 (-1.76%)
0.00 (0.09%)
H-NTI.
86.63 +2,08
9.80 11.11
94.93 =0.57
7.80 20.35
71.63 1184
16.00 12.55
(Fall)
-0.06 (-9.07%)
57.80 (85.50%
1 1-1.53 (-1.64%)
27..00 (78.09%)
1 3.97 (5.25%)
15.33 (49.42%)
effectiveness of our style augmentation strategies for the source-only task. On SD-MT and VT- (32 x 32), the source domain accuracies of sNTL slightly exceed our H-NTL (1.1% and 0.07%). However, regarding the degradation of target domain accuracy, H-NTL outperforms sNTL by large margin (3.93% and 8 8.26% For the task of C10-S10 (32 x 32). due to the low resolution of images, both sNTL and H-NTL are hard to learn effective non- -onttraanfereale representations when only the source domain is available. Owing to the disentanglemenn, H-NTT. can still capture more effective content factor and style factor and learn better non- non-transferable representations, with the difference on cource-target domain performance exceeding sNTLL 3.97%.
4.3 ABLATION STUDIES
In this section, we explore the effectiveness of main components in our H-NTL. We conduct ablation studies on three paired datasets both target-specified and source-only tasks: C10-S10, VT-VV, and OC-OP, with images resized to 64 x 64. As shown in Table 3, without the guidance of content factors in the source domain (i.e., H-NTL w/o C'), both the source and target domain performance will drop to random classification accuracy (either target-specified guidance of style factors in target domain (i.e., H-NTL w/o S), the source and target domain accuracies exceed the accuracies of supervised learning (SL). This is because the student NTL network fuu fits the disentangled content factors from the paired source-target domain (in target-specified NTL) or original source and augmented source domain (in source-only NTL), which leads to better generalizability. Only with the full guidance from content and factors, i.e., H-NTL (full), the non- -onttransfeealle representation learning can be successfully achieved.
source-only task). Without the
comparable to
4.4 ADDITIONAL EXPERIMENTS AND ANALYSES
Due to the page limitation, present additional experiments and analyses in Appendices, which include but are not limited to a toy experiment (Appendix A), additional experiments on high- resolution images (Appendix D.1), IP protection (Appendix D.2), more baseline (Appendix D.3), the influence of KD loss weight (Appendix D.4), the influence of style augmentation number (Appendix D.5), disentanglement analysis (Appendix D.6) and visualization results (Appendix D.7).
5 CONCLUSION
In this paper, we propose a H-NTL method handle the NTL problem. To understand the non- transferable mechanism, we introduce a causal model to formalize the data generation process, in which we separately model the unobservable contents and styles by two latent factors. Then, we propose a variational inference framework to disentangle the naturally mixed content factor and style factor from observable data. Furthermore, we harness the disentangled two factors as guidances for learning non-transferable representations. By dual-path knowledge distillation, we enforce the source domain representations learned by the student model to content factors and the target domain representations to fit style factors. The learned representations follow optimal untransferability toward the target domain and minimal impact on the source domain, thus boosting the NTL performance. Extensive experiments are conducted validate the effectiveness of H-NTI..
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
Li Shen is supported by STI Major Projects (No. 20212D0201405), Ziming Hong is supported by JD Technology Scholarship for Postgraduate Research in Artificial Intelligence No. SC4103. Mingming Gong is supported by ARC DE2 10101414 Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, IC190100031.
REFERENCES
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In International Conference on Learning Representations (ICLR), 2019.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, (8X518))859887, 2017.
Abhishek Chakraborty, Ankit Mondai, and Ankur Srivastava. Hardware-assisted intellectual property
protection of deep learning models. In 2020 57th ACM/IEEE Design Automation Conference (DAC). pp. 1-6. IEEE, 2020.
Jin Chen, Zhi Gao, Xinxiao Wu, and Jiebo Luo. Meta-causal learning for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
Shiming Chen, Wenjin Hou, Ziming Hong, Xiaohan Ding, Yibing Song, Xinge You, Tongliang Liu, and Kun Zhang. Evolving semantic prototype improves generative zero-shot learning. In International Conference Machine Learning (ICML), pp. 4611-4622 PMLR, 2023b.
Yonggiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, MA Kaili, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for -iitt--diittiuutionn generalization on graphs. In Advances in Neural Information Processing Systems (NeurIPS), pp. 22131-22148, 2022.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intel- ligence and Statistics (AISTATS), pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 248-255. Iece, 2009.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 1416)11 2012.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adversaria training of neural networks. The Journal of Machine Learning Research, 17(1):206-2030, 2016.
Saurabh Garg, Sivaraman Balakrishnan, and Zachary Lipton. Domain adaptation under open set label shift. In Advances in Neural Information Processing Systems (NeurIPS), pp. 22531-22546,, 2022.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International Journal of Computer Vision, 129:1789-1819) 2021.
Ziming Hong, Shiming Chen, Guo-Ser Xie, Wenhan Yang. Jian Zhao, Yuanjie Shao, Qinmu Peng, and Xinge You. Semantic compression embedding for generative zero-shot learning. In Proceedings of the Thirty-Firs: International Joint Conference on Artificial Intelligence (IJCAI), pp. 956-963, 2022.
10

---

Published as a conference paper at ICLR 2024
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. In Advances in Neural Informa- fion Processing Systems (NeurIPS), pp. 3635-3649) 2021.
Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing out-of-distriuttion examples via augmenting content and style. International Conference Learning Representations (ICLR), 2022.
Zhuo Huang, Muyang Li, Li Shen, Jun Yu, Chen Gong, Bo Han, and Tongliang Liu. Winning prize comes from losing tickets: Improve invariant learning by exploring variant parameters for out-of-distribution generalization. arXiv preprint arXiv:2310. 16391, 2023a.
Zhuo Huang, Miaoxi Zhu, Xiaobo Xia, Li Shen, Jun Yu, Chen Gong, Bo Han, Bo Du, and Tongliang Liu. Robust generalization against photon-limited corruptions via worst-case sharpness minimiza- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 16175-16185, 2023b.
Jonathan Hull. A database for handwritten text recognition research. IEEE Transactions on Puttern Analysis and Machine Intelligence, 16(5):550-55) 1994.
Yibo Jiang and Victor Veitch. Invariant and transportable representations for anti-causal domain shifts. In Advances Neural Information Processing Systems (NeurIPS), 2022.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14,161111222 2013.
David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations (ICLR), 2021. URL
https://cpenreviii
net htte://ooommddddddeeeeeee Lor.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning (ICML), pp. 11455-11472. PMLR, 2022.
Erwan Le Merrer, Patrick Perez, and Gilles Trédan. Adversarial frontier stitching for remote neural network watermarking. Neural Computing and Applications, 32:9233-922 -2444, 2020.
Bo Li, Peng Qi, Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. Trustworthy ai: From principles to practices. ACM Computing Surveys, 55(9):1-46 2023.
Sheng Li and Zhixuan Chu. Machine Learning for Causal Inference. Springer Nature, 2023.
Yexiong Lin, Yu Yao, Xiaolong Shi, Mingming Gong, Xu Shen, Dong Xu, and Tongliang Liu. Cs-isolate: Extracting hard confident examples by content and style isolation. Advances Neural Information Processing Systems (NeurIPS), 36, 2024.
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang. Tao Li. Tao Qin, Wei Chen, and Tie-Yar Liu. Learning causal semantic representation for out- of-distributtin prediction. In Advances Neural Information Processing Systems (NeurIPS), pp. 6155-6115 5570, 2021.
Chaochao Lu, Bernhard Schölkopf, and José Miguel Hernández-Lobato. Deconfounding reinforce- ment learning observational settings. arXiv preprint arXiv: 1812. 2018
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal representation learning for out-of-distribution generalization. In International Conference on Learning Representations (ICLR), 2021.
Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8046-8056 2022.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Repre- sentation learning via invariant causal mechanisms. In International Conference on Learning Representations (ICIR), 2021.
11

---

Published as a conference paper at ICLR 2024
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Poojan Oza, Vishwanath A Sindagi, Vibashan Vishnukumar Sharmini, and Vishal Patel. Unsuper- vised domain adaptation of object detectors: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Judea Pearl et Models, reasoning and inference. Cambridge, UK: CombridgeUnniverityyrrsss, 19 (2), 2000.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924 2017.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference Machine Learning (ICML.), pp- 18347-18377. PMLS. 2022.
Prasun Roy, Subhankar Ghosh, Saumik Bhattacharya, and Umapada Pal. Effects of degradations on deep neural network architectures. arXiv preprint arXiv: 180:::00001000 2018.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In Advances in Neural Information Processing Systems (NeurIPS), pp- 16282-16292, 2020.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with open-set consistency regularization. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems (NeurlPS), 2021. URL
https: pppnreview net/ wwtptteettt////eee//eeeeeeeeeett forum? (d7.ummmddd77ccccjjppr
Bernhard Schölkopf. Causality for machine learning. In Probabilistic and Causal Inference: The Works Judea Pearl, pp. 765-804. 2022.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke. Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-3344 2021.
Paras Sheth, Raha Moraffah, K Selçuk Candan, Adrienne Raglin, and Huan Liu. Domain generalization- a causal perspective. arXiv preprint (2ki::2209.1517., 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems (NeurIPS), pp. 596-608, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5018-5027, 2017.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in Neural Information Processing Systems (NeurIPS), pp- 16451-16467, 2021.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yigiang Chen, Wenjun Zeng. and Philip Yu. Generalizing to unseen domains: survey on domain generalization. JEEE Transactions Knowledge and Data Engineering, 2022a.
12

---

Published as a conference paper at ICLR 2024
Lixu Wang. Shichao Xu, Ruiqi Xu, Xiao Wang, and Qi Zhu. Non-transferable learning: A new approach for model ownership verification and applicability authorization. In Internationaa Conference Learning Representations (ICLR), 2022b.
Yasi Wang, Hongxun Yao, and Sicheng Zhao. Auto-encoder based dimensionality reduction. Neuro-
computing, 184:232-242, 2016.
Zhenyi Wang, Li Shen, Tongliang Liu, Tiehang Duan, Yanjun Zhu, Donglin Zhan, David Doermann, and Mingchen Gao. Defending against data-free model extraction by distributionally robust defensive training, Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generating framework for any-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 10275-10284, 2019.
Jingfeng Yang, Hongye Jin, Ruixiang Tang. Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin, and Xia Hu. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304. 122.304411222.. 12 2023.
Yu Yao, Tongliang I Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under structural causal model. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4409-4422 2021.
Yu Yao, Mingming Gong, Yuxuan Du, Jun Yu, Bo Han, Kun Zhang, and Tongliang Liu. Which is better for learning with noisy labels: the semi-supervised method modeling label noise? In International Conference on Machine Learning (ICML), pp. 39660-39673, PMLR, 2023.
Shuo Ye, Shujian Yu, Wenjin Hou, Yu Wang, and Xinge You. Coping with change: Learning invariant and minimum sufficient representations for fine-grained visual categorization. Computer Vision and Image Understanding, 237:103837, 2023.
Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Under- standing robust overfitting of adversarial training and beyond. In International Conference on Machine Learning (ICML). pp. 25595-25610. PMLR, 2022.
Guangtao Zeng and Wei Lu. Unsupervised non-transferable text classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
Cheng Zhang, Kun Zhang, and Yingzhen Li. A causal view on robustness of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp. 289-301, 2020.
Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Mare Ph Stoccklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. In Proceedings of the 2018 on Asia Conference on Computer and Communications Security, pp. 159-172, 2018.
Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Causaladv: Adversarial robustness through the lens of causality. In International Conference on Learning Representations (ICLR), 2022.
Jiyang Zheng, Yu Yao, Bo Han, Dadong Wang, and Tongliang Liu. Enhancing contrastive learning for ordinal regression via ordinal content preserved data augmentation. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https:/ //tp://////////ee.///// net/ forum? 1 wtt.2ldkxxxxxx2g......
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Lanyun Zhu, Tianrun Chen, and Yan WANG. Controllable adaptive learning, 2023. URL https: //opnreeiiiw wttp://wwwwweeeeeee.memeeeemeded o0tt://ommmm000mm22222
13

---

Published as a conference paper at ICLR 2024
APPENDICES
OVERVIEW:
Appendix A contains a toy experiment to present the vulnerability of statistical methods.
Appendix B contains full derivation of formulas in the main paper.
Appendix C contains complementary experimental details.
Appendix D contains additional experimental results and analyses.
Appendix E contains discussion of the rationality of our causal model.
A ToY EXPERIMENT
In this section, through toy experiment, show that statistical methods (e.g., Wang et al. (2022b)) are easily misled to (i) fit spurious correlation between style factors and labels, and (ii) model fake independence between confent factors and labels, rather than the intrinsically causal relationship. At the same time, we present a target domain masquerade attack toward these statistical methods by simply imposing intervention on the source domain.
1//1/11)/4
111///1/11
(a) MNIST (source)
(b) USPS (target)
(c) (MNIST) (intervention)
Figure A toy NTL experiment from MNIST to USPS.
As shown in Fig. 5, we conduct the toy NTL experiment on digit datasets, where we use MNIST (Deng, 2012) as the source domain and USPS (Hull, 1994) as the target domain. We only use two classes in these digit datasets for the toy experiment, i.e., number "0" and number "1". As shown in Fig. 5 (a), each digit in the source domain has a human-added class-wise color (red for number "O" and green for number "1"). In the target domain, as shown in Fig. (b), we keep the original data. We randomly selected 400 class-balanced samples from each domain for training and 200 for testing.
Moreover, the testing phase, we are not satisfied with evaluating these methods by regular samples. As shown in Fig. 5 (c), we intervene the source domain by simply changing the color of digits to white, i.c., (M(NITT In the current task, changing color will not influence the contents which are the intrinsical causes of labels. What we expect is that NTL model can fit content factors in the source domain and style factors in the target domain, thus leading to the ideal and robust NTL mechanism in line with human consciousness. If an NTL model cannot maintain source domain performance under the intervention do(NNTT)) it means that the model fits (i) spurious correlation between style factors and labels, and (ii) fake independence between content factors and labels, rather than the intrinsically causal relationship. At the same time, such intervention can be seen as target domain masquerade attack toward these methods.
Based on the above considerations, we train our causal-inspired H-NTL and the statistical-based method proposed by Wang al. (2022b) (we denote name). Following Wang et al. (2022b), we use VGG-11 (Simonyan & Zisserman, 2015) pretrained on ImageNet- 1K (Deng et al., 2009) backbones for both H-NTL and tNTL. To evaluate, we present top-1 accuracy (Acc) on both the source and target domain. Results are reported as average with standard deviation over three independent runs.
(NTL avoid being confused with the task
As shown in Table 4, both tNTL and H-NTL reach excellent NTL performance in regular testing set, with the source domain (MNIST) accuracy reaching 100% and the target domain (USPS) performance degrading to random classification. However, facing intervened MNIST data, the statistical-basee method (tNTL) failed to implement correct recognition, with the accuracy dropping to 49. 1971 This phenomenon illustrates that the tNTL fits the spurious correlation from style factors to labels (i.e., "0" and green-*"1)) rather than the real content factors. Simultancously. the tNTL relaxes the statistical dependence between labely and content factors as well as other style factors (such as the
14

---

Published as a conference paper at ICLR 2024
Table 4: Results of the toy NTI. experiment (MNIST- USPS). show averaged accuracy Aec (%)
with standard deviation from three independent runs.
MNIST
USPS (target)
do(MNIST) (intervention)
Method
(source)
INTL (Wang et al. (2022b), caatistical-bsedd
100.00 =000
50.83 +1.04
49.17 +3333
H-NTL (ours, causal-inspired)
100.00 +000
51.50 +2.00
97.17 +202
digital styles) to implement untransferabilltty In particular, the imposed relaxation between labels and content factors belongs to fake independence. On the contrary, our causal-inspired H-NTL. correctly practices the ideal NTL mechanism in line with human consciousness, with the accuracy still reaching 97.17% under intervention.
Overall, this toy experiment shows the vulnerability of statistical-basee NTL methods. In practice, fitting spurious correlation and fake independence will lead to the learned con-transferable rep- resentations being fragile and sensitive to small natural intervention-base further leads to the limitation on both source domain performance maintenance and target domain performance degradation. Moreover, this also indicates the risk of being attacked by malicious attackers through target domain masquerading.
domain shifts. This
B THE DERIVATION OF ELBO
In this section, we provide detailed derivation of the evidence lower-bound (ELBO) in Eq. (4) in the main paper. Now, we start with maximizing the log-likelihood p(r, y, d) of each datapoint (z, y, d) from the dataset D. The log- log-likelihood of the joint distribution can be written as:
logp(z, y, d) log S. p(z, y, d, c, s)deds
log [ p(z, p(z, d, 8) 2o (c s|x)
go(c,s|z) dods
(8)
p(a, d...v.d.c.s) (0(c,
log E (c
=IB(c.)) log p(z, v. c.s)
ELBO(x, d.).
go(c,8(x)
By applying the factorization in Eq. (3), we have:
p(c,s)pa, (dlc (alc,s)
ELBO(x, y, d) = E(ccs)
log
(o(c, s(z)
log M),,s)
= (Rccs)))
+ E ((ocel)) [logpe, (vlc,
Co s(c,
+ +B(c))))a(c..ll [log Po, (dlc, s)] ((c.o)))((c.s1)) [logPo_ Po. (alc, s)].
In practice. to reduce computational costs and to allow our method efficiently infer latent factors, we approximate PCY|C, S) by assuming that C contains sufficient information about Y (Yao et al., 2021; Lu et al., 2018), i.e., P(Y|C, S) P(Y[C), and such assumption is also applicable to P(DIC,S) Moreover, separately infer C and S from only the input X by using and do, to model the posterior distributions qs, (CIX) and 4a, (SIX), respectively. Due to the fact that the complex image X contains enough information about and S in real scenarios (Yao et al., 2021; Lu et al., 2018), such assumptions are reasonable and are expected not to have very large approximation error. Therefore, the Eq. (9) can derived as:
encoder modules do.
ELBO( y, d) KI(qe, (cl=))l((C) KL (s|x))lp(())
Ee-gg,(ela) (elx) [log qo_ (vle)] Ex (s1z) [log go, (d]s) + Ee- (clm) -- (s|=>) [log Po_ (xlc,s)]
(10)
which is the final ELBO in the main paper (Eq. (4)).
15

---

Published as a conference paper at ICLR 2024
C COMPLEMENTARY EXPERIMENTAL DETAILS
In this section, we provide complementary experimental details. Appendix C.I contains introduction of all datasets. In Appendix C.2, we present more implementation details, including baselines, models, training details, evaluation metrics, and running environments. Appendix C.3 contains network architectures of our H-NTL. In Appendix C.4, we provide details about the style augmentations in our source-only H-NTL. Appendix C.5 contains the full training algorithm of the source-only H-NTL. Finally, in Appendix C.6, we provide a detailed process for conducting the ablation study.
C.I DATASETS
Our experiments involve three basic tasks on digit datasets and three challenging tasks on real- world datasets. The digit tasks contain three random-selected pairs from four digit datasets: MNIST (MT) (Deng, 2012), MNIST-M (MM) (Ganin et al., 2016), SVHN (SN) (Netzer et al., 201 1) and SYN-D (SD) (Roy et al., 2018). For challenging tasks, we involve CIFARIO to STLIO (Coates et al., 2011) (C10-S10), VisDA (Peng et al., 2017) (VT and OfficeHome (Venkateswara et al., 2017) (OP-OC). We provide more detailed introductions for each dataset as follows:
Digits: Following Wang al. (2022b), we involve digit datasets for evaluation. Digit datasets include MNIST (MT) (Deng, 2012), USPS (US) (Hull, 1994), SVHN (SN) (Netzer et al., 201 1), MNIST-M (MM) (Ganin et al., 2016) and SYN-D (SD) (Roy al., 2018). Each dataset contains ten digits collected from real scenes or artificially constructed. We conduct experiments on three random-selected pairs: MM-MT, SN-SD, and SD-MT.
CIFARIO & STLIO: We also follow Wang et al. (2022b) to evaluate on CIFARIO (C10) and STL10 (S10) (Coates et al., 2011) Both C10 and $10 are ten-class classification datasets, which contain six animal categories and four transportation categories. We use C10 as the source domain and $10 the target domain.
VisDA: VisDA (Peng et al., 2017) contains a training set VisDA-T (VT) and a validation set VisDA- V (VV) of 12 object categories. Following Wang al. (2022b), we consider the trantransferable task from VT to VV.
OfficeHome: OfficeHome (Venkateswara et al., 2017) contains four domains, where each domain consists of 65 categories. The number of categories in OfficeHome significantly more than the above datasets, and thus, it's more challenging for NTL methods to resist transferability. We conduct NTL experiments from Product (OP) to Clipart (OC), where OP contains product images without backgrounds and OC is collection of clipart images.
C.2 IMPLEMENTATION DETAILS
Baselines. We use supervised learning (SL. and the non-transferable learning method proposed by Wang et al. (2022b) as baselines. In order to avoid confusion between the method name of Wang al. (2022b) and the task name, we denote their methods as tNTL and sNTL for the tpreet-specified and source-only tasks, respectively. We provide brief introductions for baseline methods as follows:
SL: We use a standard supervised learning pipeline with cross-entropy loss.
tNTI:: target-specified NTL method inspired by information bottleneck. This method adds two statistical dependence relaxation terms on standard supervised learning to resist transferability: (i) maximizing the Kullback-Leibl (KL) divergence between target domain representation and label, and (ii) maximizing the maximum mean discrepancy (MMD) between the distribution of source and target domain representations.
sNTL: a source-only NTL method, where generative adversarial network (GAN) used to synthesize fake target domain data with domain shifts. Thus, the source-only NTL can be solved by the INTI. method.
Models. For fair comparison, we follow the backbones in (Wang et al., 2022b) to conduct all experiments. Specifically. for the backbones in VAE encoder modules ($o, and d and student network fall we apply VGG-11 (Simonyan & Zisserman, 2015) in digit tasks (MM-MT, SN-SD and SD-MT), VGG-13 (Simonyan & Zisserman, 2015) in C10-S10, and VGG- 19 in VT-VV and OP-OC. All backbone networks are initialized as the pre-trained version of ImageNet- IK (Deng et al., 2009). The detailed network architectures of the VAE encoder modules, the VAE classifier modules, the VAE decoder modules, and the student network are shown in Appendix C.3.
16

---

Published as a conference paper at ICLR 2024
Training details. For basic digit tasks, resize images 32x32. For other challenging tasks, we resize images to two different resolutions: 32x32 and 64x64. For training SL, we employ the SGD as optimizer with Ir 0.001 and set the batch size to 32. For tNTL and sNTL, use their released code' and the same hy hyperparameters settings reported in their paper to run experiments. For our proposed H-NTL, we employ the SGD as an optimizer with Ir = 0.1 and set the batch size to 128. The disentanglement VAE is trained for 20 epochs, and the dual-path knowledge distillation is trained for 30 epochs. The hyper-parameter X is set to 1.0 for all datasets. In addition, the number of image style augmentations set to 10 when performing source-only NTL tasks. Following Wang et al. (2022b), we randomly select 8,000 samples as training data and 1,000 samples as testing data without overlap for digit tasks, C10-S10, and VT- For OP-OC, we use 3,000 for training data and 1,000 samples due to the limitation of the dataset size.
Evaluation metrie. We show top-1 classification accuracy (Ace) on source/target domain respectively, Results are reported average with standard deviation three independent runs. In addition, we calculate the accuracy drop (A = Acca - Acc,v) and relative drop (A% A Acc,a) on source/target domain, respectively.
Environment. Our code is implemented in Python 3.8.8 and PyTorch 1.8.0. All experiments are conducted on server running Ubuntu 20.04 LTS, equipped with an NVIDIA RTX A6000 GPU.
C.3 NETWORK ARCHITECTURE
In this section, we present the detailed network architectures of our VAE encoder modules (go, and ). VAE classifier modules (po. and po. ), VAE decoder module (fo.) ) and the student network fall- In the VAE encoder modules (ās. and (Ø,) and the student network fru. we follow Wang et al. (2022b) to use several popular architectures as backbones to extract image features. Specifically, we apply VGG-11 (Simonyan & Zisserman, 2015) in digit tasks (MM-MT, SN-SD and SD-MT), VGG-13 C10-S10, and VGG-19 in VT-VV and OP-OC.
First of all, the architecture of the VAE encoder modules (io, and (o,) is shown in Table 5. Specifi- cally, img is a parameter corresponding to the input image size. If the image size 32x32 (for the three basic digit tasks and three challenging tasks), the img = 1. If the image size 64x64 (only for the three challenging tasks), the img 2.
Table 5: The architecture of the VAE encoder modules (9o. and ao where the parameter img corresponds to the input size.
Mean μ
Variance a
Latent Mapping
Linear(512, 256)
Linear(512, 256)
Pooling Layer
SaarrieeevvgPooonn((((5122 img, img]. 512) Backbone Network
Feature Extractor
Then, the architecture of the VAE classifier modules (Po. and Po. is shown in Table 6. In particular, the content factor classifier Po_ contains linear layer which maps the estimated 256-dim content factor C to the classification space (10-dim for digits and C10-S10, 12-dim for VV-VT, and 65- dim for OP-OC). Similarly, through linear layer, the style factor classifier fo, maps the estimated 256-dim style factor S to the domain classification space whose dimension equals to domain num (in this paper, the domain_not 2).
Table 6: The architecture of the VAE classifier modules (po, and pa.) The class num equals to the number of categories, and the domain num equals to the number of domains (= 2 in this paper).
content factor
style factor
Classifier
Lincar(256, class num) Linear(256, domain num)
Next, the architecture of the VAE decoder modules (i.e., Po_) is shown in Table Particularly, the Conv4 layer only exists when the input image size equals to 64 x 64. The input layer of the decoder
http::/ggttuuu.ccm/oooniiiiionno Wang//TTT.
17

---

Published as a conference paper at ICLR 2024
Table The architecture of the VAE decoder module (Po.) ). The Conv4 layer will be removed if the input image size equals to 32x32.
Sigmoid, Conv2d 3, I)
Output
LeakyReLU, BatchNorm(322
Conv ConnTraaspose22(32,, 32, 3, 2, 1) LeakyReLU, BrachNor33))
Conv4
Con/Transposse2dc32,, 32, 3, 2,
LeakyReLU, NatchNor33))
Conv3
Con/Transpose22((64... 32, 3, 2,
LeakyReLU, BatchBaahhhhmmcmcc
Conv2
Con/Transpose2d((23 64, 3, 2, 1,
LeakyReLUU, S5atchNoomm1225
Conv1
Conv ConvTranspose2d(256, 128, 2, 1, Linear(512, 1024)
Input
Concat
content factor
style factor
first concat the 256-dim content factor C and the style factor S, and then maps them high dimension feature to perform transposed convolution. The parameters of the transposed convolution operation (i.e., Conv'T Transpose2d" are: in channels, out outoooottttto mot_hheanls, kernel_sie size, stride, padding, and output propput_pdddig.... In the final output layer, the parameters of the convolution operation (i.e., Conv2d") are: in channels, out channels, kernel size, stride, and padding.
Finally, the architecture of the student model fnit is shown in Table 8. The architecture is the same Wang et al. (2022b) for a fair comparison. Especially, the input image size equals to 32x32, the parameter img 1. If the image size equals to 64x64 (only for the three challenging tasks), the img 1. Besides, the class_ corresponds to the number of categories in each dataset.
Table 8: The architecture of student models futl, where the parameter img corresponds to the input size. The class_num equals to the number categories.
Linear(256, class num).
Classifier
Linear(256, 256), ReLU, Dropout:
Linear(5122img"inng img. 256), ReLU, Dropout;
Feature
Backbone Network (BG-11//GG-13/VGG-:
Extractor
C.4 STYLE AUGMENTATION FOR SOURCE-ONLY NTL TASK
In this section, we provide details about the style augmentations used in our source-only H-NTL, As mentioned in the main paper, our aim is impose interventions on the style factor S (i.e., do))). which do not impact P(Y|C). In practice, we cannot obtain naturally entangled content factor C and style factor from single source domain and subsequently manipulate the style factor value. Thus, we leverage image style augmentations (Sohn et al., 2020; Berthelot et al., 2019; Saito et al., 2021; Zheng al., 2024) to implicitly impose interventions on source domain data. Our image style augmentations are derived from RandAugment (Cubuk et al., 2020). For completeness, we briefly present the list of image style augmentations we used in Table 9. For more details and corresponding parameters, please refer to Cubuk et al. (2020). Specifically, in experiments, all perameersssssrs in these style augmentations follow Saito et al. (2021), and we do not tune any hyper-parameter
In Fig. and Fig. 7, show the original and augmented source domain data C10 and VT, respectively. Each source domain image is augmented by a kind of style augmentation randomly selected from the Table 9, We can that these style augmentations do not influence the content factor but significantly change the image style, thus satisfying our intervention aims.
"htpss//pyyorch.orr/ddss/// 8. 8.O.generated/oocch..n Cony Satprraaeeeooooaaooooooeeeeeelaa "htps://yytorch.or//ddoss/// http:///www......o.ee//c/annnmaa 2dd..od..mmm..
18

---

Published as a conference paper at ICLR 2024
Table 9: List of image style augmentations. We only present names with brief descriptions. For details and corresponding parameters, please refer to (Cubuk et al., 2020).
Augmentation
Brief Description
Autocontrast
Maximizes the image contrast by setting the darkest (lightest) pixel to black (white), and then blends with the original image.
Brightness
Adjusts the brightness of the image.
Color
Adjusts the color balance of the image.
Contrast
Controls the contrast of the image.
Cutout
Sets random square patch of side-length pixels gray.
Equalize
Equalizes the image histogram, and then blends with the original image. Inverts the pixels of the image, and then blends with the original image. Reduces each pixel to certain bit.
Invert
Posterize
Rotate
Rotates the image.
Sharpness
Adjusts the sharpness of the image.
Shear_x
Shears the image along the horizontal axis.
Shear
Shears the image along the vertical axis.
Solarize
Inverts all pixels above certain threshold value. Translates the image horizontally by certain pixels.
Translate_r x
Translate
Translates the image vertically by certain pixels.
(a) Original images
(b) Augmented images
Figure 6: Comparison of original images and augmented images on C10.
(a) Original images
(b) Augmented images
Figure Comparison of original images and augmented images on VT.
C.5 FULL ALGORITHM OF SOURCE-ONLY H-NTL
In this section, we provide the full algorithm for training source-only H-NTL. Briefly, in the source- only NTL task, we conduct image style augmentations on source domain data to obtain OOD data. Then, we see all OOD data augmented by different styles the target domain, and we use the accessible source domain and the target domain to train the H-NTL in the same way as target- specified N'TL. The detailed training procedure of source-only H-NTI is shown in A Algorithm 2.
19

---

Published as a conference paper at ICLR 2024
Specifically, the augmentation pool A {a+)=i contains N₂ kinds of style augmentations which randomly selected from the total style augmentation list (see Appendix C.4 and Table 9), where No a hyper-parameter In both disentanglement and dual-path knowledge distillation, we randomly select different style augmentations from the pool A to augment each image in the source domain D,, This augmentation process repeated in every epoch (i.e., the inner for-loop in the disentanglement and dual-path knowledge distillation), effectively expanding the distribution range of the augmented source domain data.
Algorithm 2 Train source-only H-NTL
1: Training set in source domain Dj (az, Uz. d2 0};;; Augmentation pool A {a,) i-; Batch- size n; Disentanglement epoch Exiss KD epoch EEa, VAE modules (āo. Io. Pe,, Pe, po,) and the NTL network fntl-
2: for i = 1 to Edis do N, do Fetch the j-th sample 2; in the source domain D,; 5: Random select augmentation function a E .A;
3:
for j
4:
6: Perform image style augmentation on the j-th sample: ūt = a(xz);;
7: end for
8: See augmented data target domain D. = D. {ēj. Bo dj = 1);
9: Using D, U D, train VAE (ie.) Is,, Do, D0. Do pog} by maximizing ELBO;
10: end for
11: for i = 1 E do
12:
for j= to N₂ do
13:
Fetch the j-th sample vy in the source domain D,; Random select augmentation function a E .A; Performing image style augmentation on the j-th sample: ă, a(z;): end for See augmented data as target domain D, = DD,- { Vy d2 1 1)
14:
16:
18: Using D, u 'D, to train fall through minimizing Lidi
19:
end for
C.6
THE DETAILED PROCESS FOR CONDUCTING THE ABLATION STUDY
The ablation studies in the main paper aim to explore the effectiveness of the main components (the content factor C and the style factor S) the H-NTI. To validate the importance of C for H-NTL,, train the student model to only fit C of the source domain by using single-path knowledge distillation (denoted as H-NTL w/o C). The only difference between H-NTL w/o and the original H-NTL is that the target domain path released. Similarly, to validate the importance of S, we train the student model to only fit S of the target domain (denoted as H-NTL w/o S). Then, we compare them with the original H-NTL our paper, i.e H-NTL (full). The disentanglement processes are the same for H-NTL w/o C, H-NTL w/o 8, and H-NTL (full).
D ADDITIONAL EXPERIMENTE AND ANALYSES
This section contains additional experiments and analyses. In Appendix D.I, we show results of the proposed H-NTL on high-resolution images. In Appendix D.2, we show results of H-NTL on intellectual property protections. In Appendix D.3, we compare the proposed H-NTL with an additional baseline. In Appendix D.4 and Appendix D.5, we analyze the influence of main peperaarameters our H-NTL. In Appendix D.6, we empirically analyze the disentanglement between conteni factors and factors. In Appendix D.7, we present visualization results.
D.1 RESULTS ON HIGH RESOLUTION DATASETS
We additional evaluations with rishettinoolutin images. In detail, experiments the original resolution VT (112x112) (Peng et al., 2017) and OP-OC (224x224) (Venkateswara al., 2017) without resizing, thus showing the effectiveness of the proposed H-NTL higher
20

---

Published as a conference paper at ICLR 2024
resolution images. Results of the proposed H-NTL on the target-specified NTL setting and source- only NTL setting are shown in Table 10 and Table 11, respectively. On higher-resolution datasets, the proposed H-NTL can still effectively maintain source domain performance (comparable to SL) and significantly degrade the target domain performance.
Table 10: Results of target-specified H-NTL on high-resolution datasets.
Source
Target domain
Image
SL.
H-NTL.
source
target
domain
resolution
(source/target)
(source/target)
drop
drop
VT
VV
32x32
89.7/22.4
91.7/8.1
-2.0
14.3
VT
VV
64x64
93.4/35.6
94.6/8.2
-1.2
27.4
VT
VV
112x112
97.5/42.6
97.8/8.3
-0.3
34.3
OP
OC
32x32
65.6/23.6
65.6/5.5
0.0
18.1
OP
OC
64x64
75.6/31.6
76.4/6.7
-0.8
24.9
OP
OC
224x224
86.0/35.5
84.9/3.6
1.1
31.9
Table 11: Results of source-only H-NTL high-resolution datasets.
Source
Target domain
Image
SL
H-NTL
source drop
target
domain
resolution
(source/target)
(source/target)
drop
VT
VV
32x32
89.7/22.4
91.6/9.3
-1.
13.1
VT
VV
64x64
93.4/35.6
94.9/7.8
-1.5
27.8
VT
VV
112x112
97.5/42.6
96.7/8.2
0.8
34.4
OP
OC
32x32
65.6/23.6
63.7/14,4
1.9
9.2
OP
OC
64x64
75.6/31.6
71.6/16.0
4.0
15.6
OP
OC
224x224
86.0/35.5
83.3/10.1
2.7
25.4
D.2 RESULTS ON IP PROTECTIONS
The practical application of the proposed H-NTL lies in the model intellectual property (IP) protec- tions. Our H-NTL can provide two kinds of IP protection: ownership verification, and applicability authorization. As you suggested, we discuss and run more experiments on ownership verification and applicability authorization to further show the effectiveness of our H-NTL real-world applications.
Ownership verification. The aim of ownership verification to verify the ownership of deep learning model. The proposed H-NTI. provides a solution for ownership verification by triggering misclassification on the target domain (Wang et al., 2022b). Specifically, we can add a pre-detined trigger patch (only known by the model owner) on the original dataset (i.e., the source domain) and see them as the target domain. is worth noting that such a pre- pre-defined trigger patch can be controlled to be shallow that normal supervised learning (SL) models trained the original source domain can still have normal performance on the patched source domain. Then, we train deep learning model by using the proposed H-NTL these domains. After that, the training model will perform poorly on the data with the patch but have a good performance on the data without the patch. Thus, by observing the performance difference of trained model on the source domain data with and without the pre-defined trigger patch, we can verify whether a deep learning model belongs the model owner.
We have verify the strong performance of the proposed H-NTL on watermark data with different patch values (results are shown in Section 4.1 and Fig. 4 in the main paper). which serves as the core task of ownership verification. Here we run additional experiments of ownership verification, and results are shown in Table 12. We can see that models trained with H-NTI. behave differently on the data with and without the patch, whereas SL models perform nearly the same. Accordingly, the ownership of trained models can be verified.
Applicability authorization. Applicability authorization aims authorizing models to certain data for preventing their usage on unauthorized data (Wang et al., 2022b), which can be solved by applying source-only H-NTL to restrict the model generalization ability to only the authorized domain. Specifically, we add a pre-defined authorized patch to the original data to be authorized and them the source domain. We regard the union of the original data (without the authorized
21

---

Published as a conference paper at ICLR 2024
Table 12: Results of H-NTL ownership verification. "(p)" means patched data. The results of H-NTL are highlighted in bold.
Source
Target
SL
H-NTL
domain
domain
(patch/non-patcl)
(patch/non-patch)
MT
MT(p)
98. 98.8/98.5
98.6/10.6
US
US(p)
98.9/98.8
99,4/14.3
SN
SN(p)
87.9/87.4
87.8/8.5
MM
MM(p)
92.8/92.0
93.4/9.7
SD
SD(p)
96.3/95.9
96.1/9.5
CIO
C10(p)
86.5/60.2
87.9/8.9
S10
S10(p)
88. 18.1/82.0
89.5/10.9
VT
VT(p)
93.4/92.2
95.1/8.3
patch), the augmented original data with and without the authorized patch as the target domain. Then, we train a deep learning model by using the proposed H-NTL these two domains. After that, the trained model will only perform well on the authorized data (i.e., the original data with the authorized patch). For unauthorized data (c.g.. the original data without the authorized patch, the data from other domains with or without the authorized patch), the trained model will perform poorly, Thus, we achieve the model applicability authorization.
We run experiments of applicability authorization, and results are shown in Table 13 (Digits datasets) and Table 14 (complex datasets). We can see that the model trained by H-NTL has good performance on the authorized domain (highlighted in bold in each row) and poor performance on all unauthorized domains. This shows the effectiveness of H-NTL in applicability authorization.
Table 13: Results of H-NTL applicability authorization (digits datasets). "(p)" means data with the authorized patch. In each row, the authorized domain is highlighted in bold. The last column shows the averaged results on authorized and unauthorized domains (also highlighted in bold)
Source
MT
US
SN
MM
SD
H-NTL
MT
US
SN
MM
SD
domain
(p)
(p)
(p)
(p)
(p)
(auth/unauth)
MT(p)
98.2
12.3
10.1
12.3
9.6
9.2
9.7
10.1
8.4
9.6
98.2/10.1
US(p)
9.5
99.2
6.0
10.8
10.2
9.5
17.2
6.0
10.8
10.2
99.2/8.9
SN(p)
16.5
18.8
88.3
11.9
10.5
8.9
9.2
17.0
10.3
10.0
88.3/12.6
MM(p)
20.3
17.2
14.5
92.2
10.9
9.7
17.2
5.8
12.5
11.0
92.2/13.2
SD(p)
8.7
29.5
22.0
17.1
94.4
8.1
7.5
9.3
10.9
11.9
94.4/13.9
Table 14: Results of H-NTL on applicability authorization (complex datasets). "(p)" means data with the authorized patch. In each row, the authorized domain is highlighted in bold. The last column shows the averaged results on authorized and unauthorized domains (also highlighted bold).
Source domain C10(p)
S10(p)
C10
S10
H-NTL(auth/unauth
C10(p)
87.4
9.7
10.6
8.9
87.4/9.7
S10(p)
12.7
88.3
8.9
11.2
88.3/10.9
Source domain
VT(p)
VV(p)
VT
VV
H-NTT (auTT_aauthinuuth)
VT(p)
92.0
8.6
10.9
9.5
92.0/9.7
D.3 MORE BASELINES
For a comprehensive evaluation, we further design an additional baseline method (denoted as TargetClass) which can be used for non-transferable learning. Specifically, considering an N-class non-transferable learning task, we take the same backbone as our main paper as feature extractor and use classification head with N+I classes to predict the N classes in the source domain, but predict an additional "target class" as class N+1 for all samples in the target domain. Therefore, the model trained by TargetClass will map all the target domain data to isolated cluster that stays away from the N nource-domain knowledge transfer.
clusters in the feature space, thus hindering the source-te torget
22

---

Published as a conference paper at ICLR 2024
We rum experiments under the same setting main paper, and results are shown in Table 15 (on natural datasets) and Table 16 (on watermark target-domain). From the comparison, the proposed H-NTL generally has better performance.
Table 15: Comparison of the new baseline (denoted as TargetClass) and the proposed H-NTL on natural data. The best results are highlighted in bold.
Source
Target
Image
TargetClass
H-NTL
domain
domain
resolution
(source/target)
(source/target)
MM
MT
32x32
92.879.5
93.1/9.9
SN
SD
32x32
87.8/10.4
88.1/9.2
SD
MT
32x32
97.1/12.6
97.1/11.0
C10
S10
32x32
81.3/50.2
80.6/28.1
C10
S10
64x64
85.8/22.0
87.6/9.6
VV
VT
32x32
91.4/10.99
91.7/8.1
VV
VT
64x64
94.2/7.9
94.6/8.2
OP
OC
32x32
64.2/8.6
65.6/5.5
OP
OC
64x64
72.2/6.6
76.4/6.7
Table 16: Comparison of the new baseline (denoted as TargetClass) and the proposed H-NTL on natural data. The best results are highlighted in bold.
Source
Target
Patch
TargetClass
H-NTL
domain
domain
value
(source/target)
(source/target)
CTO
C10(p)
20
83.0/T0.7
87.8/7.9
C10
C10(p)
40
85.5/10.0
88.7/10.1
C10
C10(p)
60
85.5/8.8
87.3/9.8
STO
STO(p)
20
85.4/53.4
83.6/16.7
S10
S10(p)
40
85.9/20.4
87.4/12.5
S10
S10(p)
60
86.9/15.9
88.3/9.0
D.4 INFLUENCE OF KD WEIGHT
The main hyper-papre parameter in proposed H-NTL is the KD weight A which is used to balance the loss values from source and target domain distillation paths. Actually, we always set the weight A to and do not need tune its value. In order to analyze its influence the total NTL performance, we change its value in a large range (from 10 to 10) and conduct both target- wunget-speciffed/ssurcee.e NTL tasks on C10-S10 and VV. The results of target-specified and source-only NTL tasks are shown in Fig. 8 and Fig. 9, respectively. In the target-specified NTL task, we can see that we set the A₂ too small, the network fnt is difficult to fit the style factors in the target domain, and thus, the target domain performance cannot be degraded. the weight X, is set too large, the network fnu will be hard to fit the content factors in the source domain, which leads to the failure of source domain performance maintenance. In the source-only NTL task, the influences of X on maintaining source domain performance and degrading target domain performance are both similar the target-specified NTL task. One difference that the total NTL performance source-only H-NTL is more vulnerable the weight due the unseen target domain in source-only setting.
D.5 INFLUENCE OF THE NUMBER OF STYLE AUGMENTATION IN SOURCE-ONLY TASKS
main syper-parameter in source-only H-NTL the number of image style augmentations. To analyze its influence on the total performance of source-only NTL, we change its value from 1 to 12. The results on C10-S10 and VV-VV are shown in Fig. 10. We can see that the source domain performance is not sensitive to the number of augmentations, but the target domain performance degradation is limited under a small number of augmentations (e.g., =3). This limitation arises from the restricted distribution of the augmented source domain data which is augmented with small number of styles and fails to encompass sufficient range oft-oo-diitribution data. Thus, the target domain performance degradation is hard generalize to the real unseen target domain.
23

---

Published as a conference paper at ICLR 2024
100
100
80
Accuracy (%)
(%)
60
40
40
20
1o44
16-3
1o-2
le.4
le-2
0.1
A
(a) C10-S10
(b) VT-VV
Figure 8: The influence of X in target-specified NTL tasks.
100
100
80
80
Avcuracy (%)
60
Accuracy
60
40
STC
20
20
Egt
1o4
1c-3
lc-2
0.1
10
1c44
16:3
1c-2
0.1
5
(a) C10-S10
(b) VT
Figure 9: The influence of X in source-only NTL tasks.
100
100
80
Accunit (%)
60
Accuracy
60
20
20
0
2
9
12
2
6
12
Augmentations
Number Augmentations
(a) C10-$10
(b) VT-VV
Figure 10: The influence the number of style augmentations in source-only tasks.
D.6 DISENTANGLEMENT OF CONTENT AND STYLE FACTORS
In this section, we empirically analyze the effectiveness of the disentanglement between content factors and style factorss For experimental validation, we conduct cross-prediction. In detail, we employ the VAE encoders (is, and de. ) extract the content factor C' and style factor S from data, and then we use both of them to predict class Y and domain D through VAE classifiers (po, and fo.). The results are shown in Fig. 1. with subfigures (a) (d) presenting the predictions of C-to-Y. C-to--, S-to-Y, and S-to-D, respectively.
The results show that only content factors are meaningful in predicting class, and style factors are meaningful in predicting domain. If we use content factors to predict domain (or use style factors to predict class), the results are similar to random classification. These phenomenons empirically indicate the effectiveness of our disentanglement between content factors and factors.
D.7 VISUALIZATION
t-SNE visualization of non-transferable feature representations. As shown in Fig. 12 and Fig. 13, we use t-SNE visualization (Van der Maaten & Hinton, 2008) to present the learned non-tram non-transferable representations C10-$10 and VT-VV, respectively. For intuitive comparison, present feature
24

---

Published as a conference paper at ICLR 2024
Class prediction
Domain prediction
0
2
5
7
85.13
1,03
3.59
1.54
2.05
0.00
0.00
0.51
3.08
3.08
61.54
38.46
0.00
90.64
0.00
0.49
0.00
0.00
0.00
8.37
62.56
37.44
4.25
36.32
1.08
0.00
5.95
63.78
3.78
15.68
6.49
1.08
1.62
0.54
37.30
content
factor
0.00
0.52
2.07
4.66
85.49
2.07
2.07
2.07
0.52
0.52
content
factor
44.56
55.44
0.52
0.00
4.17
8.33
2.08
77.60
4.69
2.60
0.00
0.00
50.52
49.48
0.97
0.00
6.31
2.91
0.97
3.88
84.47
0.49
0.00
0.00
62.62
37.38
1.42
0.47
0.47
2.36
3.77
4.72
2.36
83.96
0.47
0.00
63.68
36.32
2.39
0.52
0.52
0.52
0.00
0.52
0.52
1.04
92.75
1.04
34.20
65.80
1.44
6.22
0.00
0.00
0.00
0.00
0.00
2.87
89.00
40.67
59.33
(a)
(b)
Class prediction
Domain prediction
0
7
8
style
facior
14,10
10.50
7,30
7.30
9.20 10.20
10.40
14.30
7.60
9.10
style foctor
0.00
8.00
8.00
15.50
13.00
10.70
8.70
7.40
6.10
11.90
10.70
0.10
99.90
(c)
(d)
Figure 11: Visualization of disentanglement between the content factor and the style factor on C10-S10. The number in each cell denotes the top-1 classification accuracy (%).
representations learned by supervised learning (SL), target-specified H-NTL, and source-only H-NTL. Particularly, those opaque dots represent source domain features, and transparent dots represent target domain features. As shown in Fig. 12 (a) and Fig. 13 (a), although only trained in the source domain, the target domain features of SL are overlapped with the source domain features and maintain certain discriminability. This leads to the meaningful performance SL on the target domain. Compared to SL, both target-specified and source-only H-NTL can clearly separate the source and target domain features with a certain distance. randomly distributed due to the constraint of fitting style factors in the target domain distillation path. Thus, the target domain performance is degraded to the accuracy of random classification. Moreover, because of the disentanglement from paired source-target domains, the discriminability of features in the source domain is maintained (or even enhanced) compared to SL.
addition, the feature representations in the target domain are
E MORE DISCUSSION OF THE CAUSAL MODEL
In this section, we provide further explanation and discussion of our causal model. In Appendix E. we discuss the statistical dependence between the content factor C and the style factor in our causal model. In Appendix E.2, we provide more explanation about the causal direction from the style factor to the domain D. In addition, we also consider another causal direction (i.c., D causes S) and conduct basic experiments.
25

---

Published as a conference paper at ICLR 2024
(a) SL
(b) larget-specified H-NTL
(c) source-only H-NTL
Figure 12: t-SNE Visualization of features on CI0-S10, Different color denotes different class, and opaque/transparent dots represent source/target domain data.
(a) SL
(b) target-specified H-NTL
source-only H-NTL
Figure 13: t-SNE Visualization of features VT-VV, Different color denotes different class, and opaque/transparent dots represent source/target domain data.
E.1 DEPENDENCE OF THE CONTENT FACTOR C. ON THE STYLE FACTOR S
The definition of the content factor C' and the style factor S. In order to avoid being misled to fit spurious correlation and fake independence, we distinguish the unobservable and naturally mixed contents and styles by separately modeling them in our causal model. As shown in Fig. in the main paper, we decompose instance into two latent variables:
Content factor C:: This corresponds to the intrinsic, class-related information that is a cause of the label Y both the source and the target domain. For example, in the case where use CIFAR10 as the source domain and STL10 as the target domain, C' represents the object category that is common both the CIFARIO and STL10 (c.g.. bird, airplane, deer, etc.).
Style factor S: Except for the content factor C, all other factors can belong to style factors. For example, in the NTL task from the wildlife park to the zoo (Fig. in the main paper), the environments of wild and zoo are style factors. In the toy experiment (Appendix A), both the digit colors and the digit typefaces belong style factors.
The motivation of modeling the statistical dependence between C' and S. As mentioned in the introduction (Section in the main paper), the common co-occurrence between contents and styles in real-world scenarios causes that existing methods inadvertently fitting (i) spurious correlation between styles and labels, and (ii) fake independence between contents and labels. We assume the co- oe-ocuuurence between contents and styles is caused by latent confounders (Von Kügelgen et al., 2021; Schölkopf, 2022; Klindt al., 2021), ), and thus, we model it by statistical dependence between the style factor S and the content factor C in our causal model (i.e., the dashed line in Fig. in the main paper). is worth noting that in this paper, we do not aim discovering the latent confounders precisely. Motivated by existing problems in NTL, we propose disentangle the C' and S, thus avoiding the contradictions between models and human understanding.
Assumptions about the statistical dependence between C' and S. We only assume that the statistical dependence between C' and S has a weak relation in the target domain. Thus, the in the target domain barely can predict Y. which motivates us to fit S in the target domain during dual-path knowledge distillation. We do not need to assume the strength of the statistical dependence in the source domain, which means that the relation between S and can be either strong weak in the
26

---

Published as a conference paper at ICLR 2024
source domain. In real world scenarios, such assumptions always satisfied. Empirically, experiments (Section 4 in the main paper) illustrate that by fitting in the target domain, the fait always has random classification accuracies. Thus, our assumptions reasonable.
E.2 CAUSAL RELATIONSHIP BETWEEN THE STYLE FACTOR AND THE DOMAIN D
As shown in Fig. 14 (a), we mainly focus on the situation of S causing D in this paper S D) (Huang et al., 2022; Liu et al., 2021; ; Mitrovic et al., 2021; Zhang et al., 2022). Besides, as shown in Fig. 14 (b), D causing S (i.e., D S) is also popular situation considered by the community of domain adaptation and domain generalization (Kong ct al., 2022; Liu al., 2021; Lu et al., 2021; Lin et al., 2024). We argue that both directions are reasonable and do not contradict each other. In our definition, the complex style factor contains all other factors except the content content C. Parts of the style factor may depend to the environment, it belongs to D - S. In contrast, sometimes we determine the domain by other parts of the style factor S (e.g., whether containing human-added watermark style augmentations). In these scenarios, assuming S - D is more appropriate. Thus, both directions are reasonable and may simultaneously exist between the style factor and the domain D.
the sampling environment. If we determine the domain according
S
s
X
(a)
(b)
Figure 14: Causal models with different causal direction between S and D.
More importantly we do not not focus on discovering the concrete causal directions between variables in expand our H-NTL to D (we denote it H-NTL-D2S), We
this paper. For completeness, illustrate the main principle of our H-NTL-D2S, based on which we conduct basic NTL experiments show the effectiveness of H-NTL in the situation of D S
Main principle of the H-NTL--D2S. In order to simultaneously infer content factor C and style factor S from sampled data D D,, according the causal model in Fig. 14 (b), we factorize the joint distribution P(X, Y, C, S, D) as follows:
KXX.YDDCCSR =
(11)
Similar to the H-NTL focusing on D the main paper, we also use VAE (Kingma & Welling, 2013) to infer latent factors C' and S. Specifically, model the posterior distributions go. (CIX) and go (SIX) with learnable parameters de and o.. respectively. In addition,
two encoder modules 4s. and ás, to
decoder Po. to model the distribution Pa_ (XIC,S) parameterized by 8,, To reduce computational costs and to allow our method efficiently infer latent factors, we approximate P(C, D) by assuming that X has sufficient information about C while inferring C from S and D is insignificant (Yao et al., 2021; Lu et al., 2018). 1). Similarly, we assume C contains sufficient information predict the label Y. Thus, Po, (S)D) with learnable parameter 0, and classifier Pa, to model the distributions pe,, (Y|C) with learnalle parameter 0y. Totally, the VAE can be presented as: (go.: -o,- Pu,, Pa., po.
encoder Pa. model the distributions
We follow the variational inference framework (Blei et al., 2017) maximize the evidence lower- bound (ELBO) from the sampled observed data (I, 1, d) E D. The ELBO(x, d) is derived as:
ELBO(E,y,d) - -KL(u6. (clx))l - KL(gs, (s]x:)lpe, (sld)
+ Ee--s., log (ulc)]
(12)
+ +E
[log Po, (x|c,
The detailed derivation of Eq. (12) is similar to Appendix B. We train the VAE by maximizing ELBO(x, y, d) and thus obtain two encoders 9s. (x) and qo, (a:) for estimating unobservable content factor and style factor S, respectively. The remaining steps for non non-transferable representation learning the same as main paper.
27

---

Published as a conference paper at ICLR 2024
Experiments. As shown in Table 17, present both target-specified and source-only NTL results for H-NTL-D2S. These results demonstrate that H-NTL-D2S can significantly degrade target domain performance and maintain source domain performance simultaneously, thus achieving the goal of NTL. In summary, our H-NTL is still effective when considering D - S.
Table 17: Results of H-NTI--D2S on target-specified and source-only tasks. For each table cell, the first line shows averaged accuracy Acc (%) with standard deviation, and the second line shows accuracy drop A (behind 1) and relative drop A % (in brackets) compared to supervised learning (SL).
H-NTL--D2S
H-NTL-D22
Source- Target
Img Size
SL
(target-ppcciffed)
(source-only)
Source
Target
Source
Target
Source
Target
94.30 +0.9
97.47 40.40
93.15 +021
9.70
90.50 +0.99
19.90 #2.12
MM-MT
32
1 1.15 (1.229))))))
1 87.77 (90059))
3,000 (4.03%)
777.57 (19.888)
87.47 20.5
50.33 #5.32
88.00 +0.26
10.67 20:83
83.93 #1.00
9.57
SN-SD
32
1-0.53 (-0.11))
1 39 19.66 (78.808)
3.54 (4.05%)
4.000 (80.9098)
98.23 10.06
55.30 13.00
97.13 10.32
11.13 x3.11
91.77 11.56
12.80 17.57
SD-MT
32
1.1 (1.129)
(79888)))
6.46 (6.58%)
42,500 (76.85%)
86.57 10.38
67.60 40.95
88.07 +1.35
11.80 #3.56
88.53 +0.32
9.77 40.71
C10-S10
64
1 (-1.7%%))
1 55.80 (82.5453)
1-1.96 (-2.26%))
175.83 (85.55%)
93.40 +0.70
35.60 +1.56
93.85 +0.21
7.30 +0.42
95.05 +0.49
7.95 +1.77
VT-VV
64
1-0045 (-0.48%)
28.20 (79.4998)
1-1.65 (-1.77%6)
1/27.66 (77.67%)
75.60 11
31.63 21.02
73.30 21:13
7.80 20
69.30 21.84
15.05 13.04
OP-OC
64
1 22.30 (3.044%)
23.83 (75.3458)
6.30 (8.33%)
16.58 (52.42%6)
E.3
THE MOTIVATION OF USING VARIATIONAL INFERENCE FRAMEWORK
We follow the variational inference framework (Blei et al., 2017) to model the distribution of latent variables S) from the observed data, thus modeling the generative process. Other frameworks (such as auto-encoder (Wang et al., 2016)) cannot model the distribution. Moreover, there are also alternative options (for example, the VAEGAN (Xian et al., 2019; Chen et al., 2023b; Hong et al., 2022)) but with more complexity and may facing the problem of unstable training. Thus, the above factors are major motivations for us to the variational inference framework.
F LIMITATIONS AND FUTURE RESEARCH
In this section. discuss the potential limitations of our work and areas for future research. The major limitation of the proposed H-NTL its training efficiency. Due to the two-stage training processing (i.e., stage 1: disentanglement, stage 2: dual-path knowledge distillation), H-NTL needs more time and computation resources for training. Of course, the sacrifice of training time is worth if we need better performance. Nevertheless, improving the training efficiency of H-NTL could still become a valuable research direction in the future. However, it is worth noting that after being deployed in practice, our H-NTL will not have any disadvantages inference phase, we only need the student model fatt to predict. If the fall has the same network architecture as the network used in other NTL methods, H-NTL will occupy the same memory and have the same inference speed as other NTL methods. Moreover, because of the effective disentanglement and dual-path knowledge distillation, H-NTL allows a more lightweight student network to be taught to learn better non-transferable representations. This implies the potential advantages in the efficient inference of H-NTL in practical applications.
computational cost. In the
28


Reviews:
[{"review_id": "OsoOV8aErK", "reviewer": "Reviewer_AgKr", "summary": "The authors explore an important problem named non-transferable learning, which aims to reduce the performance of a method on target domains while keeping considerable performance on source domains. The authors discover that existing methods suffer from two challenges, i.e., spurious correlation and fake independence. To deal with these challenges, the authors propose a variational inference framework that explicitly considers the contents and styles in various domains. The extensive experiments conducted on various datasets further demonstrate the effectiveness.", "questions": "Have the authors considered the benefits of using a variantiational inference framework? Is it due to the stochasticity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The authors explore the important problem of non-transferable learning with two challenges investigated.\n\n3. The authors conduct extensive experiments to showcase the superior performance of the work.", "weaknesses": "1. The authors state that existing methods suffer from the limitations of spurious correlation and fake independence. However, the authors do not provide any quantitative evaluation regarding these two challenges, except only intuitions.\n\n2. The authors do not provide further details about the datasets used in the experiments. For example, the number of samples in each domain and each class. This will result in difficulties in understanding different datasets.\n\n3. In Table 2 presented in the experimental part, the authors only consider two baselines for comparison, which is not fair enough for a comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plznPpayZQ", "reviewer": "Reviewer_fXnt", "summary": "The paper proposes H-NTL, a new NTL method to address the issues of spurious correlation and fake independence present in many real-world implementations of NTL. The H-NTL method leverages a causal framework with two content and style latent factors to learn a variational inference framework. Empirical evaluations further support the strength of the proposed H-NTL method under various settings.", "questions": "Are there other standard NTL experimental setups using higher resolution datasets beyond the 32x32 or 64x64 images presented in the current paper?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written with presents a well-motivated justification for the H-NTL method.\n2. Additional supplementary and ablation studies provide further evidence for the methodology.\n3. H-NTL presents strong empirical performance in comparison with prior works.", "weaknesses": "No major weakness to note, however the reviewer would like to see additional evaluations with higher resolution images if possible (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zCJy8hMB4d", "reviewer": "Reviewer_BF2B", "summary": "The paper introduces a novel approach to non-transferable learning (NTL) that focuses on disentangling content and style factors, harnessing them as guidance for improved non-transferable representation learning. The concept of separating content and style for NTL is innovative and not commonly explored in existing literature. This approach presents a fresh perspective on NTL that has the potential to address the limitations of previous statistical methods. The novelty of this concept makes the paper stand out.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is technically sound and demonstrates a comprehensive understanding of the issues in NTL. It effectively presents a causal model to explain non-transferable learning and provides a well-structured method to address these issues. The utilization of a variational inference framework for disentanglement and dual-path knowledge distillation for learning non-transferable representations is well-reasoned and technically sound.\n\n2. The paper is well-organized and clearly written. It provides a coherent and logical progression from the problem statement to the proposed solution. The use of illustrative figures and the clear description of the causal model enhance the paper's overall clarity. However, the high complexity of the method and the underlying theoretical framework might make it challenging for some readers. To enhance clarity, it would be helpful to include examples or case studies illustrating the practical application of the method.\n\n3. The paper is significant in the context of non-transferable learning. It offers a promising approach that addresses the limitations of previous methods, particularly the challenges related to statistical dependence between source and target domains. The disentanglement of content and style factors is an important contribution as it aligns with human-like understanding and, as demonstrated through experiments, improves NTL performance. The potential impact on the AI research community lies in its ability to advance the field of NTL, making it more effective and practical.", "weaknesses": "1. The paper is generally well-structured, but it could benefit from the inclusion of practical examples or use cases to demonstrate the application of the proposed method.\n\n2. The experiments demonstrate the superiority of H-NTL over competing methods, which is a significant point in favor of the paper's claims. It would be helpful to include discussions on potential real-world applications where H-NTL could be particularly valuable.\n\n3. The introduction could be more concise and direct, focusing on the problem and motivation for the proposed approach.\n\n4. Minor grammatical and typographical errors should be addressed for a more polished final version.\n\n5. Including discussions on potential limitations and areas for future research could provide a more well-rounded view of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)