Harnessing Out-Of-Distribution Examples via Augmenting Content and Style
Authors: Zhuo Huang
Xiaobo Xia
Li Shen
Bo Han
Mingming Gong
Chen Gong
Tongliang Liu
Venue: ICLR 2023 poster (2023)
Keywords: out-of-distribution
open-set learning

Abstract: Machine learning models are vulnerable to Out-Of-Distribution (OOD) examples, such a problem has drawn much attention. However, current methods lack a full understanding of different types of OOD data: there are benign OOD data that can be properly adapted to enhance the learning performance, while other malign OOD data would severely degenerate the classification result. To Harness OOD data, this paper proposes HOOD method that can leverage the content and style from each image instance to identify benign and malign OOD data. Particularly, we design a variational inference framework to causally disentangle content and style features by constructing a structural causal model. Subsequently, we augment the content and style through an intervention process to produce malign and benign OOD data, respectively. The benign OOD data contain novel styles but hold our interested contents, and they can be leveraged to help train a style-invariant model. In contrast, the malign OOD data inherit unknown contents but carry familiar styles, by detecting them can improve model robustness against deceiving anomalies. Thanks to the proposed novel disentanglement and data augmentation techniques, HOOD can effectively deal with OOD examples in unknown and open environments, whose effectiveness is empirically validated in three typical OOD applications including OOD detection, open-set semi-supervised learning, and open-set domain adaptation.

Published as a conference paper at ICLR 2023
HARNESSING OUT-OF-DISTRIBUTION EXAMPLES VIA AUGMENTING CONTENT AND STYLE
Zhuo Huang', Xiaobo Xia', Li Shen?, Bo Han", Mingming Gong', Chen Gong",, Tongliang Liu
Syndey AT Centre, The University of Sydney; IJD Explore Academy:
$ "Department Computer Science, Hong Kong Baptist University; "The University Melbourne; KKey Laboratory of Intelligent Perception and Systems for High- figh-DDimnsiooal Information
Ministry Education, School Computer Science and Engineering. Nanjing University of Science and Technology (zhua0420) xxia5420) @uni. sydney.... edu. au, mathshenl @mmaill mila..... com, bhanml @ @emp...... hkbu. edu. hk, mingming saargmingg.gongennii.ee gongeunimee.......em..... edu.au. chen. leen.gonn@nnuste edu. cn, tongliang.o www.sydduenyyy edu. au
ABSTRACT
Machine learning models are vulnerable to Out-Of-Distribttion (OOD) examples, and such a problem has drawn much attention. However, current methods lack a full understanding of different types of OOD data: there are benign OOD data that can be properly adapted to enhance the learning performance, while other malign OOD data would severely degenerate the classification result. To Harness OOD data, this paper proposes a HOOD method that can leverage the content and style from each image instance to identify benign and malign OOD data. Particularly, we design a variational inference framework to causally disentangle content and style features by constructing structural causal model. Subsequently, we augment the content and style through an intervention process to produce malign and bemign OOD data, respectively. The benign OOD data contain novel styles but hold our interested contents, and they can be leveraged to help train style-invariant model. contrast, the malign OOD data inherit unknown contents but carry familiar styles, by detecting them can improve model robustness against deceiving anomalies. Thanks to the proposed novel disentanglement and data augmentation techniques, HOOD can effectively deal with OOD examples in unknown and open environments, whose effectiveness is empirically validated in three typical OOD applications including OOD detection, open-set remi-supervised learning, and open-set domain adaptation.
INTRODUCTION
Learning in the presence of Offist--Ditiiiutioonn (OOD) data has been challenging task in machine learning, as the deployed classifier tends to fail if the unseen data drawn from unknown distributions are not properly handled (Hendrycks & Gimpel, 2017: Pan & Yang, 2009). Such critical problem ubiquitously exists when deep models meet domain shift (Ganin et al., 2016; Tzeng et al., 2017) and unseen-class data (Hendrycks Gimpel, 2017: Scheirer et al., 2012), which has drawn a lot of attention in some important fields such as OOD detection (Hein et al., 2019; Hendrycks & Gimpel, 2017; Lee al., 2018; Liang et al., 2018; Liu et al., 2020; Wang et al., 2022a; 2023; 2022b), Open- Set Domain Adaptation (DA) (Liu et al., 2019; Saito et al., 201 8), and Open-Set Semi- Seni-Supervised Learning (SSL. (Huang al., 2021b; 2022b;a; Oliver et al., 2018; Saito et al., 2021; Yu et al., 2020).
In the above fields, OOD data can be divided into two types, namely benign OOD data' and malign OOD data. The benign OOD data can boost the learning performance on the target distribution through DA techniques (Ganin & Lempitsky, 2015; Tzeng et al., 2017), but they can be misleading if not being properly exploited. To improve model generalization, many positive data augmentation techniques (Cubuk al., 2018; Xie et al., 2020) have been proposed. For instance, the performance of SSL. (Berthelot et al., 2019; Sohn et al., 2020) has been greatly improved thanks to the augmented benign OOD data. On the contrary, malign OOD data with unknown classes damage the
"Corresponding to Tongliang Liu and Chen Gong.
We follow (Bengio al., 2011) to regard the augmented data a type of OOD data

---

Published as a conference paper at ICLR 2023
D
Y
D
Y
D
Y
intorvationn
intervention
S
C
S
C
S
S
C
X
X
X
X
(a)
(b)
Figure 1: (a) An ideal causal diagram which reveals the data generating process. (b) Illustration of
disentanglemenn. The brown-edged variables Č' and are approximations of content C' and style S. The dashed lines indicate the unwanted causal relations that should be broken. (c) Illustration of the data augmentation of HOOD. The green lines and red lines denote the augmentation of benign OOD data X and malign OOD data X, respectively. In all figures, the blank variables are observable and the shaded variables are latent.
classification results, but they are deceiving and hard to detect (Hendrycks & Gimpel, 2017; Liang et al., 2018; Wei et al., 2022b;a). To train a robust model against malign OOD data, some works (Kong & Ramanan, 2021: Sinha et al., 2020) conduct negative data augmentation to generate "hard" malign data which resemble in-distribution (ID) data. By separating such "hard" data from ID data, the OOD detection performance can be improved. When presented with both malign and benign OOD data, it is more challenging to decide which to separate and which to exploit. As a consequence, the performance of existing open-set methods could be sub-optimal due to two drawbacks: 1) radically exploiting too much malign OOD data, and 2) conservatively denying too much benign OOD data.
In this paper, we propose a HOOD framework (see Fig. 2) to properly harness OOD data in several OOD problems. To distinguish benign and malign OOD data, we model the data generating process by following the structural causal model (SCM) (Glymour et al., 2016; Pearl, 2009; Gao et al., 2022) in Fig. 1 (a). Particularly, we decompose an image instance X into two latent components: 1) content variable C' which denotes the interested object, and 2) style variable 5 which contains other influential factors such as brightness, orientation, and color. The content C' can indicate its true class Y, and the style S is decisive for the environmental condition, which is termed as domain D. Intuitively, malign OOD data cannot be incorporated into network training, because they contain unseen contents, thus their true classes are different from any known class; and benign OOD data can be adapted because they only have novel styles but contain the same contents as ID data. Therefore, we can distinguish the benign and malign OOD data based on the extracted the content and style features.
In addition, we conduct causal disentanglement through maximizing an approximated evidence lower-bound (ELBO) (Blei et al., 2017; Yao et al., 2021; Xia et al., 2022b) of joint distribution P(X, Y. D). As a result, we can effectively break the spurious correlation (Pearl, 2009; Glymour et al., 2016; Hermann et al., 2020; Li et al., 2021b; Zhang et al., 2022) between content and style which commonly occurs during network training (Arjovsky et al., 2019), as shown by the dashed lines in Fig. 1 (b). In the ablation study, we find that HOOD can correctly disentangle content and style, which can correspondingly benefit generalization tasks (such as open-set DA and open-set SSL) and detection task (such OOD detection).
To further improve the learning performance, we conduct both positive and negative data augmentation by solely intervening the style and content, respectively, as shown by the blue and red lines in Fig. 1 (c). Such process is achieved through backpropagating the gradient computed from an intervention objective. As result, style-changed data X must be identified as benign OOD data, and content- changed data X should be recognized as malign OOD data. Without including any bias, the benign OOD data can be easily harnessed to improve model generalization, and the malign OOD data can be directly recognized as harmful ones which benefits the detection of unknown anomalies. By conducting extensive experiments on several OOD applications, including OOD detection, open-set SSL, and open-set DA, we validate the effectiveness of our method on typical benchmark datasets. To sum up, our contributions are three-fold:
We propose unified framework dubbed HOOD which can effectively disentangle the content and style features to break the spurious correlation. As a result, benign OOD data and malign OOD data can be correctly identified based on the disentangled features.
We design a novel data augmentation method which correspondingly augments the content and style features to produce benign and malign OOD data, and further leverage them to enhance the learning performance.
We experimentally validate the effectiveness of HOOD on various OOD applications, including OOD detection, open-set SSL, and open-set DA.
2

---

Published as a conference paper at ICLR 2023
48, (CIX)
ls. ls. (YIc)
(X,Y,))
Ps (x,,5,
X
9o, (SIx)
S
4o, (DIS)
D
Figure 2: Architecture of the HOOD, The solid lines denote the inference flow, the dashed lines indicate the disentanglement of content and style, and the tildes stand for the approximation of the corresponding variables.
METHODOLOGY
In this section, propose our HOOD framework as shown in Fig. 2. Specifically, utilize the class labels of labeled data and the pscudo labels (Lee, 2013) of unlabeled data as the class supervision perform different types of data augmentation and
to capture the content feature. Moreover, regard the augmentation types as the domain supervision for each style. Thereby, each instance paired with class label y and a domain label d. Then, we apply two separate encoders ge and parameterized by 0g and 0g to model the posterior distributions qo, (C' X)) and qo, (S X). respectively. Subsequently, the generated C' and S are correspondingly fed into two fully-connected classifiers fc and fs parameterized by de and és. which would produce the label predictions qa. (Y C') and 9a, (D S). respectively. To further enhance the identifiability of C and S, decoder h with parameter v is employed to reconstruct the input instance based on its content and style.
Below, we describe the detailed procedures and components during modeling HOOD. We first introduce the proposed variational inference framework for disentangling the content and style based on the constructed SCM. Subsequently, we conduct intervention to produce benign OOD data and malign OOD data. Further, we appropriately leverage the benign and malign OOD data to boost the learning performance. Finally, we formulate the deployment of HOOD in three OOD applications.
2.1 VARIATIONAL INFERENCE FOR CONTENT AND STYLE DISENTANGLEMENT
First, we assume that the data generating process be captured by certain probability distributions. Therefore, according to the constructed SCM in Fig. 1 (a), the joint distribution P(X, D,, of the interested variables can be factorized as follows:
(I) Based on the SCM in Fig. 1 (a), Y and D are conditionally independent each other, i.e., Y T I D S), so we have P(Y, D C) = P(Y C, P(D) C,S). Similarly, we have P(C,S) P(S). Moreover, we can also know that Y is not conditioned on S, and D is not conditioned on C. Hence, we can further derive P(Y, | C, S) = P(Y CDD S).
P(X, Y, D. C,S) = P(C, PY, D C, S)X C,
However, the aforementioned spurious correlation frequently appears when facing OOD examples (Ar- jovsky et al., 2019). As a consequence, when variational inference is based on the factorization in Eq. 1. the approximated content C' and style Š could both directly influence Y and D, i.e., Y + ++C D and Y + 5 + D, thus leading to inaccurate approximations, However, the desired condition is Y C D and Y + S S D. We can see that the unwanted correlations Ö + D and S Y in Fig. 1 (b) is caused by erroneous posteriors P(D C) and P(Y (S). Therefore, to break the correlations, the posteriors 96. (D C") and qo. (Y S) which are correspondingly approximated by the decoders Φ. and de can be used as denominators to qo (o_(Y (Y C') and go. (D SS), respectively. In this way, we can successfully disentangle content C' and style S and ensure the decoding process of Y and would not be influenced by spurious features from and C, respectively. To this end, our factorization in Eq. I can be approximated as:
P() I C)P(() S)P(X C.S)
(2)
Lo (D O)qo. (Y S)
Then, we maximize the log- lieelieelioood of the joint distribution p(x, y. d) of each data point (x, y, d):
log pogp(x y. d) == log /.] p(x, u. d. s)dods,
3

---

Published as a conference paper at ICLR 2023
in which lower case to denote the values of corresponding variables. Due the integration
of latents C' and S is intractable, we follow variational inference (Blei et al., 2017) to obtain an approximated evidence lower-bound ELBO(x, y, d) of the log- likelihood in Eq. 3:
p(x,y, y. d)
log p(x, y, 2 (C.,))
log
= ELLOOOO,, y, d.
(4)
go(c, x))
Recall the modified joint distribution factorization in Eq. 2, we can have:
ELBO(x, v.d) = -B(c.))a((C.S10) log p(c)p(s) (y c)go, (d 8) s)p,
c,
4a (o(c, s x)go. (y s)gs, (d
-KI(a (c x))lp(()) - K [qo, (s x) ()))
Beraqa (CIx) [log qo. (y c)- log 1a. (d c)]
Banggo (S x) [log uo. (d 8) log yo. (y 8)]
((c.)))oo(C,SI)) [log (x c, s)]
(5a)
-ELBO (x d) (ergo:((())) [log qo. (d c)] E Bg (5|x) [log to. (v
8)]
(5b)
In Eq. 5a, the first two terms indicate the Kullback-Leibler divergence between the latent variables C' and and their prior distributions. In practice, we assume that the priors p(C') and p(S) follow standard multivariate Gaussian distributions. The third and fourth terms contain the approximated log-lite log-likelihoods of label predictions and the disentanglement
the content and style. The last term stands for estimated distribution of x. Note that in Eq. 5b, our approximated ELBO is composed of two parts: the original ELBO which could be obtained from the factorization in Eq. 1, and two regularization terms that aims to disentangle C' and S through maximizing the log-likelihoods log qo. (d <) and log fo. (y/ 8). which is shown by the dashed lines in Fig. 2. By maximizing ELBO, we can train an accurate class predictor which invariant to different styles. The detailed derivation is provided in supplementary material. Next, we introduce our data augmentation to assist in harnessing OOD examples.
2.2 DATA AUGMENTATION WITH CONTENT AND STYLE INTERVENTION
After disentangling content and style, we try to harness OOD examples via two opposite augmentation procedures, namely positive data augmentation and negative data augmentation which aim to produce benign OOD data and malign OOD data &, respectively, so to further enhance model generalization and improve robustness against anomalies. Specifically, to achieve this, positive data augmentation only conducts intervention on the style feature meanwhile keeping the content information the same; and the negative data augmentation attempts to affect the content feature while leaving the style unchanged, so as to produce malign OOD data, as shown in Fig. (b).
To achieve this goal, we employ adversarial data augmentation (Goodfellow et al., 2014; Miyato et al., 2018; Volpi et al., 2018) which can directly conduct intervention on the latent variables without influencing each other, thus it is perfect for our intuition of augmenting content and style. Particularly, by adding learnable perturbation e to each instance x, we can obtain malign OOD data & and benign OOD data with augmented content and style, respectively. For each data point (x, y, d), the perturbation e can be obtained through minimizing the intervention objective L(-):
c arg min L(x + y. d: Bc, Pe, Gz, px).
<:]|e||,<
where denotes the magnitude of the perturbation with -o-nrm. Since our goal of positive and negative data augmentation completely different, here the intervention objective is designed differently for producing x and x. For positive data augmentation, the intervention objective is:
Epos = (x( 0c), (x + e; 0.)) Loe (x. e; (x).
where the first term Ca(-) indicates the distance measured between the contents extracted from the original instance and its perturbed version, and the second term (-e(.) denotes the cross-entropy loss. By minimizing Epos. the perturbation e would not significantly affect the content feature, meanwhile introducing novel style that is distinct from its original domain d. Consequently, the augmented
4

---

Published as a conference paper at ICLR 2023
benign data with novel styles can be utilized to train style- model that is resistant to domain shift. Moreover, a specific style with domain label d' can be injected via modifying pos as:
p'p (x( 0e), 9x(x + e; 02)) + (g + e; 0s); d').
(8)
Different from Eq. 7, we hope minimize the cross-entropy loss such that the perturbed instance can contain the style information from a target domain d'. As result, the augmented benign data can successfully bridge the gap between source domain and target domain, and further improve the test performance in the target distribution.
As for negative data augmentation, the intervention objective is defined as:
Long La(gs (x; 0s): gx(x e; 0.)) Loe (So(g( + e; (c);
By minimizing Eneg. the perturbation would not greatly change the style information but would deviate the content from its original one with class label y. Subsequently, by recognizing the augmented malign data unknown, the trained model would be robust to deceiving anomalies with familiar styles, thus boosting the OOD detection performance.
To accomplish the adversarial data augmentation process, here we perform multi-step projected gradient descent (Madry et al., 2018; Wang et al., 2021). Formally, the optimal x and & can be iteratively found through:
xx' arg min Cxos(x + e'),, x = X * + arg min Gneg (x' + te)). ef:
(10)
et:lett|pp<e
where the final iteration / is set to 15 in practice. Further, the optimal augmented data will be incorporated into model training, which is described in the next section.
2.3 MODEL TRAINING WITH BENIGN AND MALIGN OOD DATA
Finally, based on the aforementioned disentan- glement and data augmentation in Sections 2.1 and 2.2, we obtain benign OOD data x and malign OOD data & from each data point (x, y, d), which will be appended to the benign dataset D and malign dataset D., respectively. For utilization of benign OOD data x. we assign with the original class label y and perform supervised training. For separation of malign OOD data x, we employ a one-vs-all classi- fier (Padhy et al., 2020) recognize them unknown data that is distinct from its original class label y. The proposed HOOD method is summarized in Algorithm 1. Below, we specify the proposed HOOD algorothm to three typi- cal applications with OOD data, namely OOD detection, open-set SSL, and open-set DA.
Algorithm 1 Training process of HOOD
1: Labeled set D) = {(x,, B.) unlabeled set D {(x:) )
2: for to Mar__ter do
Pre-train the variational inference framework
through maximizing ELBO in Eq. 5a;
4: Assigning pseudo labels y" for unlabeled data
D := [(xx; "f")
if == Augmentation Iter then
6:
Conduct Adversarial Data Augmentation to obtain X and x through Eq. 10;
Add and & into D and D, respectively;
8:
end if
9:
Enumerate D and conduct supervised training
for each x;
10:
Enumcrate D and recognize each unknown;
11:
end for
2.4 DEPLOYMENT TO OOD APPLICATIONS
Generally, in all three investigated applications, we are given a labeled set D {(x; 31k,99))1 11 containing labeled examples drawn from data distribution pl, and an unlabeled set Du = {xi composed of unlabeled examples sampled from data distribution pa Moreover, the label space of D' and D are defined as yd and y"M respectively.
OOD detection. The labeled set is used for training, and the unlabeled set is used as test set which contains both ID data and malign OOD data. Particularly, the data distribution unlabeled ID data Qid is the same distribution P. but the distribution of OOD data Pie is different from P. f.e., P% p * P The goal is to correctly distinguish OOD data from ID data in the test phase, During training, we conduct data augmentation to obtain domain label d, and then follow the workflow described by Algorithm obtain the model parameters. During test, only use the
5

---

Published as a conference paper at ICLR 2023
Table 1: Comparison with typical OOD detections methods. Averaged AUROC (%) with standard deviations are computed over three independent trails. The best results are highlighted in bold.
OOD dataset
LSUN
DTD
CUB
Flowers
Caltech
Dogs
TD dataset
SVHN
Likelihood
52.25
= 0.3
50.33
0.7
18. 06.76-0
0.6
17.38
0.2
51.54
+ 0.4
51.34
0
ODIN
55.72
0.2
53.37
0.3
52.70
0.4
50.47
0.7
56.41
= 0.4
61 16
0.3
Likelihoud Ratio
79.34
0.5
78.42
0.3
75.90
0.7
74.53
0.4
76.25
0.3
83.55
0.4
OpenGAN
83.77
0.4
80.36
0.5
77.40
0.8
79.26
0.5
86.66
0..5
80.84
0.5
HOOD
84.10
0.6
80.68
0.6
79.24
0.5
80.93
0.7
85.34
0.7
8T.58
0.8
ID datasct
CIFARIO
Likelihood
54.32
0.5
52.16
0.4
50.67
0.4
49.26
0.3
53.86
0.4
56.92
0.2
ODIN
58.60
0.3
55.39
0.6
58.48
0.7
51.44
0.9
59.36
0.4
64.82
0.5
Likelihord Ratio
81.41
0.6
79.77
0.5
79.35
0.8
77.
0.7
80.67
0.5
86.76
0.3
OpenGAN
81.03
0.4
81 29
0.8
82.81
1.0
82.32
0.4
86.78
+ 0.3
90 14
0.5
HOOD
86.12
0.6
83.64
0.5
83.53
0.6
81.56
0.8
87.24
0.8
90.86
0.6
content branch predict the OOD score which is produced by the one-vs-all classifier. An instance is considered as an ID datum if the OOD score is smaller than 0.5, and an OOD datum otherwise.
Open-set SSL. The labeled set D' and unlabeled set D" are both used for training, and they are sampled from the same data distribution with different label spaces. Specifically, the unlabeled data contain some ID data that have the same classes as DD' and the rest unlabeled OOD data are from some unknown classes that do not exist in D', formally, ) C """""" y 3M * Ø and ('(x 1) = (x( y), / E y² The goal is to properly leverage the labeled data and unlabeled ID data without being misled by malign OOD data, and correctly classify test data with labels in y The training process is similar to OOD detection, except that HOOD would produce an OOD score for each unlabeled data. If an unlabeled instance is recognized OOD data, it would be left out.
Open-set DA. The labeled set drawn from source distribution pl which is different from the target distribution P" of unlabeled set. In addition, the label space ydl is also subset of ) Therefore, the unlabeled data consist of benign OOD data which have the same class labels as labeled data, and malign OOD data which have distinct data distribution as well as class labels from labeled data, formally, p + p y C y , 3 Ø. The goal is to transfer the knowledge of labeled data the benign OOD data, meanwhile identify the malign OOD data unknown. In this application, we assign each target instance with domain label to distinguish them from other augmented data. Then we alter the positive data augmentation objective from Eq. 7 Eq. 8 and train the framework through A Algorithm 1. During test, HOOD would predict each target instance some class if is benign OOD data, and as unknown otherwise.
EXPERIMENT
In this section, we first describe the implementation details. Then, we experimentally validate our method on three applications, namely OOD detection, open-set SSL, and open-set DA. Finally, we present extensive performance analysis on our disentanglement and intervention modules. Additional details and quantitative findings can be found in the supplementary material.
3.1 IMPLEMENTATION DETAILS
In experiments, we choose Wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) for OOD detection and Open-set SSL tasks, and follow (You et al., 2020; Cao et al., 2019) to utilize ResNet50 pre-trained on Imagenet (Russakovsky et al., 2015) for Open-set DA. For implementing HOOD, we randomly choose 4 augmentation methods from the transformation pool in RandAugment (Cubuk et al., 2020), to simulate different styles. The pre-training iteration Augmentalion. Her is set to 100,000, and the perturbation magnitude = 0.03, following (Volpi et al., 2018) in all experiments. Next, we will experimentally validate HOOD in three applications.
3.2 OOD DETECTION
In OOD detection task, we use SVHN (Netzer et al., 2011) and CIFARIO (Krizhevsky et al., 2009) as the ID datasets, and use LSUN (Yu et al., 2015), DTD (Cimpoi et al., 2014), CUB (Wah et al., 2011), Flowers (Nilsback & Zisserman, 2006), Caltech (Griffin et al., 2007), and Dogs (Khosla et al., 2011) datasets the OOD datasets that occur during test phase. Particularly, to explore the model generalization ability, we only sample 100 labeled data and 20,000 unlabeled data from each class and conduct semi-supervised training, then test the trained model on the unlabeled OOD dataset.

---

Published as a conference paper at ICLR 2023
Table 2: Comparison with typical Open-set SSL methods. Averaged test accuracies (%) with standard deviations are computed over three independent trails. The best results are highlighted in bold.
Training dataset
CIFARIO
CIFAR100
of Labeled data
100
50
100
UASD
1: 0.9
76.74
1.7
1
61.68
1.2
65.97
2.4
DS3L
74.44 + 1.3
76.80
= 1.5
78.80
0.6
60.40
+ 0.5
64.35
1.5
67.65
+ 1.3
Clean Acc.
OpenMatch
MTCF
84.10
79.88 1: 1.3
85.30
81.41
+.4
1 1.0
87.92
83.92
1: 0.8
1.0
65.76
62.78
1 0.5
0.9
68.46
65.84
1 2.1
0.5
72.87
69.46
1: 0.6
1.4
12T
82.74 + 1.2
83.56
1.4
85.97
+ 0.8
65.16
- 1.2
67.58
0.9
71.96
1.1
HOOD
83.55 1: 1.2
84.16
1.5
86.22
1 2.7
66.39
1.7
68.03
2.6
73.32
0.6
UASD
39.36 + 1.2
41.38
+ 0.7
42.66 1.8
31.55 + 2.0
33.39 + 1.7
35.20
0.8
DS3L
39.97
0.8
12.58
0..
33.72
112::00 0.8
34.67
0.6
Compptd Acc.
MTCF
40.16
1.2
40.58
1.1
43.33
1: 0.7
32.72
0.8
34.33
2.3
35.53
0.6
OpenMatch
41.38
0.7
42.00
+ 0.6
45.79
0.8
35.98
+ 1.3
36.47
0.7
38.56
+ 0.6
T2T
41.39
1.6
45.56
89.88-5
1.5
1.7
39.61
+
1.6
HOOD
44.42
1.7
48.38
0.9
50.74
4 0.6
40.82
4 1.5
41.65
0.9
43.72
2.2
To evaluate the performance, we utilize AUROC (Hendrycks & Gimpel, 2017) which is an essential metric for OOD detection, and higher AUROC value indicates better performance.
For comparison, we choose some typical OOD detection methods including Likelihood (Hendrycks & Gimpel, 2017) which simply utilizes softmax score as the detection criterion, ODIN (Liang et al., 2018) which enhances the performance of Likelihood through adding adversarial attack, Likelihood Ratio (Ren et al., 2019) which modifies the softmax score through focusing on the semantic feature, and OpenGAN (Kong& & Ramanan, 2021) which can further improve the performance via separating the generated "hard" examples that are deceivingly close to ID data.
The experimental results are shown in Table 1, we ODIN, and Likelihood Ratio, and can outperform OpenGAN in most scenarios. When compared with softmax-prediin predi-piondddbaaneeeeeeee -bsed methods such as Likelihood and ODIN, HOOD surpasses them in large margin, as HOOD can correctly separate some overconfident OOD examples from ID data. As for Likelihood Ratio, method can achieve better performance through producing "hard" malign OOD data, thus successfully avoiding deceiving examples that are extremely close to ID data. Although both OpenGAN and HOOD generate "hard" malign data to train an open classifier, HOOD can successfully distinguish content and style thanks to the aforementioned disentanglement, thus avoid rejecting too much benign OOD data and further yield better detection performance than OpenGAN.
that HOOD can greatly surpass Likelihood,
3.3 OPEN-SET SSL
In open-set SSL task, we follow (Guo et al., 2020) to construct our training dataset using two benchmark datasets CIFAR10 and CIFAR100 (Krizhevsky et al., 2009), which contains 10 and 100 classes, respectively. The constructed dataset has 20,000 randomly sampled unlabeled data and a varied number of labeled data. Here the number of labeled data is set to 50, 100, and 400 per class in both CIFAR10 10 and CIFAR Moreover, to create the open-set problem in CIFAR 10, the unlabeled data is sampled from all 10 classes and the labeled data is sampled from the 6 animal classes. As for CIFAR 100, the unlabeled data are sampled from all 100 classes and the labeled data is sampled from the first 60 classes. For evaluation, and CIFAR 100 and denote the test accuracy as "Clean Acc.". Further, to evaluate the capability of handling OOD examples, we test on CIFARIO-C and CIFAR 100- (Hendrycks & Dietterich, 2019) which add different types of corruptions to CIFARIO and CIFAR 100, respectively. The test accuracy from the corrupted datasets can reveal the robustness of neural networks against corruptions and perturbations, and it is denoted as "Corrupted Acc.".
first the test dataset from the original CIFAR 10
For comparison, we choose some typical open-set SSL methods including Uncertainty- Self- Distillation method UASD (Chen et al., 2020) and T2T (Huang et al., 2021 a) which filters out the OOD data via using OOD detection, Safe Deep Semi- Semi-Supervised Learning DS3L. (Guo al., 2020) which employs meta-lcarning to down-weight the OOD data, Multi- Task Curricuuum Framework MTCF (Yu et al., 2020) which recognizes the OOD data as different domain, and OpenMatch (Saito al., 2021) which utilizes open-set consistency training OOD data.
The experimental results are shown in Table 2. Compared to the strongest baseline method Open- Match, which randomly samples eleven different transformations from a transformation pool, our method has transformations that are limited to only four types. In CIFAR1O and CIFAR100 regarding the Clean Acc., the proposed HOOD is slightly outperformed by OpenMatch. However, thanks to
7

---

Published as a conference paper at ICLR 2023
Table 3: Comparison with typical Open set DA methods. Averaged test accuracies (%) with standard deviations are computed over three independent trails. The best results are highlighted in bold.
Dataset
Office
VisDA
Domain
A-+W
A-D
D-W
W-+
D-+
W-+
Synthetic- Seatiic--ees Real
OSBP
56.5
2.0
88.6 + 1.4
97.0
+ 1.0
97.9 + 0.9
88,9 + 2.5
85.8
2,5
62.9 +: 1.3
LAN
57.7
+ 1.2
87.0 0.8
93.5
+ 1.3
97.2
= 166
85.4
0.7
87.8
+ 1.6
63.8
+ 2.4
STA
80.5 1. 0.6
93.7111.
1 1.5
97.5
1
0.2
99.5
1 0.2
89.1
0.5
87.9
1 0.9
66.4
1 1.3
HOOD
90.1 1.5
94.2
1.4
99.6
1 0.6
98.3
1 0.9
89.8
0.8
91.3
+ 1.8
72.4 1.6
the disentanglement, HOOD can be invariant to different styles and focus on the content feature. Therefore, when facing corruption, HOOD can be more robust than all baseline methods. As shown by the Corrupted Acc. results, our method surpasses OpenMatch for more than 3%.
3.4 OPEN-SET DA
In open-set DA task, we follow (Saito et al., 2018) to validate on two DA benchmark datasets Office (Saenko et al., 2010) and VisDA (Peng et al., 2018). Office dataset contains three domains Amazon (A), Webcam (W), and DSLR (D), and each domain is composed of 31 classes. VisDA dataset contains two domains Sythetic and Real, and each domain consists of 12 classes. To create an open-set situation in Office, we follow (Saito et al., 2018; Liu et al., 2019) to construct the source dataset by sampling from the first 21 classes in alphabetical order. Then, the target dataset is sampled from all 31 classes. As for VisDA, we choose the first 6 classes for source domain, and use all the 12 classes for target domain. We use "A- to indicate the transfer from "A" domain to "W" domain.
For comparison, we choose three typical open-set DA approaches including Open-Set DA by Back- Propagation OSBP (Saito et al., 2018) which employs an OpenMax classifier to recognize unknown classes and perform gradient flipping for open-set DA, Universal Adaptation Network UAN (You et al., 2020) which utilize entropy and domain similarity to down-weight malign OOD data, and Separate To Adapt STA (Liu et al., 2019) which utilizes SVM to separate the malign OOD data.
The experimental results are shown in Table 3. Compared to the baseline methods, the proposed HOOD is largely benefited from the generated benign OOD data, which have two major strengths: 1) they resemble target domain data by having common styles, and 2) their labels are accessible as they share the same content their corresponding source data. Therefore, through conducting supervised training such benign OOD data, the domain gap can be further mitigated, thus achieving better performance than baseline methods. Quantitative results show that HOOD can surpass other methods in most scenarios. Especially in VisDA, HOOD can outperform the second-bes method with 6% improvement, which proves the effectiveness of HOOD in dealing with open-set DA.
3.5 PERFORMANCE ANALYSIS
Table 4: Ablation study on necessity of cach module.
Ablation Study: To verify the effective- ness of each module, conduct an ablation study on three OOD applications by climi- nating one component at time. Specifically, our HOOD can be ablated into: "w/o disen- tanglement" which indicates removing the disentanglement loss in Eq. 5a, "w/o benign OOD data" which denotes training without benign OOD data, "w/o malign OOD data" which stands for discarding malign OOD data, and "w/o both augmentations" indicates training without both benign and malign OOD data. Here in OOD detection, we use CIFAR10 as the ID dataset, and use LSUN as the OOD dataset. In open-set SSL, we choose CIFAR10 with 400 labels for each class. As for open-set DA, we use VisDA dataset.
Application
detection
Open-Set SSL
Open-Set
w/o disentanglement w/o benign OOD data
81.91 = 1.3
82.55
+ 1.1
61.6 + 0.9
85.95 + 1.8
83.32
+ 2.0
66.3
+ 2.5
w/o malign OOD data
62.50
1 2.2
85,40
0.8
71.8
1 1.2
w/o both augmentations
80.83 + 0.8
81.14
1.2
65.4
+ 1.2
HOOD
86.12
86.22
1 2.7
72.4
The experimental results are shown in Table 4. We can see each module influences the performance differently in three applications. First, we can see that the malign OOD data is essential for OOD detection, as can act as unknown anomalies and reduce the overconfidence in unseen data. Then, benign OOD data can largely improve the learning performance in open-set SSL and open-set DA, as they can enforce the model to focus on the content feature for classification. Additionally, we can see that discarding both benign and malign OOD data shows performance degradation compared to both "w/o benign OOD data" " and "w/o malign OOD data". Therefore, our HOOD can correctly change the style and content, which can correspondingly benefit generalization tasks (such as open-set DA
8

---

Published as a conference paper at ICLR 2023
Analysis Augmentation Numbhr
Origized
bird
cal
car
ship
dog
cal. airplane
deer
Test Accuracy
90-
Negative
15
65-
Open SSL
OOD desection
Posllve
Frog
dog
Irog
airplane
deer
deer
hird
deer
Openeet DA
No. Data Augmentations
bird
cal
car
ship
dog
cat.
airplane
deer
Figure 4: Augmentation number analysis.
Figure 5: CIFAR1O Visualization of our data augmentation.
and open-set SSL.) and detection tasks (such as OOD detection). Moreover, open-set DA relies more on the disentanglement than the rest two modules, owing to the disentanglemen can exclude the style changing across different domains. Hence, our disentanglement can effectively eliminate the distribution change from different domains and help learn invariant features.
Analysis on Augmentation Number: Since
Class Prediction
Domain Prediction
HOOD does not introduce any hyper-parameter the most influential setting is the number of data augmentation. To analyze its influence the learning results, we vary the number of augmen- tations that are sampled from the Randengment
9.03
301
0.99
9/0
0RL
OAL
Content Input
Content Input
COL
0.34
Pool (Cubuk et al., 2020) from 2 to 6. The re- sults shown in Fig. 4. We
+
that both too less and too many augmentations would hurt the results. This is because a small augmentation number would undermine the generalization to various styles; and a large augmentation num- ber would increase the classification difficulty of the style branch, further making the disentan- glement hard to achieve. Therefore, setting the augmentation number to is reasonable.
5
033 026
0.25
60 0.00
8:22
0.00
301
6.11
0.44
8.00
02
0.0
0:20
035
0.4
6.2
Cluss Prediction
Domain Prediction
5
Style Input
115
8.07
C00
022
9/0
035
Furthermore, to show the effect of our data augmentations, we visualize the aug- mented images by applying large perturbation magnitude (4.7) Tsipras et al. (2018) in Fig. 5. The model prediction is shown below each im- age. We can see that the negative data augmen- tation significantly changes the content which is almost unidentifiable. However, positive data augmentation can still preserve most of the content information and only change the style of images. Therefore, the augmented data can be correctly leveraged to help train a robust classifier.
Visualization:
304
627
800
047
0. 34
Figure 3: Illustration of disentanglement between con- tent and style on CIFAR10. The number each cell denotes the prediction probability.
Disentanglement of Content and Style: To further testify that our disentanglement between content and style is effective, we select the latent variables from different content and style categories, and use the learned class and domain classifiers for cross- poos-ppreiition. Specifically, there are four kinds of input-prediction types: content-class, content-domann style-class, and style-domain. As we can see in Fig. 3, only the content features are meaningful for class prediction, and the same phenomenon goes for style input and domain prediction. However, neither of the style and content features can be identified by the class predictor and domain predictor, respectively. Therefore, we can reasonably conclude that our disentanglemen between content and style is effectively achieved.
CONCLUSION
In this paper, we propose HOOD to effectively harness OOD examples. Specifically, we construct a SCM to disentangle content and style, which can be leveraged to identify benign and malign OOD data. Subsequently, by maximizing ELBO, we can successfully disentangle the content and style feature and break the spurious correlation between class and domain. As result, HOOD can be more robust when facing distribution shifts and unseen OOD data. Furthermore, we augment the content and style through novel intervention process to produce benign and malign OOD data, which can be leveraged to improve classification and OOD detection performance. Extensive experiments are conducted to empirically validate the effectiveness of HOOD three typical OOD applications.

---

Published as a conference paper at ICLR 2023
5 ACKNOWLEDGEMENT
Li Shen was partially supported by the Major Science and Technology Innovation 2030 "New Generation Artificial Intelligence" key project No. 1021ZD0111700 Zhuo Huang was supported by JD Technology Scholarship for Postgraduate Research in Artificial Intelligence No. SC4103. Xiaobo Xia was supported by Australian Research Council Projects DE-190101473 and Google PhD Fellowship. Bo Han was supported by NSFC Young Scientists Fund No. 62006202. Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652 and RGC Early Career Scheme No. 22200720. Mingming Gong was supported by ARC 10210101 112111211101144 624. Chen Gong was supported by NSF of China No. 61973162, NSF of Jiangsu Province No. B72021013, NSF for Distinguished Young Scholar of Jiangsu Province No. BK20220080, and CAAI-Huawei MindSpore Open Fund. Tongliang Liu was partially supported by Australian Research Council Projects IC-190100031, LP-220100527, DP-220102121, and FT-220100318.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0289) 2019.
Mohammad Taha Bahadori, Krzysztof Chalupka, Edward Choi, Robert Chen, Walter F Stewart, and Jimeng Sun. Causal regularization. arXiv preprint arXiv:/702..0 011.77222222244 2017.
Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boolanger-Lewandowwkee Thomas Breuel, Youssouf Chherawala, Moustapha Cisse, Myriam Côté, Dumitru Erhan, Jeremy Eustache, et Deep learners benefit more from aut-of-distribution examples. In A/STATS, pp. 164-172. JMLR Workshop and Conference Proceedings, 2011.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, pp. 5049-5059, 2019.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association (JASA), 112(518):599--8777 2017.
Kaidi Cao, Maria Brbic, and Jure Leskovec, seni-s-ureddssmmi-ssseerrriese learning. In ICLR, 2022. URL hwwps:tt///... wpw.review.eeeeeeeeeeeeeeeeee.ee n wtatt//foummiiio o
Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang. Learning transfer examples for partial domain adaptation. In CVPR, pp. 2985-2994, 2019.
Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In AAAI, 2020.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613,2011
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment:
Learning augmentation policies from data. In CVPR, 2018.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In NeurIPS, volume 33, pp. 18613-18624, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, pp. 1 1180-1189, PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research (JMLR), (111):009-20030 2016.
Erdun Gao, Ignavier Ng, Mingming Gong, Li Shen, Wei Huang, Tongliang Liu, Kun Zhang, and Howard Bondell. Missdag: Causal discovery in the presence of missing data with continuous additive noise models. In NeurlPS, 2022.
10

---

Published as a conference paper at ICLR 2023
Yong Ge, Sergey Demyanov, Zetao Chen, and Rahil Gamavi. Generative openmax for multi- class open set classification. arXiv preprint arXiv:1707.07418, 2017.
Madelyn Glymour, Judea Pearl, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. ICML, pp. 2839-2848. PMLR,
2016.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, pp. 2672-2680, 2014.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Lan-Zhe Guo, Zhen-Yu Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semi- supervised learning for unseen-class unlabeled data. In ICML, 2020.
Rundong Hc, Zhongyi Han, Xiankai Lu, and Yilong Yin. Safc-student for safe deep semi-superrised learning with unseen-class unlaheled data. CVPR, pp. 14585-14594, 2022a.
Rundong He, Zhongyi Han, Yang Yang, and Yilong Yin. Not parameters should be treated equally: Deep safe supervuper learning under class distribution mismatch. In AAAI, volume 36, pp. 6874-6883, 2022b.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In CVPR, pp. 41-50, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
Katherine Hermann, Ting Chen, and Simon Korbllith. The origins and prevalence of texture bias in convolutional neural networks. In NeurlPS, volume 33, pp. 19000-19015, 2020.
Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning. In ICCV. pp. 8310-8319, 2021a.
Zhe Huang, Mary-Joy Sidhom, Benjamin S Wessler, and Michael C Hughes. Fix-a-step: Effective semi-supervised learning from uncurated unlabeled sets. arXiv preprint arXiv:2208. 11870, 2022a.
Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurlPS, volume 34, 2021b.
Zhuo Huang. Jian Yang, and Chen Gong. They are not completely useless: Towards recycling transferable unlabeled data for class-mismatched semi-supervised learning. IEEE Transactions on Multimedia, 2022b.
Maximilian Ilse, Jakub Tomezak, and Patrick Forré. Selecting data augmentation for simulating interventions. In International Conference Machine Learning, pp. 4555-4562. PMLR, 2021.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In CVPR Workshop on Fine-Grained Visual Categorization (FGVC), volume 2. Citeseer, 2011.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 131..61 14, 2013.
arXiv preprint
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In ICCV, pp. 813-822, 2021.
11

---

Published as a conference paper at ICLR 2023
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, 2013.
Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. In NeurIPS, volume 34, 2021.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of out-of-distribution samples and adversarial attacks. In NeurlPS, volume 31, 2018.
Guangrui Li, Guoliang Kang, Yi Zhu, Yunchao Wei, and Yi Yang. Domain consensus clustering for universal domain adaptation. In CVPR, pp. 9757-9766 2021a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial
feature learning. In CVPR, pp. 5400-5409, 2018.
Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan Yuille, and Cihang Xie. Shape-texture debiased neural network training. In ICLR, 2021b.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of -uttof-distribution image detection in neural networks. In ICLR, 2018
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie- Yan Liu. Learning causal semantic representation for out-of -f-isiiibbutin prediction. In NeurlPS, volume 34, 2021.
Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate to adapt: Open domain adaptation via progressive separation. In CVPR, pp. 2927-2936 2019.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurlPS, volume 33, pp. 14444-224775 2020.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato and Bernhard Schölkopf. Invariant causal representation learning for out-of-distribution generalization. In ICLR, 2021.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research (JMLR), 9(Nov):2579-2605, 2008.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Representa- tion learning via invariant causal mechanisms. arXiv preprint arXiv:2010.07922. 2020..
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 1178) 2018.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. NewrlPS, volume 33, pp. 20673-20684, 2020.
Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng- Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In ECCV, pp. 613-628, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning, NeurIPS Workshop, 2011.
M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In CVPR, volume 2, pp. 1447-1454. IEFE, 2006.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS, pp. 3235-3246, 2018
12

---

Published as a conference paper at ICLR 2023
Shreyas Padhy, Zachary Nado, Jie Ren, Jeremiah Liu, Jasper Snoek, and Balaji Lakshminarayanan. Revisiting one-vs-an ae-lvs classifiers for predictive uncertainty and dittrf--dittibuttion detection in neural networks. arXiv preprint arXiv:2007.05134, 2020.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering (TKDE), 22(10))1345-1355 2009.
Judea Pearl. Causality. Cambridge university press, 2009.
Xingehao Peng, Ben Usman, Neela Kaushik, Dequan Wang, Judy Hoffman, and Kate Saenko. Visda: A synthetic-to-real benchmark for visual domain adaptation. In CVPRW, pp. 2021-2026, 2018.
Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for nut-of-distribution detection. In NeurIPS, volume 32, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 (113))211-22551 112212, 2015.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, pp. 213-226. Springer, 2010.
Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In ECCV, pp. 153-168, 2018.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In NeurIPS, 2020.
Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with open-set consistency regularization. NeurlPS, volume 34, 2021.
Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition, IEEE transactions on pattern analysis and machine intelligence (TPAMI), 35 (7):1757-1772, 2012.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, and Kun Zhang. Robust learning via cause-effect models. arXiv preprint arXiv: 2011.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint 1804 10745, 2018
Zheyan Shen, Peng Cui, Kun Kuang, Bo Li, and Peixuan Chen. Causally regularized learning with agnostic data selection bias. In ACM Multimedia, pp. 411-419, 2018.
Abhishek Sinha, Kumar Ayush, Jiaming Song, Burak Uzkent, Hongxia Jin, and Stefano Ermon. Negative data augmentation. In /CLR, 2020.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi saiisuuer learning with consistency and confidence. arXiv preprint arXiv:2001.07653 2020.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In ICML, pp. 6056-6065. AMLR.20199
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2018.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, pp. 7167-7176, 2017.
13

---

Published as a conference paper at ICLR 2023
Riccardo Volpi. Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS, volume 2018.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In NeurIPS, volume 34, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mi Alex Smola, and Zhangyang Wang. Partial and asymmetric contrastive learning for cuttrf-distribution detection in long-tailes recognition. In ICML, pp. 23445883458, PMLR, 2022a.
Qizhou Wang, Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Probabilistic margins for instance reweighting in adversarial training. In NeurlPS, volume 34, pp. 33258-23200 2021.
Qizhou Wang, Feng Liu, Yonggang Zhang, Jing Zhang, Chen Gong, Tongliang Liu, and Bo Han. Watermarking for out-of-distribution detection. In NeurIPS, 2022b.
Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Kalander, Tongliang Liu, Jianye Hao, and Bo Han. Out-of-diitriiuttoo detection with implicit outlier transformation. In ICLR, 2023.
Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. Open-sampling Exploring out- of-distribution data for re-balancing long-tailed datasets. In ICLR, pp. 23615-23630. PMLR, 2022a.
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML. pp. 23631-23644, PMLR, 2022b.
Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Jun Yu, Gang Niu, and Masashi Sugiyama. Instance correction for learning with open-set noisy labels. arXiv preprint arXiv:2106.00555, 2021.
Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended t: Learning with mixed closed-set and open-set noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022a.
Xiaobo Xia, Wenhao Yang, Jie Ren, Yewen Li, Yibing Zhan, Bo Han, and Tongliang Liu. Pluralistic image completion with gaussian mixture models. NeurIPS, 2022b.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. NeurlPS, 2020.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. In NeurlPS, volume 34, pp.
2021.
Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Universal domain adaptation. In CVPR, 2020.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 550.03365, 2015.
Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In ECCV, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. ICML, pp. 819-827. PMLR, 2013.
Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Causalady: Adversarial robustness through the lens of causality. In ICL.R, 2022.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
In this supplementary material, we first discuss some related work in Section B. Then, we complement the implementation details of HOOD in Section C. Further, we present the detailed derivation of ELBO in Section D. Additionally, we discuss two notable causal graph assumptions in Section E. Moreover, to further understand the effectiveness of HOOD, we provide additional analysis in Section F. Finally, we will discuss the limitations and social impact of our method in Section G.
B RELATED WORK
OOD applications contains three typical problems. namely OOD detection, open-set SSL, and open-set DA. OOD detection Hendrycks & Gimpel (2017); Liang et al. (2018) aims to train a robust model which can accurately identify the newly-emerged malign OOD data during the test phase. Open-set SSL Guo et al. (2020); Chen et al. (2020); He et al. (2022a;b); Yu et al. (2020) deals with the problem when labeled data are scarce and the unlabeled data are contaminated by malign OOD data. As for open-set DA Saito et al. (2018); Liu et al. (2019), it tries to transfer the knowledge from source ID data to the benign OOD data in target domain, meanwhile detecting the malign OOD data that are encountered during transferring. In both three applications, the predictive confidence has been frequently leveraged to separate malign OOD data (Hendrycks & Gimpel, 2017: Liang et al., 2018; Ren et al., 2019; Xia et al., 2022a). Moreover, ID data and OOD data can be distinguished via using a discriminator (Kong& & Ramanan, 2021; Neal et al., 2018; Yu et al., 2020; Xia et al., 2021). Further, various open classifiers are designed to predict OOD dataset as unknown (Ge et al., 2017: Padhy et al.. 2020; Saito et al., 2018). Thanks to the advances in unsupervised learning, many approaches employ self-supervised learning to make ID data and OOD data separable (Cao et al., 2022; Li et al., 2021a; Saito et al., 2020) from each other.
Causality in OOD problems mainly focuses on learning invariant representations that stay constant when other causal factors are changing, thus achieving better performance when facing non-stationary data distribution. To accomplish this goal, it is common to learn causal factors and non-causal factors through the variational auto-encoder framework (Blei et al., 2017; Kingma & Welling, 2013). Thanks to which, domain adaptation (Gong et al., 2016; Schölkopf et al., 2011; 1; Zhang et al., 2013) and domain generalization (Li et al., 2018; Shankar al., 2018) can be tackled through extracting the domain invariant features. Moreover, based on causal effects, the biased feature can be eliminated through re-weighting (Bahadori et al., 2017; Shen al., 2018). Additionally, the spurious correlation which is harmful for inference could be alleviated through do-calculus (Lee et al., 2021; Nam et al., 2020; Pearl, 2009). Recent methods (Ilse et al., 2021; Mitrovic et al., 2020; Von Kügelgen al., 2021) conduct data augmentations with self-supervised learning to train a robust model that can handle distribution shifts and corruptions.
In general, HOOD has two major differences from existing methods in OOD applications and causality. On one hand, instead of treating an image instance as a whole commonly done in many approaches, HOOD can properly leverage OOD examples through their disentangled contents and styles. Moreover, augmenting content and style can help improve generalization and robustness simultaneously. On the other hand, current causal approaches are incapable of dealing with malign OOD data, but HOOD is able to learn style-invariant features from benign OOD data, meanwhile avoiding the damage brought by malign OOD data.
C COMPLEMENTARY DETAILS
Each trial of our experiments is conducted in one single NVIDIA 3090 GPU. In the open-set SSL and OOD detection tasks, we follow (Saito et al., 2021) by using the Wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) as the network backbone and train from scratch. In open-set DA application, we follow (Saito et al., 2018) to fine-tune ResNet50 pre- pre-trained on Imagenet (Russakovsky et al., 2015). The employed Stochastic Gradient Descent (SGD) optimizer starts with an initial learning rate 3e - 2 which is decayed by following the cosine function cos(conu (const& current iteration without warm-up, 500,000
in which const is constant, we follow (Saito et al., 2021) to set it as मे. The momentum factor is set to 0.9, which is also the same as (Saito et al., 20.1). For choosing the pseudo labels of unlabeled data, we follow (Lee, 2013) to set the pseudo label threshold as 0.95. The unlabeled examples with confidence smaller than 0.95 are excluded from training the class branch. However, all of them are leveraged to optimize the domain branch. Moreover, the content and style features are the output of
15

---

Published as a conference paper at ICLR 2023
the penultimate layer, which are further fed into the fully-connected layer produce the class and domain prediction. Here the class number is decided by the class labels, and the domain number is the number of augmentations plus one logit that stands for the original input.
D DERIVATION OF ELBO
In the main paper, the modified structural causal model factorized as:
P,,, = P(C))(S)P(Y C)P(D S)P(X CC,)) P(D C)P( |S)
(11)
We employ two encoders to model the distribution content and style, respectively:
ge. (C X), q8, (S X)..
(12)
Moreover, we utilize two classifiers to model the posteriors of class and domain, respectively:
q (Y CC, qs. (D) S).
(13)
Additionally, a decoder is employed to reconstruct the input instance through following distribution:
q (X C,S).
(14)
Then, our goal is to maximize the log- likelihood of the joint distribution p(x. y, d):
log p(x, y. d) log / p'(x,,, c,,
[./ U',x, s)
go(c, x)
log
deds
go x)
(15)
(' d, c,
log (C
go(c, x)
log p' (x..V.d..,s)
BB(cs)=no
EEBO(xx y, v,
go(c, x)
By applying Eq. 2 to pxx,, y, d, c, s). we have:
log p(c)p(s)gs. (y c)qs, (d s)pp(( (.8)) go(c, s x)qo. (y s)go, (d c) p(c)p(s)
ELBO (x, y.
=E ((c.)))00(.C51))
log Jo. c)
(CC.3))
log Q0. (c) x)40, (8 x)
B(c.))
2o. (d c)
log 9o. (d ())
(C
+ B(c.s) [Log s)]
qo. (u ())
p(c)
log p(c)p(s)
=E -E(o)_v. (CIx)
log 40 (c xx)
+ (ss)~ (Six)
ge, (s x)
log 1o. (y c)
log lo 8)
1 (c1x)
+ B(())go,
to. (d c)
To. (u 8)
E (cC.))
[logPx p (x c,
KT. x))) K T(qo, (8 x) x))(p(())
+ Eo~go, (CIx) [log lo. (y c) log 4o, (d c)]
+ E (91x) [log Go. (d 8) log qo (y) s)]
+ ((c.s) 1 (x c, c,,)),
(16)
which is the final ELBO in main paper.
16

---

Published as a conference paper at ICLR 2023
D
S
C
X
Y
D
(a)
Figure 6: Comparison of two different causal relationship assumptions.
E DISCUSSION OF THE CAUSAL RELATIONSHIP BETWEEN STYLE AND DOMAIN
There are two different assumptions for the causal relationship between style S and domain D: a) Style is caused by domain; and b) Style causes domain (as shown in Figure 6). The first one is more commonly used in existing literature Liu et al. (2021): Lu et al. (2021); Suter et al. (2019), which usually assume that the style of data depends the sampling environment. Such an assumption perfectly fits the natural data generating process. However, we follow Mitrovic et al. (2020); Von Kügelgen al. (2021) by using data augmentation to obtain different styles. During this process, each data augmentation only introduces specific style, then we manually assign domain label based on the style type. Hence, in this scenario, assuming that the style causes the domain label as scenario b) is more appropriate than the first one.
Furthermore, the causal direction between style and domain depends on the problem context. the domain is partitioned according to the style in the images, content is the cause of domain. For example, image recognition, one can partition images into domains according to the illumination levels. The domain is determined by the brightness of the images and the causal relation is brightness-ydomain. >dmmai.. On the other hand, one partition the domains according to time (daytime or night), the brightness level will be effect of the domain (or time), iee., domain -brighhtbrig Therefore, assumption b) holds in certain real-world scenarios.
F ADDITIONAL ANALYSIS
Original
ambulance
telephone
botdog
airliner
Negative
goblet
banjo
parking
Positive
ambulance
dial telephone
holdog
airliner
Figure 7: ImageNet30 visualization of our data augmentation. The model prediction is shown below each image.
F.1 VISUALIZATION OF HIGHER RESOLUTION IMAGES
In the main paper, we have provided the visualization augmented CIFAR10 images. To testify that positive data augmentation and negative data augmentation are still effective higher resolution images, we conduct experiments using ImageNet30 with resolution 256 x 256, and show the augmented benign data and malign data in Fig. We see the similar phenomenon in
17

---

Published as a conference paper at ICLR 2023
CIFARIO: In negative data augmentation, the objects completely unidentifiable which lead to erroneous model predictions. On the contrary, the positive data augmentation only changes the style (there three style types are presented: purple style, green style, and sharp-texture style) but leave the objects intact. As a result, the model predictions are usually correct. Therefore, our augmentation method can be effectively deployed to high resolution images.
F.2 T-SNE VISUALIZZTTION
To further demonstrate the effectiveness of disentanglement, show the t-SNE (Maaten & Hinton, 2008) visualization of the feature extracted with or without disentanglement, as shown in Fig. We can see that when training without disentanglement, the features are closely gathered in each cluster. However, the malign OOD data represented by gray and pink points are also intensely aligned in the clusters, which would damage the model robustness. In contrast, when training with disentanglement, there is slight gap between OOD data and ID data, which means that the model trained with disentanglement can avoid overfitting specific styles, and show better robustness against OOD data.
(a) without disentanglement
with disentanglement
Figure 8: t-SNE visualization on CIFARI0 10 dataset. The blue, brown, yellow, green, orange, and purple points are data, and the gray and pink points are OOD data.
F.3 EFFECT OF ADDING BENIGN AND MALIGN OOD DATA INTO TRAINING
To give a illustrative comparison of adding benign OOD data and malign OOD data into training, we conduct experiments under the open-set SSL setting and separately augmenting benign OOD data and malign OOD data to compare their effects. Moreover, we conduct plain training as a baseline that after
result which do not either augmentations. The results are shown in Fig. 9. We adding augmented data, the effect of malign OOD data causes sudden performance degradation. On the contrary, benign OOD data further improve the learning result compared the plain training baseline. Which again shows that preserving content and augmenting style is beneficial for improving generalization, and eliminating content harmful for learning.
Effeect of Denign and Malign OOD Data
1.0
0.9
0.8
Test Accumny
Add Augmented Data
Add Malign OOD Data
Add Benign OOD Data
Training Iteration
Figure 9: Effect of adding benign and malign OOD data into training.
18

---

Published as a conference paper at ICLR 2023
F.4 OOD SCORE
To show the effectiveness of identifying malign OOD data from benign OOD data, we test the performance of HOOD three applications observe the OOD scores of benign OOD data and malign OOD data and show the averaged OOD scores in Table 5. We can see that the OOD score produced by our one-vs-all classifier can clearly distinguish benign and malign OOD data during the test phase, which again validates the effectivvness of HOOD.
Table 5: Averaged OOD scores with standard deviations on three applications.
OOD score
Application
Benign OOD data
Malign OOD data
OOD detection
0.16 + 0.3
0.83 + 0.6
Open-Set SSL
0.08 + 0.5
0.91 + 0.4
Open-Set DA
0.21 + 0.4
0.88 + 0.3
F.5 EXECUTION EFFICIENCY
Additionally, to give quantitative comparison the execution efficiency of HOOD, here we provide the running time on 3090 GPU compared to some typical baseline methods. The results are shown in Table 6. Note that method involves causal disentanglement as well as adversarial training, therefore, the training time is more than other opponents.
Table 6: Execution efficiency comparisons on three applications.
OOD Detection
Open-set SSL
Open-set DA
Method
Time
Method
Time
Method
Time
Likelihood
6.2h
DS3L
15.4h
UAN
8.5h
OpenGAN
7.8h
OpenMatch
10.5h
STA
9.1h
HOOD
11.4h
HOOD
13.7h
HOOD
12.4h
G LIMITATION AND SOCIAL IMPACT
The proposed HOOD method has many advantages which have been demonstrated in the main paper. However, HOOD is still limited in some aspects. Firstly, HOOD contains an extra phase that computes the gradient to produce the augmented data, so it cannot be conducted in an end- to-end manner. Therefore, it is feasible to improve HOOD by designing more compact method and incorporating augmentation into the training process. Secondly, HOOD utilizes existing data augmentation techniques to simulate different styles, which cannot perfectly cover all the possible styles in the real world. Hence, better style simulation might further improve the learning performance of HOOD.
Regardless of the limitations, our method could have some positive social impacts. First, HOOD can be safely deployed into many open situations where many unknown classes exist. As practical problems contain lots of uncertainties and novel instances occur constantly. Thanks to the negative data augmentation of HOOD, the novel instances can be successfully identified and would not harm the prediction accuracy. Secondly, in many non-stationary environments, the knowledge of the backbone model can be easily transferred thanks to the positive data augmentation of HOOD, which further broadens the practical usage of HOOD in modern industry.
19


Reviews:
[{"review_id": "hfSegNyPvsh", "reviewer": "Reviewer_sqif", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To address three applications of OOD: OOD detection, Open-set SSL and Open-set DA, this work proposes a novel data augmentation method, named HOOD. They analyze the data generation process from a causal perspective, and uses a variational inference network to disentangle the content and style from the input data. Then, they perform data augmentation on the content and style, respectively, to obtain benign and malign OOD samples. Finally, the authors use these augmented samples to improve the generalization of the model. Extensive experiments on three OOD applications demonstrate the effectiveness of this method.", "summary_of_the_review": "Please refer to the “Weakness”.", "strength_and_weaknesses": "Strength:\n1. The proposed method can improve performance on all 3 OOD applications.\n2. The causal disentanglement method proposed in this paper can separate content and style, which is novel.\n\nWeakness:\n1. According to Algorithm 1, each iteration needs to pre-train the VAE, which consumes a lot of computational cost and running time cost. The author needs to give the time complexity of running the algorithm, and the running time of the experiment, and compare it with the baseline methods.\n2. The author only carried out three kinds of OOD applications through experiments, but what is the proposed methods in the traditional OOD setting with supervised learning, i.e., the training set and the test set do not belong to the same distribution. And how does this data augmentation method perform compared to other traditional generalization algorithms, such as ERM, DRO [1] or IRM [2].\n3. Why use adversarial data augmentation, the author did not give sufficient reasons and proofs. For example, after we can decouple the content and style, why not use random data augmentation, or other data augmentation methods.\n4. Whether the adversarial data augmentation can hurt the performance. For example, adversarial training will degrade the performance of the model on clean image.\n\n[1] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization\n\n[2] Invariant Risk Minimization\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DqKvXks-Kq5", "reviewer": "Reviewer_btZ1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a data augmentation method that separates the content and style of images to solve the OOD (out-of-distribution) problem. To this end, the content and style features of the image are causally separated. By keeping one of the two features similar to the original image and widening the distance of the other features, benign OOD and malign ODD data are generated, respectively. Experimental results show that the proposed algorithm is applicable to various OOD problems.", "summary_of_the_review": "Overall, I read this paper as interesting and positive for acceptance. Adding some qualitative and quantitative experiments may further improve the quality of this paper.", "strength_and_weaknesses": "This paper is well-written and interesting. The formulations used in this paper are novel. Various organized experiments show that the proposed algorithm works practically well in various situations and datasets and, in most cases, shows SOTA performance. It was good to show the effect of using benign and malign samples in an ablation study.\n\nDespite the many advantages of this paper mentioned earlier, I am puzzled by the qualitative results of data augmentation. According to Figure 4, despite the data augmentation, the image content does not seem to change significantly between positive and negative. If more qualitative results were provided, it would be better to understand the operational consequences of data augmentation. Also, I wonder if the augmentation algorithm will work well for higher-resolution images (e.g., ImageNet image examples).\n\nComparing the effects of two data augmentation methods not covered in this paper (maintaining both content and style or moving them away) would also be helpful in an ablation study.\n\nSimple error: In chapter 3.1, on page 3, the operator for a conditional independent is duplicated twice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "A3fgrBG6aF", "reviewer": "Reviewer_Jm4f", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework that identifies and utilizes benign and malign OOD based on the disentanglement of content and style features. The benign OOD data may contain new styles but hold contents, and so they can help improve style-invariant model training. The malign OODs are vice versa and can confuse OOD applications. The proposed method is designed as a variational inference framework based on a structural causal model. Data augmentation based on this decomposition improves performance in three representative OOD applications. ", "summary_of_the_review": "This paper introduces a novel view and framework for handling OOD instances, which seems interesting and reasonable. However, more comparisons with recent models and a clearer description of the model details would improve the paper further. ", "strength_and_weaknesses": "[+] The idea of defining and discriminating benign and malign OOD data based on the disentanglement of content and style is novel and interesting. \n\n[+] The paper presents a clear explanation of how to harness the ODD data with the proposed framework in several OOD problems. Its effectiveness is validated through various experiments. \n\n[-] The authors need to provide a more specific model architecture for HOOD for reproducibility. \n\n[-] It is not clearly presented how well the proposed method disentangles content and style, although its effectiveness is shown in terms of performance at OOD applications. It will help if the authors can provide at least brief validation results on this. Figure 4 is not very informative and quite hard to recognize. \n\n[-] According to Figure 3, the results seem sensitive to the number of augmented samples (benign OOD data). It is not in line with the assumption that benign OOD data can improve model performance.\n\n[Q1] In Algorithm 1, the usage of unlabeled input data is not well represented. Is it a required input or optional, or depends on the target task? Which part of the pseudo-code does it affect? \n\n[Q2] The authors explain that the pseudo-labels can be utilized in the case of unlabeled data, but I am curious about how sensitive the result is depending on how the pseudo-labels are obtained. \n\n[Q3] It will be helpful if the authors can show examples of benign and malign OOD samples using images. \n\n[Q4] The authors chose to use adversarial data augmentation that adds a learnable perturbation. I wonder if there is any other option for this, and whether it’s optimal. \n\n[Q5] I wonder if there are more recent models for comparison (e.g., ODIN was introduced in 2017, and other methods shown in Table 1 are quite traditional, except OpenGAN) and how this one performs in comparison with those. \n\n[Minor]\n\n- The term ID is used without being introduced. \n- p.2:  by the blue and red lines in Fig. 1 -> by the green line and ... \n- p.9 (right above Figure 4) can effective eliminate \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "31HCYW1NEW", "reviewer": "Reviewer_MQ8c", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new approach called HOOD to disentangling the content and style features, and then basing on them, design a data augmentation method to construct benign and malign OOD data. They validate the proposed approach on the tasks of OOD detection, open-set SSL, and open-set DA.", "summary_of_the_review": "My main concerns are on the assumptions over the causal diagrams and on the identifiability, and the experimental results also seems not quite impressive. ", "strength_and_weaknesses": "**Strength:**\n\nThe problem studied in this paper is quite important to the community. The motivation is convincing and the logic is easy to follow. The paper is generally well written. \n\n **Weaknesses:**\n\n+ My main concern is about the causal diagrams presented in Fig. 1. \n   - In Fig. 1a, the authors assume that the style $S$ causes the environment/domain $D$. Apparently, it contradicts the common assumption widely used in the literature that the environment $D$ causes the style $S$, *e.g., Magliacane et al., 2018; Suter et al., 2019; Wang and Jordan, 2021; Sun et al., 2021; Lu et al., 2022; Quinzan et al., 2022*. Why do the authors make this assumption? Is there any particular reason for this? How practical is it in comparison with the common assumption?\n   - Also, in Fig. 1a, the authors do not assume any dependence between $S$ and $C$, which is quite limited in real world scenarios. One commonly spurious correlation between $S$ and $C$ is due to the existence of the environment $D$, see *Sun et al., 2021; Liu et al., 2021; von Kugelgen et al., 2021*. In other words, we usually have the spurious path $S \\leftarrow D \\rightarrow C$, which also contradicts the assumed causal diagram in Fig. 1b.\n   -  In fact, the caption of Fig. 1a is incorrect. Note that, a SCM is defined as a set of functional assignments accompanying with its corresponding causal diagram. Hence, Fig. 1a is only a causal diagram, no a SCM. \n   - In Fig. 1c, both causal edges and non-causal edges are mixed up, which could be misleading to readers. \n\n+ Another main concern is about the identifiability. That is, how is it guaranteed, both theoretically and practically, that the content can be distinguished from the style only from the observed data? This is one of the key problems in the thread of work on causal disentanglement, *e.g., Arjovsky et al., 2019; Sun et al., 2021; von Kugelgen et al., 2021*.\n\n+ From the experimental results, it seems that the performance improvement is small, most only having around 1% improvement.\n\n+ notation abusing, e.g., the capitals are used in Fig. 1, whilst the letters are used in the text, etc. \n+ unexplained terms, e.g., \n  - what do you mean by ID data? There exists no explanation before it first appears in the paper. \n  - no explanation on the open-set methods in the introduction.\n+ grammar errors, e.g., \n  - the first sentence in the abstract contains two independent clauses, but without a conjunction, etc.\n  - on line 6 of the abstract, the indefinite article \"a\" is missed before \"HOOD method\".\n  - on line 4, \"deep models meet with domain shift ...\" should be replaced with \"deep models meet domain shift\"?\n\n**References**\n\n- Suter et al., *Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness.* ICML 2019.\n- Wang and Jordan. *Desiderata for representation learning: A causal perspective.* 2021.\n- Quinzan et al., *Learning counterfactually invariant predictors.* 2022.\n- Magliacane et al., *Domain adaptation by using causal inference to predict invariant conditional distributions.* NeurIPS 2018.\n- Lu et al., *Invariant causal representation learning for out-of-distribution generalization.* ICLR 2022.\n- Sun et al., *Recovering Latent Causal Factor for Generalization to Distributional Shifts.* NeurIPS 2021.\n- Liu et al., *Learning causal semantic representation for out-of-distribution prediction.* NeurIPS 2021.\n- Arjovsky et al., *Invariant risk minimization.* 2019.\n- von Kugelgen et al., *Self-supervised learning with data augmentations provably isolates content from style.* NeurIPS 2021. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster