Energy-Based Test Sample Adaptation for Domain Generalization
Authors: Zehao Xiao
Xiantong Zhen
Shengcai Liao
Cees G. M. Snoek
Venue: ICLR 2023 poster (2023)
Keywords: domain generalization
energy-based model
test-time sample adaptation
variational inference

Abstract: In this paper, we propose energy-based sample adaptation at test time for domain generalization. Where previous works adapt their models to target domains, we adapt the unseen target samples to source-trained models. To this end, we design a discriminative energy-based model, which is trained on source domains to jointly model the conditional distribution for classification and data distribution for sample adaptation. The model is optimized to simultaneously learn a classifier and an energy function. To adapt target samples to source distributions, we iteratively update the samples by energy minimization with stochastic gradient Langevin dynamics. Moreover, to preserve the categorical information in the sample during adaptation, we introduce a categorical latent variable into the energy-based model. The latent variable is learned from the original sample before adaptation by variational inference and fixed as a condition to guide the sample update. Experiments on six benchmarks for classification of images and microblog threads demonstrate the effectiveness of our proposal.

Published as a conference paper at ICLR 2023
ENERGY-BASED TEST SAMPLE ADAPTATION FOR DOMAIN GENERALIZATION
Zehao Xiao', Xiantong Zhen 12,, Shengeai Liao", Cees G. M. Snoek' AIM Lab, University of Amsterdam 'Inception Institute of Artificial Intelligence
ABSTRACT
In this paper, we propose energy- -assed sample adaptation at test time for domain generalization. Where previous works adapt their models to target domains, we adapt the unseen target samples source-trained models. To this end, design a discriminative energy-based model, which is trained on source domains to jointly model the conditional distribution for classification and data distribution for sample adaptation. The model is optimized to simultaneously learn classifier and an energy function. To adapt target samples to source distributions, we iteratively update the samples by energy minimization with stochastic gradient Langevin dynamics. Moreover, to preserve the categorical information in the sample during adaptation, we introduce a categorical latent variable into the energy-based model. The latent variable is Iearned from the original sample before adaptation by varia- tional inference and fixed as condition guide the sample update. Experiments on six benchmarks for classification of images and microblog threads demonstrate the effectiveness of our proposal.
INTRODUCTION
Deep neural networks are vulnerable to domain shifts and suffer from lack of generalization on test samples that do not resemble the ones in the training distribution (Recht et al., 2019; Zhou et al., 2021; Krueger et al., 2021; Shen et al., 2022). To deal with the domain shifts, domain generalization has been proposed (Muandet et al., 2013; Guirajani & Lopez-Paz, 2020; Cha et al., 2021). Domain generalization strives to learn a model exclusively on source domains in order to generalize well on unseen target domains. The major challenge stems from the large domain shifts and the unavailability of any target domain data during training.
To address the problem, domain invariant learning has been widely studied, e.g., (Motiian al., 2017; Zhao et al., 2020; Nguyen et al., 2021), based on the assumption that invariant representations obtained on source domains are also valid for unseen target domains. However, since the target data is inaccessible during training, it is likely an "adaptivity gap" (Dubey et al., 2021) exists between representations from the source and target domains. Therefore, recent works try to adapt the classification model with target samples at test time by further fine-tuning model parameters (Sun et al., 2020; Wang et al., 2021) or by introducing an extra network module for adaptation (Dubey et al., 2021). Rather than adapting the model to target domains, Xiao et al. (2022) adapt the classifier for each sample at test time. Nevertheless, a single sample would not be able to adjust the whole model due to the large number of model parameters and the limited information contained in the sample. This makes it challenging for their method to handle large domain gaps. Instead, we propose to adapt each target sample to the source distributions, which does not require any fine-tuning or parameter updates of the source model.
In this paper, propose energy based test sample adaptation for domain generalization. The method is motivated by the fact that energy-based models (Hinton, 2002; LeCun et al., 2006) flexibly model complex data distributions and allow for efficient sampling from the modeled distribution by Langevin dynamics (Du & Mordatch, 2019; Welling & Teh, 1). Specifically, we define a new discriminative energy-based model as the composition of classifier and netural-netoor-baaaa energy function in the data space, which are trained simultaneously on the source domains. The trained model iteratively
"Currently with United Imaging Hcaltheare, Co., Ltd., China.

---

Published as a conference paper at ICLR 2023
updates the representation of each target sample by gradient descent of energy minimization through Langevin dynamics, which eventually adapts the sample to the source data distribution. The adapted target samples then predicted by the classifier that is simultaneously trained in the discriminative energy-based model. For both efficient energy minimization and classification, we deploy the energy functions on the input feature space rather than the raw images.
Since Langevin dynamics tends to draw samples randomly from the distribution modeled by the energy function, it cannot guarantee category equivalence. To maintain the category information of the target samples during adaptation and promote better classification performance, we further introduce a categorical latent variable our energy- basse model. Our model learns the latent variable to explicitly carry categorical information by variational inference in the classification model. We utilize the latent variable as conditional categorical attributes like in compositional generation (Du et al., 2020a; Nie et al., 2021) to guide the sample adaptation to preserve the categorical information of the original sample. At inference time, we simply ensemble the predictions obtained by adapting the unseen target sample to each source domain as the final domain generalization result.
We conduct experiments on six benchmarks for classification of images and microblog threads to demonstrate the promise and effectiveness of our method for domain generalization'.
METHODOLOGY
In domain generalization, we are provided source and target domains as non-overlapping distributions on the joint space X' x y/ where X' and ) denote the input and label space, respectively. Given dataset with S source domains D.= { {Dj}_
T ==1 and T target domains D,= {Di) (=1' a model is trained only on DD and required to generalize well on DD, Following the multi-source domain generalization setting (Li et al., 2017; Zhou al., 2021), we assume there are multiple source domains with the same label space to mimic good domain shifts during training.
In this work, we propose energy-based test sample adaptation, which adapts target samples to source distributions to tackle the domain gap between target and source data. The rationale behind our model that adapting the target samples to the source data distributions is able to improve the prediction of the target data with source models by reducing the domain shifts, shown in Figure (left). Since the target data is never seen during training, we mimic domain shifts during the training stage to learn the sample adaptation procedure. By doing so, the model acquires the ability to adapt each target sample to the source distribution at inference time. In this section, we first provide a preliminary on energy-based models and then present our energy-based test sample adaptation.
2.1 ENERGY- BASED MODEL PRELIMINARY
Energy-based models (LeCun al., 2006) represent any probability distribution p(x) for RDD as po (x() = (xp(-En(x)) where Eg (x) RD + R is known as the energy function that maps each input sample to scalar and Z. exp( Es (x)) denotes the partition function. However, Ze is usually intractable since it computes the integration over the entire input space of x. Thus, we cannot train the parameter f of the energy-based model by directly maximizing the log- likellhoooodd xx) - Zog Nevertheless, the ligeliillhood has the derivative (Du & Mordatch, 2019; Song & Kingma, 2021):
Zo
log (x) = -E (x)
alog pa (x) E Exx(x)
(E)
[Ep(x)
Exp (XC)
00
00
00
where the first expectation term is taken over the data distribution pa(x) and the second one is over the model distribution po(x) (x).
The objective function eq. (1) encourages the model to assign low energy to the sample from the real data distribution while assigning high energy to those from the model distribution. To do so, we need to draw samples from po (x).. which is challenging and usually approximated by MCMC methods (Hinton, 2002). An effective MCMC method used in recent works (Du & Mordatch, 2019; Nijkamp et al., 2019; Xiao et al., 2021b; Grathwohl et al., 2020) is Stochastic Gradient Langevin
Code available: https: /ii thub. com/ 7zzxx 22. SA- CIRR2023.
2

---

Published as a conference paper at ICLR 2023
Source classiter
Class
Class
Source
Target
Adaptation:
Adaptation
Figure I: Illustration of the proposed energy-based model. It aims to adapt the target samples to the source distributions, which can be more accurately classified by the source domain classifier (left). With Langevin dynamics, each target sample is adapted iteratively to the source data distributions, which are represented as the gradient colors from green to red (right). Best viewed in color.
Dynamics (Welling & Teh, 2011), which simulates samples by
X xOEo(x') Ox
+e, s.l., 6~ ((1,A)
where À denotes the step-size and x" is drawn from the initial distribution (x(),, which is usually uniform distribution (Du & Mordatch, 2019; Grathwohl et al., 2020).
Actually, maximizing log (x) is equivalent to minimizing the KL divergence DRr. (pax(pa(x) (l)))())))))))))))))))))))) (Song & Kingma, 2021), which is alternatively achieved in (Hinton, 2002) by minimizing contrastive divergence:
DeL (pa(x))lpo DD (go (x)(<x)xx
where (x() IT (x)),, representing t sequential MCMC transitions starting from p(x) (Du et al., 2021a) and minimizing eq. (3) achieved by minimizing:
Ep.(x) [x)] Estop (x))) [x)) + Eg (x0) [esto__raad(0) gr) (x) + E (x0) (logge
Eq. (4) avoids drawing samples from the model distribution po(x), which often requires an exponen- tially long time for MCMC sampling (Du et al., 2021a). Intuitively, (x() is closer to pe (x) than (x)x). which guarantees that DKL (px))))) 2 DDL (x))) and eq. (3) can only be zero
when (x) = (x).
2.2 ENERGY-BASED TEST SAMPLE ADAPTATION
We propose energy-based test sample adaption tackle the domain gap between source and target data distributions. This is inspired by the fact that Langevin dynamics simulates samples of the distribution expressed by the energy-baad based model through gradient-based updates, with no restriction on the sample initialization if the sampling steps are sufficient (Welling & Teh, 2011; Du & Mordatch, 2019). We leverage this property to conduct test sample adaptation with Langevin dynamics by setting the target sample as the initialization and updating it iteratively. With the energy- eergy-based model of the source data distribution, as shown in Figure (right), target samples are gradually updated towards the source domain and with sufficient update steps, the target sample will eventually be adapted the source distribution.
Discriminative energy-based model. We propose the discriminative energy-based model po.s (x, y) on the source domain, which is constructed by a classification model and an energy function in the data space. Note that x denotes the feature representations of the input image 1, where x is generated by neural network backbone x=f (T). Different from the regular energy- based models that generate data samples from uniform noise, goal is to promote the discriminative task, i.e., the conditional distribution p(y[x), which preferred to be jointly modeled with the feature distributions p(x) of the input data. Thus, the proposed energy-based model is defined the joint space of
3

---

Published as a conference paper at ICLR 2023
X' x 3' to consider both the classification task and the energy function. Formally, the discriminative energy-based model of source domain is formulated as:
(x. (x. y) p)(y)x) exp( 1 Ex(
po.
Z
where (x()(x) denotes the classification model and (x) is an energy function, which is nowadays implemented with neural networks. Eq. (5) enables the energy-based model to jointly model the feature distribution of input data and the conditional distribution
the source domains. An unseen target sample X₁ is iteratively adapted to the distribution of the source domain D, by Langevin dynamics update with the energy function (x)) and predicted by the classification model (
The model parameters 0 and & can be jointly optimized following eq. (3) by minimizing:
DKL (pa(x. y)ll Pe (x. y)) - (ge p) y)),
(6)
which is derived as:
L = (x)()))) [log p + (x)(x))) [x( gradpp (x,s)) [Eo
+ Eg (x.y) [Estop (x) log graoppgrad( (y/x)]
where Pa(x. y) denotes the real data distribution and qo.> y) = IT's p(x. y) denotes sequential MCMC samplings from the distribution expressed by the energy-based model similar to eq. (4) (Du et al., 2021a). We provide the detailed derivation in Appendix A.
In eq. (7), the first term encourages to learn discriminative classifier on the source domain. The second and third terms train the energy function to model the data distribution of the source domain by assigning low energy the real samples and high energy the samples from the model distribution. Different from the first three terms that directly supervise the model parameters and o, the last term stops the gradients of the energy function Eg and classifier while back-propagating the gradients to the adapted samples qo y). Because of the stop-gradien gradient, this term does not optimize the energy or log-likelihood of a given sample, but rather increases the probability of such samples with low energy and high log-likelihood under the modeled distribution.
Essentially, the last term trains the model 0 to provide variation for each sample that encourages its adapted version to be both discriminative on the source domain classifier and low energy on the energy function. Intuitively, it supervises the model to learn the ability to preserve categorical information during adaptation and find a faster way to minimize the energy.
pabel-preserving adaptation with categorical latent variable. Since the ultimate goal to correctly classify target domain samples, it is necessary to maintain the categorical information in the target sample during the iterative adaptation process. Eq. (7) contains a supervision term that encourages the adapted target samples to be discriminative for the source classification models. However, as the energy function Fg operates only in the X' space and the sampling process of Langevin dynamics tends to result in random samples from the sampled distribution that are independent of the starting point, there is no categorical information considered during the adaptation procedure.
To achieve label-preserving adaptation, we introduce a categorical latent variable into the energy function to guide the adaptation of target samples to preserve the category information. With the latent variable, the energy function Eg is defined in the joint space of X' Z. The categorical information contained in z will be explicitly incorporated into the iterative adaptation. To do so, we define the energy-based model with the categorical latent variable as:
/ Pa (x, Z)dz = |polylz (z, (z(x) exp(-Ein (xoo)))))))) dz,
(x, y)
(8)
Zo
where @ denotes the parameters of the classification model that predicts z and y and Fg denotes the energy function that models the distribution of x considering the information of latent variable z. z is trained to contain sufficient categorical information of x and serves as the conditional attributes that guide the adapted samples x preserving the categorical information. Once obtained from the original input feature representations x, z is fixed and taken as the input of the energy function together with the updated x in each iteration. Intuitively, when is updated from the target domain to the source domain via Langevin dynamics, z helps it preserve the classification information contained in the original x, without introducing additional information.
4

---

(10) Published as a conference paper at ICLR 2023
Pot (z|(x))
Langevin dynamics
E
To (x')
E
Shared backbone
Φ
pai (z'(x')
E energy of positive samples
gx d')
E energy of negative samples
Figure 2: Overall process of the proposed sample adaptation by discriminative energy-based model. In each iteration, we train the classification model 6² and energy function Eg: of one source domain D'. I' and 1s denote the images from domains D' and D respectively. I' denotes one batch of images, which generate the center features d'. The energy function Eg. is trained by using x² positive samples and adapted samples qa (x') generated by x from other domains negative samples. The adaptation is achieved by Langevin dynamics of Egs. During inference, the target samples are adapted by Langevin dynamics of the energy function Eg of each source domain and then predicted by eq. (11).
To learn the latent variable z with more categorical information, estimate by variational inference and design a variational posterior go (z)z))), where dx is the average representation of samples from the same category as the source domain. Therefore, go (z)zee xx) can be treated as probabilistic prototypical representation of a class. By incorporating qs z dx) into eq. (8), we obtain the lower bound of the log-likellng likelihood:
(9) Note that in eq. (9), the categorical latent variable z is incorporated into both the classification model ( z, x) and the energy function Eg (x|z). The energy function contains both data information and categorical information. During the Langevin dynamics update for sample adaptation, the latent variable provides categorical information in each iteration, which enables the adapted target samples to be discriminative.
log Pe, (x. 2 Ege [log po(y - Ex z) - log Ze + ID [gol (
By incorporating eq. (9) into eq. (6), we derive the objective function with the categorical latent variable as:
Cj E p)(().)) [ea(e)] log p (y (y|z,, Iz, x)] + DKLlgg (p (z)x)]
(x)
(pp()) [Ea [x (x/z)]
(rtop_grad(as(x)) (E (x ]
+ Bxe( E E Estop Bopp
gradd()) (x, 2)
log Pstop_ (y/z, x DKL. (itop.(sto_grra(( dx) (stop (z/x)]
where px(x) and (x) denote the data distribution and the sequential MCMC samplings from the energy-based distribution of the source domain D,. Similar to eq. (7), the first term trains the classification model on the source data. The second term trains the energy function to model the source data distribution. The last term is conducted on the adapted samples to supervise the adaptation procedure. The complete derivation is provided in Appendix A. An illustration of our model shown in Figure 2. We also provide the complete algorithm in Appendix B.
Ensemble inference. Since the target data is inaccessible during training, we train the specific parameters 0 and ø to model each source distribution by adapting the samples from other source domains to the current source distribution. In each iteration, we train the energy-based model 8 of one randomly selected source domain DJ. The adapted samples generated by samples x x* from the other source domains DJ, D are used as the negative samples while x' as the positive samples train the mnergy-based model. During inference, the target sample is adapted to each source distribution with the specific energy function and predicted by the specific classifier. After that, we combine the predictions of all source domain models obtain the final prediction:
p(y.)
Σ Po (yiz",
z" p(z" xx), x ~ po (x).
(11)
Here of and 6 denote the domain specific classification model and energy function of domain D Note that since the labels of the x' are unknown, dy in eq. (10) is not available during inference.
5

---

Published as a conference paper at ICLR 2023
Table Benefit of energy-based test sample adaptation. Experiments on PACS using ResNet-18 averaged over five runs. Optimized by eq. (7), our model improves after adaptation. With the latent variable (eq. 10) performance improves further, both before and after adaptation.
Adaptation
Photo
Art-painting
Cartoon
Sketch
Mean
Without latent variable (eq. 7)
94.73 #0.22
78.66
+0.59
78.24 40.71
78.34
#0.62
82.49 #026
94.59 10.16
80.45 +0.52
79.98
+0.51
79.23
+0.32
83.51 #0.30
With latent variable (eq. 10)
x
95.12 +0.41
79.79
60
79.15
+0.37
79.28
+0,82
83.33
#0.43
96.05 +0.37
82.28 +0:31
81.55
20.65
79.81
20.41
84,92
20.59
Therefore, we draw z" from the prior distribution p(z") ((z"[xe), |xe), where x; is the original target sample without any adaptation. With fixed z", x Pg. (x) are drawn by Langevin dynamics as in eq. (2) with the target samples x' as the initialization sample. Pa. Pa. (x) denotes the distributions modeled by the energy function Eg. (x|z"). Moreover, to be efficient, the feature extractor & for obtaining feature representations x is shared by all source domains and only the energy functions and classifiers are domain specific. We deploy the energy-based model on feature representations for lighter neural networks of the Nomain-specitf energy functions and classification models.
3 EXPERIMENTS
Datasets. We conduct our experiments on five widely used datasets for domain generalization, PACS (Li et al., 2017), Office-Home (Venkateswara et al., 2017), DomainNet (Peng et al., 2019), and Rotated MNIST and Fashion-MNIST. Since we conduct the energy- -bargy distribution the feature space, our method can also handle other data formats. Therefore, we also evaluate the method on PHEME (Zubiaga et al., 2016), dataset for natural language processing.
PACS consists of 9,991 images of seven classes from four domains, i.e., photo, art-painting, cartoon, and sketch. We use the same training and validation split (Li al., 2017) and follow their "leave- one-out" protocol. Office-Home also contains four domains, i.e., art, clipart, product, and real-world, which totally have 15,500 images of 65 categories. DomainNet is more challenging since it has six domains i.e., clipart, infograph, painting, quickdraw, real, sketch, with 586,575 examples of 345 classes. We use the same experimental protocol as PACS. We utilize the Rotated MNIST and Fashion-MNIST datasets by following the settings in Piratla et al. (Piratla et al., 2020). The images are rotated from 0 to 90 in intervals of 15,, covering seven domains. We use the domains with rotation angles from 15 to 75% as the source domains, and images rotated by 0 and 902 as the target domains. PHEME is a dataset for rumour detection. There are a total of 5,802 tweets labeled as rumourous non-rumourous from 5 different events, i.c., Charlie Hebdo, Ferguson, German Wings, Ottawa Shooting, and Sydney Siege. Same as PACS, we evaluate our methods on PHEME also by the "leave-one-out" protocol.
Implementation details. We evaluate on PACS and Office-Homa with both a ResNet-13 18 and ResNet- 50 (He et al., 2016) and DomainNet with ResNet-50. The backbones are pretrained ImageNet (Deng et al., 2009). On PHEME we conduct the experiments based on a pretrained DiST (Sanh et al., 2019), following (Wright Augenstein, 2020). To increase the number of sampling steps and sample diversity of the energy functions, we introduce a replay buffer B that stores the past updated samples from the modeled distribution (Du & Mordatch, 2019). The details of the models and hyperparameters are provided in Appendix C.
Benefit of energy-based test sample adaptation. We first investigate the effectiveness of our energy-baa based test sample adaptation in Table 1. Before adaptation, we evaluate the target samples directly by the classification model of each source domain and ensemble the predictions. After the adaptation to the source distributions, the performance of the target samples improves, especially on the art-painting and cartoon domains, demonstrating the benefit of the iterative adaptation by our energy- mnaegy-based model. The results of models with latent variable, i.e., trained by eq. (10), are shown in the last two rows. With the latent variable, the performance is improved both before and after adaptation, which shows the benefit of incorporating the latent variable into the classification model. The performance improvement after adaptation is also more prominent than without the latent variable, demonstrating the effectiveness of incorporating the latent variable into the energy function.

---

Published as a conference paper at ICLR 2023
[o]
(b)
[c]
Souroa: photo
Source: cartoon
Source: aketch
larget: ort prittiig
Target: ort-poleting
Target: ortspoieting
Category:
Eategory: person
Category: gultar
-to Probabiitty house: 0.02%
de 25
Probability person: 46.0%
Probability on guiter:
99.7%
86.7%
1.9%
84.8%
Figure 3: Iterative adaptation of target samples. We adapt the samples from the target domain (art-painting) to different source domains (photo, cartoon, and sketch in (a), (b), (c)). Each subfigure shows the adaptation of one target sample. The adaptation procedure of the target samples is represented by the gradient color from green to red. In each figure, the target sample has the same label as the source data, but is mispredicted due to domain shifts. During adaptation, the target samples gradually approach the source distributions and eventually get correct predictions.
Effectiveness of iterative test sample adaptation by Langevin dynamics. We visualize the iterative adaptation of the target samples in Figure 3. In each subfigure, the target and source samples have the same label. The visualization shows that the target samples gradually approach the source data distributions during the iterative adaptation by Langevin dynamics. After adaptation, the predictions of the target samples on the source domain classifier also become more accurate. For instance, in Figure 3 (a), the target sample of the house category predicted incorrectly, with a probability of house being only 0.02%. After adaptation, the probability becomes 99.7%, %, which is predicted correctly. More visualizations, including several failure cases, are provided in Appendix D.
Adaptation with different Langevin dynam- ics steps. We also investigate the effect of the Langevin dynamics step numbers during adap- tation. Figure shows the variety of the aver-
90
0.4
age energy and accuracy of the target samples adapted to the source distributions with different updating steps. The experiments on PACS with ResNet-18. The target domain is art-painting. With the step numbers less than 80. the average energy decreases consistently while the accuracy increases along with the in- creased number of updating steps, showing that the target samples are getting closer to the source distributions. When the step numbers are too large, the accuracy will decrease as the number of steps increases. We attribute this to hav- ing imperfect categorical information, since approximated during inference from single target sample x, only. In this case, the label information would not be well preserved in x, during the Langevin dynamics update, which causes an accuracy drop with a large number of updates. To demonstrate this, we conduct the experiment by replacing p(z:) with (z(dx) during inference. dy is the class center of the same class as x,, Therefore, 4s (z(z)d contains categorical information that is closer to the ground truth label. We regard this as the oracle model. As expected, the oracle model performs better as the number of steps increases and reaches stability after 100 steps. We also show the results without z. We can see the performance and stability are both worse, which again demonstrates that helps preserve label information in the target samples during adaptation. More- over, the energy without conditioning on z is higher. The reason can be that without conditioning on there is guidance of categorical information during sample adaptation. In this case, the sample can be adapted randomly by the energy-based based model, regardless of the categorical information. This can lead to the conflict adapt the target features different categories of the source data, slowing
Final accuracy
80
Averau energgy
conducted
0.2
60
50
wtthout
0.1
with plzje)
with cld60) toradee
50
200
Number Langevin updating steps
Figure 4: Adaptation with different Langevin dynamics steps. As the number of steps increases, energy decreases while accuracy increases. When the number of steps is too large, the accuracy with- out z or with p(z) p(z[x) drops slightly while the accu- racy with g(z)dx) more stable and better.
7

---

Published as a conference paper at ICLR 2023
Table 2:
Comparisons on image and text datasets. Our method achieves the best mean accuracy for all datasets, independent of the backbone. Larger adaptation steps (i.e., 50) lead to better performance.
PACS
Office-Home
DomainNet
PHEME
ResNet-18 18
ResNet-50
ResNet-18
ResNet-50
ResNet-50
DistiBERT
Twasawa & Matsuo (2021)
81.40
85,10
57.00
68.30
Zhou et al. (2020a)
82.83
84,90
65.63
67.66
Gulrajani & Lopez-Paz (2020)
85,50
66,50
40.90
Wang et al. (2021)
83.09
86.23
64.13
67.99
75.8 +0.33
Dubey et al. (2021)
68.90
43.90
Xiao ct al. (2022)
84.15
87.51
66.02
71.07
76.1 +1021
This paper w/o adaptation
83.33 +0.43
86.05
+0.37
65.01
+0.47
70.44
=0:55
42.90
=0:34
75.4 +0.13
This paper w/ adaptation (10 steps)
84,25 +0.48
87.05
40.26
65.73
+0.32
71.13
#0.33
43.75
=0.49
76.0 +11.16
This paper w/ adaptation (20 steps)
84,92 10.59
87,70
10.38
66.31
10.21
72.07
10:38
44.66
:0.51
76.5 1.0.18
This paper w/ adaptation (50 steps)
85.10
10.33
88.12
10:25
66.75
10.21
72.25
10:32
44.98
10.43
76.9 10.16
down the decline of the energy. We provide more analyses of in Appendix E. In addition, the training and test time cost is also larger as the step number increases, the comparisons and analyses also provided in Appendix E.
Comparisons. PACS, Office- Home, and DomainNet are three widely used benchmarks in domain generalization. We conduct experiments PACS and Office- Home based on both ResNet- 18 and ResNet-50 and experiments on DomainNet based on ResNet-50. As shown in Table 2, our method achieves competitive and even the best overall performance in most cases. Moreover, our method performs better than most of the recent test-time adaptation methods (Iwasawa & Matsuo, 2021; Wang al., 2021: Dubey et al., 2021). which fine-tunes the model at test time with batches of target samples. By contrast, we strictly follow the setting of domain generalization. We only use the source data to train the classification and energy- -ased models during training. At test time, we do our sample adaptation and make predictions on each individual target sample by just the source-trained models. Our method more data efficient at test time, avoiding the problem of data collection per target domain in real-world applications. Despite the data efficiency during inference, our method is still comparable and sometimes better, especially on datasets with more categories, e.g., Office Hotice-Hmee and DomainNet. Compared with the recent work by Xiao et al. (2022), our method is at least competitive and often better. To show the generality of our method, we also conduct experiments on the natural language processing dataset PHEME. The dataset is binary classification task for rumour detection. The results in Table show similar conclusions as the image datasets.
Table 2 also demonstrates the effectiveness of sample adaptation. For each dataset and backbone, the proposed method achieves a good improvement after adaptation by the proposed discriminative energy-based model. For fairness, the results without adaptation are also obtained by ensemble predictions of the source-doman source-domain-see -opain-ppeeiific classifiers. Moreover, larger steps (i.e., 50) lead to better performance. The improvements of adaptation with 50 steps are slight. Considering the trade-off of computational efficiency and performance, we set the step number as 20 in our paper. We provide detailed comparisons, results on rotated MNIST and Fashion-MNIS' datasets, as well as more experiments on the latent variable, corruption datasets, and analyses of the ensemble inference method in Appendix E.
4 RELATED WORK
Domain generalization. One of the predominant methods is domain invariant learning (Muandet
al., 2013; Ghifary et al., 2016; Motian et al., 2017: Seo et al., 2020: Zhao et al., 2020; Xiao et al., 2021a; Mahajan et al., 2021; Nguyen et al., 2021; Phung et al., 2021; Shi et al., 2022). Muandet et al. (2013) and Ghifary et al. (2016) learn domain invariant representations by matching the moments of features across source domains. Li et al. (Li et al., 2018b) further improved the model by learning conditional-invarian features. Recently, Mahajan et al. (2021) introduced causal matching to model within-class variations for generalization. Shi et al. (2022) provided a gradient matching to encourage consistent gradient directions across domains. Arjovsky et al. (2019) and Ahuja et al. (2021) proposed invariant risk minimization to learn an invariant classifier. Another widely used methodology is domain augmentation (Shankar et al., 2018; Volpi et al., 2018; Qiao et al., 2020; Zhou et al., 2020a;b; Yao et al., 2022), which generates more source domain data to simulate domain shifts during training.
8

---

Published as a conference paper at ICLR 2023
Zhou et al. (2020b) proposed data augmentation the feature space by mixing the feature statisties of instances from different domains. Meta-learning-bas based methods have also been studied for domain generalization (Li et al., 2018a; Balaji et al., 2018; Dou et al., 2019; Du et al., b021b; Bui et al., 2021; Du et al., 2021c). Li et al. (201 8a) introduced the model agnostic meta-learning (Finn et al., 2017) into domain generalization. Du et al. (2020b) proposed the meta-variational information bottleneck for domain-invarisnn learning.
Test-time adaptation and source-free adaptation. Recently, adaptive methods have been proposed to better match the source-trained model and the target data at test time (Sun et al., 2020; Li et al., 2020; D' Innnocente et al., 2019; Pandey et al., 2021: Iwasawa & Matsuo, 2021; Dubey al., 2021; Zhang et al., 2021). Test-time adaptation (Sun et al., 2020; Wang et al., 2021; Liu et al., 2021: Zhou & Levine, 2021) fine-tunes (part of) a network trained on source domains by batches of target samples. Xiao et al. (2022) proposed single-sample generalization that adapts model to each target sample under a meta-learning framework. There are also some source-free free domain adaptation methods (Liang et al., 2020; Yang et al., 2021; Dong et al., 2021; 1; Liang al., 2021) that adapt the source-trained model on only the target data. These methods follow the domain adaptation settings to fine-tune the source-trained model by the entire target set. By contrast, we do sample adaptation at test time but strictly follow the domain generalization settings. In our method, no target sample is available during the training of the models. At test time, each target sample is adapted to the source domains and predicted by the source-trained model individually, without fine-tuning the models.
Energy-based model. The energy- -based model is a classical learning framework (Ackley et al., 1985; Hinton, 2002; Hinton et al., 2006; LeCun et al., 2006). Recently, (Xie et al., 2016; Nijkamp et al., 2019; 2020; Du & Mordatch, 2019; Du et al., 2021a; Xie et al., 2022) further extend the energy-based model to high-dimensional data using contrastive divergence and Stochastic Gradient Langevin dynamics. Wang et al. (2023) utilize the energy- based model for effective self-supervised pretraining of vision models. Different from most of these works that model the data distributions, some recent works model the joint distributions (Grathwohl et al., 2020; Xiao et al., 2021b). our work, we define the joint distribution of data and label to promote the classification of unseen target samples in domain generalization. We further incorporate a latent variable to incorporate the categorical information into the Langevin dynamics procedure. Eneegy-bbseed models for various tasks have been proposed, e.g., image generation (Du et al., 2020a; Nie et al., 2021), et al., 2020), and anomaly detection (Dehaene et al., 2020; Wang et al., 2022). Some methods also utilize energy-based models for domain adaptation (Zou et al., 2021; Xie et al., 2021; Kurmi et al., 2021). Different from these methods, we focus on domain generalization and utilize the mergy-based model to express the source domain distributions without any target data during training.
detection (Liu
5 CONCLUSION AND DISCUSSIONS
In this paper, we propose a discriminative energy-based model to adapt the target samples to the source data distributions for domain generalization. The energy-based model is designed the joint space of input, output, and a latent variable, which is constructed by a domain specific classification model and an energy function. With the trained energy- -hared model, the target samples are adapted to the source distributions through Langevin dynamics and then predicted by the classification model. Since we aim to prompt the classification of the target samples, the model is trained to achieve label-preserving adaptation by incorporating the categorical latent variable. We evaluate the method on six image and text benchmarks. The results demonstrate its effectiveness and generality. We have not tested our approach beyond image and text classification tasks, but since our sample adaptation is conducted on the feature space, should be possible to extend the method to other complex tasks based on feature representations.
Compared with recent model adaptation methods, our method does not need to adjust the model parameters at test time, which requires batches of target samples to provide sufficient target infor- mation. This is more data efficient and challenging at test time, therefore the training procedure is more involved with complex optimization objectives. One limitation of our proposed method is the iterative adaptation requirement for each target sample, which introduces an extra time cost at both training and test time. The problem can be mitigated by speeding up the energy minimization with optimization techniques during Langevin dynamics, e.g., Nesterov momentum (Nesterov, 1983), or by exploring one-step methods for sample adaptation. We leave these explorations for future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
REFERENCES
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive Science, 91):147-169, 1985.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- -hn--hhisttophe Gagnon- Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances Neural Information Processing Systems, volume 34, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:/907.028880 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. MetaReg: Towards domain gen- cralization using meta-reguulrrizttion In Advances in Neural Information Processing Systems, volume 31, pp. 998-1008, 2018.
Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specifee features to enhance domain generalization. Advances in Neural Information Processing Systems, volume 34, 2021.
Junbum Cha, Sanghyuk Chun. Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems, volume 34, 2021.
David Dehaene, Oriel Frigo, Sébastien Combrexelle, and Pierre Eline. Iterative energy-based
projection normal data manifold for anomaly localization. In International Conference Learning Representations, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009.
Antonio D' I'Innccette, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Learning to generalize one sample a time with self-supervision. arXiv preprint 11Xiv:1910.03915, 2019.
Jiahua Dong, Zhen Fang, Anjin Liu, Gan Sun, and Tongliang Liu. Confident induued multi- source free domain adaptation, In Advances in Neural Information Processing Systems, volume 34, pp. 2848-2860, 2021.
Qi Dou, Daniel C Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, 2019.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, volume 32, 2019.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, volume 33, pp. 6637-6647, 2020a.
Yilun Du, Shuang Li. Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training energy based models. In International Conference on Machine Learning. PMLR, 202 la.
Yingjun Du Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees G M Snock, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference Computer Vision, pp. 200-216, 2020b.
10

---

Published as a conference paper at ICLR 2023
Yingjun Du, Xiantong Zhen, Ling Shao, and Cees G M Snoek. MetaNorm: Learning normalize
few-shot batches across domains. In International Conference on Learning Representations. 2021b.
Yingjun Du, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Hierarchical variational memory for few-shot learning across domains, arXiv preprint 1rXiv:2112.08/1 2021c.
Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. IEEE Conference on Computer Vision and Puttern Recognition, pp. 14340-14349, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter compo- nent analysis: unified framework for domain adaptation and domain generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(7):1414 1e:::114444444 1430, 2016.
Will Grathwohl, Kuan-Chich Wang. Jörn- Hennik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference Learning Representations, 2020.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2020.
Hongyu Guo. Yongyi Mao, and Richong Zhang. Mixup as locally linear out-of-manifold regulariza- tion. In AAAI Conference on Artificial Intelligence, volume 33, pp. 3714-3722, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. International Conference on Learning Representations, 2020.
Geoffrey Hinton, Simon Osindero, Max Welling, and Yee-Whye Teh. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive Science, 30(4):725-731, 2006.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-188) 2002.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, volume 34, 2021.
David Krueger, Ethan Caballero, Joern- Hern-- Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. tion. In International Conference on Machine Learning, pp. 5815-5826. PMLR, 2021.
generalization via risk extrapola-
Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. /EEE/CVF Winter Conference on Applications of Computer Vision, pp. 615-625,70 2021.
Yann LeCun, Sumit Chopra, Raia Hadsell, Marc" Aurelio Ranzato, and Fu Jie Huang. Energy-based models. In Predicting structured data. MIT, 2006.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In IEEE International Conference on Computer Vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI Conference on Artificial Intelligence, volume 32, 201 8a.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9641-9650 2020.
11

---

Published as a conference paper at ICLR 2023
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In AAAI Conference on Artificial Intelligence, volume 32, 2018b.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039. PMLR, 2020.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-o--distribution detection. In Advances in Neural Information Processing Systems, volume 33, pp. 21464-22111 21475,, 2020.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi, Ti++: When does self-supervised test-time training fail or thrive? In Advances Neural Information Processing Systems, volume 34, 2021.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching, In International Conference on Machine Learning, pp. 7313-7324. PMLR, 2021.
Saeid Motiann, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In IEEE International Conference Computer Vision, pp. 5715-5725, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate O (1/k*) 2). Doklady Akademüi Nauk SSSR, volume 269, pp. 543-547, 1983.
A Tuan Nguyen, Toan Tran, Yarin Gal, and Atilim Gunes Baydin. Domain invariant representation learning with domain density transformations. In Advances in Neural Information Processing Systems, volume 34, 2021.
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. In Advances in Neural Information Processing Systems, volume 34, 2021.
Erik Nijkamp. Mitch Hill, Song-Chun Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. In Advances in Neural Information Processing Systems, volume 32, 2019.
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In AAAI Conference on Artificial Intelligence, volume 34, pp. 5272-5280, 2020.
Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh AP. Generalization on unseen domains via inference-time label-preserving target projections. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12924 12933, 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Trung Phung, Trung Le, Tung-Long Vuong, Toan Tran, Anh Tran, Hung Bui, and Dinh Phung. On learning domain invariant representations for transfer learning with multiple sources. In Advances in Neural Information Processing Systems, volume 34, 2021.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In International Conference on Machine Learning, pp. 7728- 7738. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2023
Fengchun Qiao, Long Zhao, and Xi Peng. Learning learn single domain generalization. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 12556-12565. 2020.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In European Conference on Computer Vision, pp. 53 69, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert. distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.01100 2019.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In European Conference on Computer Vision, pp. 68-83, 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross- gooss-rradiett training. In International Conference on Learning Representations, 2018. 8.
Jiayi Shen, Zehao Xiao, Xiantong Zhen, Cees GM Snoek, and Marcel Worring, Association graph learning for multi-task classification with category shifts. In Advances Neural Information Processing Systems, volume 35, 2022.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference on Learning Representations, 2022.
Yang Song and Diederik P Kingma. How train your energy-based models. arXiv preprint arXiv:2101.03288, 2021.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. International Conference on Machine Learning, pp. 9229-9244 PMLR, 2020.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In JEEE Conference on Computer Vision and Pattern Recognition, pp. --017,, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems, volume 31, pp. 5339-5349, 2018.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
Ze Wang, Yipin Zhou, Rui Wang, Tsung- Yu Lin, Ashish Shah, and Ser-Nam Lim. Few-shot fast-adaptive anomaly detection. In Advances in Neural Information Processing Systems, 2022.
Ze Wang, Jiang Wang, Zicheng Liu, and Qiang Qiu. Energy-inspired self-supervised pretraining for vision models. arXiv preprint arXiv:2302.01 384, 2023.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning, pp- 681-688, 2011.
Dustin Wright and Isabelle Augenstein. Transformer based multi-source domain adaptation. arXiv preprint arXiv:2009.07888 2020.
Zehao Xiao, Jiayi Shen, Xiantong Zhen, Ling Shao, and Cees G M Snoek. A bit more bayesian: comain-invariant learning with uncertainty. In International Conference on Machine Learning. PMLR, 2021a.
13

---

Published as a conference paper at ICLR 2023
Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees G M Snoek. Learning generalize across domains on single test samples. In International Conference on Learning Representations, 2022.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2021b.
Binhui Xie, Longhui Yuan, Shuang Li, Chi Harold Liu, Xinjing Cheng, and Guoren Wang. Active learning for domain adaptation: An energy- -based approach. arXiv preprint arXiv:21 2021.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 6635-2644 PMLR, 2016.
Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li. A tale of two flows: cooperative learning of langevin flow and normalizing flow toward energy-based model. In International Conference on Learning Representations, 2022.
Shiqi Yang, Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighbor- hood structure for source-free domain adaptation. In Advances Neural Information Processing Systems, volume 34, pp. 29393-29405, 2021.
Huaxiu Yao, Yu Wang. Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. arXiv preprint arXiv:2201.0023 000999, 2022.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.
Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems, volume 34, 2021.
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generalization via entropy regularization. In Advances in Neural Information Processing Systems, volume 33, 2020.
Aurick Zhou and Sergey Levine. Training on test data with bayesian adaptation for covariate shift. In Advances in Neural Information Processing Systems, 2021.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. European Conference on Computer Vision, pp. 561-578, 2020a.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2020b.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint arXiv:2 10t::2/03.000 023502003, 2021.
Han Zou, Jianfei Yang, and Xiaojian Wu. Unsupervised energy-based adversarial domain adaptation for cross-domain text classification. In Findings of the Association for Computational Linguistics, pp. 1218-1218, 2021.
Arkaitz Zubiaga, Maria Liakata, Rob Procter, Geraldine Wong Sak Hoi, and Peter Tolmie. Analysing how people orient and spread rumours in social media by looking conversational threads. PloS one, 1 133)xc0150998, 2016.
14

---

Published as a conference paper at ICLR 2023
A DERIVATIONS
Derivation of energy-based sample adaptation. Recall our discriminative energy-based model
po.0 = Po(y exp(-Ee(xx
(12)
Zo.
where Zu.s f |x
(xxxxxx(=(xx)dddxxe is the partition function. & and 0 denote the parameters of the classifier and energy function, respectively. To jointly train the parameters, mínimize the contrastive divergence proposed by Hinton (2002):
C DKL [Pa(x, y) 3)llpo,o(xx))) DKL y) pp (x)
(13)
where Pa(x. y) denotes the real data distribution and qo (x, y) = IT y)) denotes t sequential MCMC samplings from the distribution expressed by the energy-based model (Du et al., 2021a). The gradient of the first term with respect to 0 and Φ
= Ve,c.Bp. llog pa(x,y)
V (x..Dekkllxxxx)ll
Pa. y)
(14)
Bpa(x.y) Vv,olog Pa y) Va,logg (x,
Bpe(xcy) Volloog (x)
while the gradient of the second term is
Vo.oDkulgo.dl (x, Ilpo. y)]
= -V (x,)) loog (x.o(x.)) y) (x,
(15)
-Vo,oan xx, y) 20.0 D (x, p)l,o.,oxxxxx3333333 + Bye (x.y) Vo,olog (x.
Combining eq. (14) and eq. (15), we have the overall gradient as:
VoloCau = (Epa(x.x) Vololog (x, y)] E (g Vo,olog (x. V V0.090.0 y V DKL[q0.d (x, ( (x,,
(16)
For the first terms, the gradient can be further derived to
(xx)) V e logloo Pe, (x, (x..) V logloog pe, (x,
=E px
Ve (log P xx) - Ex(x) log [o,o)]
(17)
-E Ego y) V Vo, (log Pe (y|x) - Ex(x) - log Zo Zo,a)]
Moreover, Ve, Vlgg Ze can be written as the expectation Eps, p V olog Po(y|x) - (eEx(x)) (Song Kingma, 2021; Xiao et al., 2021b), which is therefore canceled out in eq. (17) (Hinton, 2002). We then have the loss function for the first two terms as
(V
L Epa(x>) Ex - log Po (ylx)) - Ego. | log (o())
(18)
Furthermore, we have the loss function
(x.
L2 E (x..) log
Potop grod(( (x,
(19)
& (x.y)
[logsstp SogPsttppgrad((
grad( #) (ylx) grodp log 7 stop (r
+ +E (x..)) logge &s yx.
which has the same gradient as the last term eq. (16) (Du et al., 2021a). The stop_grad here means that we do not backpropagate the gradients update the parameters by the corresponding forward functions. Thus, these parameters can be treated as constants.
Since the gradient of 0 and ø is stopped in log Zstap_ (stop_grad(0,,)) of go (x, y) and therefore remove it from the eq. (19). In addition, the term Ego. [log p [x] in eq. (18) encourages wrong prediction of the updated samples from go.s (x, y), which goes against
treat it constant independent
15

---

Published as a conference paper at ICLR 2023
goal of promoting classification by adapting target samples. The term (x.,) [logge, y)) in eq. (19) can be treated as a negative entropy of g estimate. Therefore, we remove these two terms in the final loss function by applying an upper bound of the combination of eq. (18) and eq. (19) as:
(x. y). which is always negative and hard to
C (x.(x.)) [lo (y [ (xp())))) [x)()] Bstop_ ((x,))) [x)(
(20)
+ Eg (x Estop_ (x) - log Pstop_ (y/x)]
Energy-based sample adaptation with categorical latent variable. To keep the categori- cal information during sample adaptation, we introduce categorical latent variable z into our discriminative energy-based model, which is defined as (x,) = (x. z)dz = x) exp(=Bx(((x)) dz. We optimize the parameters 0 and Φ also by the contrastive o
Ze.o
(x. D pe (x, y)l, which has similar gradient as eq. (14) and eq. (15). The latent variable z estimated by variational inference, leading to lower bound of log pa. (x, y) 2 E P ( /z, x) (x(z) - log Zo.o] DKI. La (x dx) (dx)]lee ll [x]]
divergence K [x, (onxx))lleen o - DKL (
We obtain the final loss function the contrastive divergence in similar way as eq. (20) by estimat- ing the gradient and remove the terms that are hard to estimate conflict with our final goal. The final objective function is:
Ly = (x [eg log P (y 22 x)] + Das (xll(2((z/d2)) Ip (nd (z) (2/x)] + (x
Satoo_grad(qu Ee (x, z)] + B gox) Ensop (x)
(x, 2)
log Pstop (vad() - DKI. (etto grad()) (z (ra (2/x))
(21)
B ALGORITHM
We provide the detailed training and test algorithm of our energy-based sample adaptation in Algo- rithm
C DATASETS AND IMPLEMENTATION DETAILS
Model. To be efficient, we train shared backbone for all source domains while a domain-specine classifier and a neural-networr--base
energy function for each source domain. The feature extractor backbone is a basic Residual Network without the final fully connected layer (classifier). Both the prior distribution Ps (z) x) and posterior distribution go dxx) of the latent variable z are generated by a neural network & that consists four fully connected layers with ReLU activation, which outputs the mean and variance of the distribution. The last layer of ó outputs both the mean and standard derivation of the distribution P& (z)x) x)) and 4o (z) dx) for further Monte Carlo sampling. The dimension of is the same the feature representations x, e.g., 512 for ResNet-18 and 2048 for ResNet-50. dx is obtained by the center features of the batch of samples that have the same categories as the current sample in each iteration.
Deployed on feature representations, the energy function consists of three fully connected layers with two dropout layers. The latent variable is incorporated into the energy function by concatenating with the feature representation x. The input dimension is doubled of the output feature of the backbone, i.e., 1024 for ResNet-18 18 and 4096 for ResNet-50. We use the swish function as activation in the energy functions (Du et al., 2021a). The final output of the EBM is scalar, which is processed by a sigmoid function following the stability during training. During training, we introduce a replay buffer B to store the past updated samples from the modeled distribution (Du & Mordatch, 2019). By sampling from B with 50% probability, we can initialize the negative samples with either the sample features from other source domains or the past Langevin dynamics procedure. This can increase the number of sampling steps and the sample diversity.
al. (2021a) to bound the energy to the region [0, 1] and improve
Training details and hyperparameters. We evaluate on PACS with both a ResNet- 18 and ResNet-50 pretrained on ImageNet. We use Adam optimization and train for 10,000 iterations with batch size
16

---

Published as a conference paper at ICLR 2023
Algorithm 1 Energy-based sample adaptation
TRAINING TIME
Require: Source domains D.= [D: each with joint distribution Pa; (1,y) y) of input image and label.
Require: Learning μe; iteration numbers M; step numbers K and step size of the energy function. Initialize pretrained backbone vh; &;, for ieer in M do
for each source domain D
for D; in D. do Sample datapoints {(I', Paj (I, ((P, ~ (v (I,y))ist or B with 50% probability.
Feature representations x' = fo( = fT(
for k in K do
X t- x
Vx (x - (z') w. x² (z'1d)) ~ ~(0,o)
end for
To
(x) x/
pa(x) Pay (x').
(v) 00) + (vb x*) - AV vo. (pa(x)) 0² + 0² (x))))
B' B¹ x
end for
end for
TEST TIME
Require: Target images I, from the target domain; trained backbone yh; and domain-specific model oi 0 for each source domain in {Di]
Input feature representations 2-11 for
for [L.....5) S} do
do Nt.b +- Nt. Vx (xc,-117)) + w, (o(o(()))))),, w~ w~1(00,))) end for yi Pai (yr 221 end return y: = 1/2 Ee
Table 3: Implementation details our method per dataset and backbone.
Dataset
Backbone
Backbone learning rate
Step size
Number of steps
ResNet-18
0.00005
50
20
PACS
ResNet-50
0.00001
50
20
ResNet-18
0,00001
100
20
Office-Home
ResNet-50
0.00001
100
20
Rotated MNIST
ResNet-18
0.00005
50
20
Fashion-MNIST
ResNet-18
0.00005
20
РHEME
DistiBBERT
0.00003
40
20
of 128. We set the learning rate 0.00005 for ResNet-18, 0.00001 for ResNet-50, and 0.0001 for the energy-based model and classification model. We use 20 steps of Langevin dynamics sampling to adapt the target samples source distributions, with step size of 50. We the number of Monte Carlo sampling N in eq. (11) as 10 for PACS. Most of the experimental settings on Office-Home are the same as on PACS. The learning rate of the backbone is set to 0.00001 for both ResNet-18 and ResNet-50. The number of Monte Carlo sampling is 5. For fair comparison, we evaluate the rotated MNIST and Fashion-MNIST with ResNet-18, following (Piratla et al., 2020). The other settings are also the same as PACS. On PHEME we conduct the experiments based on pretrained DistilBERT. We set the learning rate as 0.00003 and use 20 steps of Langevin dynamics with a step size of 20.
We train all models on an NVIDIA Tesla V100 GPU for 10,000 iterations. The learning rates of the backbone are different for different datasets as shown in Table 3. The learning rates of the domain- specific classifiers and energy functions are both set to 0.0001 for all datasets. For each source domain, we randomly select 128 samples as a batch to train the backbone and classification model. We also select 128 samples from the other source domains together with the current domain samples to train the -ppaif- energy function. We use a replay buffer with 500 feature representations and apply spectral normalization on all weights of the energy function (Du & Mordatch, 2019). We
17

---

Published as a conference paper at ICLR 2023
Source domalin; photo
Source domain: cartoon
Source domalin: sketch
x
samples Torget samples
Target samples after adaptation
-00
Figure 5: Benefit of energy-based test sample adaptation. Different shapes denote different classes. From left to right: adaptation to the source domains photo, cartoon, and sketch of samples from the target domain art-puinting. After adaptation, the target samples data than before
are more close to the source ), demonstrating the effectiveness of our method. Best viewed in color.
use random noise with standard deviation X = 0.001 and clip the gradients to have individual value magnitude of less than 0.01 similar to (Du & Mordatch, 2019). The step size and number steps for Langevin dynamics different for different datasets as shown in Table 3.
D VISUALIZATIONS
More visualizations of the adaptation procedure. To further show the effectiveness of the iterative adaptation of target samples, we provide more visualizations on PACS. Figure 5 visualizes the source domain features and the target domain features both before and after the adaptation to each individual source domain. Figure visualizes more iterative adaptation procedure of the target samples. Subfigures in different rows show the adaptation of samples from different target domains to source domains. Similar with the visualizations the main paper, the target samples gradually approach the source data distributions during the iterative adaptation. Therefore, the predictions of the target samples on the source domain classifier become more accurate after adaptation.
Failure cases. We also provide some failure cases on PACS in Figure 7 to gain more insights in our method. Our method is confused with samples that have objects of different categories (first row) and multiple objects or complex background (last three rows). A possible reason is that there is noisy information contained in the latent variable of these samples, leading to adaptation without a clear direction, which behaves wrong adaptation directions, e.g., visualization in row I column 4, or unstable updates with fluctuations in small regions, e.g., visualizations in row 2 column 3 and row 3 column 4. Obtaining the latent variable with more accurate and clear categorical information can be one solution for these failure cases. We can also solve the problem by achieving more stable adaptations with optimization techniques like Nesterov momentum (Nesterov, 1983). Moreover, although failing in these cases, the adaptation of the target sample to some source domains still improves the performance, e.g., the adaptation of the photo sample (row column 2) and cartoon sample (row 3 column 3) to the art-painting domain and the adaptation of the sketch sample (row 4 column 4) to the cartoon domain, which further demonstrate the effectiveness of our iterative sample adaptation through the energy- based model. The results motivate another solution for these failure cases, which is to learn to select the best source domain, or top-n source domains for adaptation and prediction of each target sample. We leave these explorations for future work.
E MORE EXPERIMENTAL RESULTS
Analyses and discussions of the categorical latent variables. In the proposed method, the categorical latent representation for the test sample will have high fidelity to the correct class. This is guaranteed by the training procedure of our method. As shown in the training objective function (eq. 10), we minimize the KL divergence to encourage the prior (z(z)x) x) to close to the variational posterior 4o (x). dy is essentially the class prototype containing the categorical information. By doing so, we train the inference model po (z|x) to learn to extract categorical information from single sample. Moreover,
also supervise the sample adaptation procedures by the predicted log- likelihood of the adapted samples (the last term in eq. (10)). The supervision is inherent in the objective function of our discriminative energy- based model as in the derivation of eq. (7) and eq.
18

---

Published as a conference paper at ICLR 2023
Source: art pointing
Sourte: cortson
phott Categony
Category: phogle
Category. gutter
-n
Probability dog: 28.3%
Probability on giraffe: 24.5%
Probability on guitar:
87.0%
89.3%
17.9%
100%
15.
75
Source: photo
Source: cortoon
Sourte slatch
Targel: pointing
Tanget: ort-pinning
Teege:
Category:
Category
Category: house
-10
Probability on horse:
Probability giraffe: 2.6%
Probability house:
26.5%
17.
25
Source: artypolnting
Soutte photo
Soue shatch
Target: contoon
Torget: Categury: home
Torto:
Category: elephant
Catugory:
eito -123 -ia -00
Probebility on elephont:
Probability on horse: 6.5%
Probebiity on person: 12.2%
33.6%
67.8%
77.8%
92.6%
Smuran photo
Source: arhpainting
sarioon
Target sletci
Torget: skedch
Torget: stetch
Catagors: dog
Catagory: hone
Category: gultar
Probability dog:
Probabiltty
Probability gultor.
8.7%
100%
33.2%
90.6%
37.7%
84.9%
Figure 6: More visualizations of the iterative adaptation on PACS. We visualize the adaptation of samples from different target domains to source domains on PACS. Each subfigure shows the adaptation of one target sample to one source domain. The adaptation procedure of the target samples is represented by the gradient color from green to red. In each figure, the target sample has the same label as the source data, but is mispredicted due domain shifts. During adaptation, the target samples gradually approach the source distributions and eventually get correct predictions.
(10). Due to this supervision, the model is trained to learn to adapt -attt--diitrittiinn samples the source distribution while being able to maintain the correct categorical information conditioned Although trained only on source domains, the ability can generalized to the target domain since it is trained by mimicking different domain shifts during training. To further show that captures the categorical information in x, we visualized the features X, and latent variables z; of the target samples in Figure 8, which shows that z, actually captures the categorical information. Moreover, z, more discriminative than X, as shown in the figure. Although z, is approximated by only x, during inference.
19

---

Published as a conference paper at ICLR 2023
ort-painting
cortoon
Source:
Label: guitar Prediction: person
Probabiltyy on guitor: 8.3% 57.5%
Probability an guitor:
Probability on guitor:
5.4%
24.5%
4.7%
14.1%
cartoon
Source: sketch -2a
45
dá
14
da
25
Label: dog Prediction: horse
Probability on dag: 0.01%
Probabliity on dog:
Probability dog:
10.3%
0.01%
0.01%
0.01%
7.5%
Source: photo
Source: art-painting
Source: sketch
400
Label: person Prediction: dog
Probability on person: 3.7%
Probability on person:
Probability on person:
3.6%
3.1%
71.4%
0.8%
0.1%
art-painting
Label: person Prediction: guitar
Probability on person:
Probability an person:
Probability an person:
5.8%
10.3%
12.3%
21.8%
Figure 7: Failure case visualizations of method PACS. The visualization settings the same as Figure 6. Our method makes wrong predictions on samples with complex background or multiple objects. However, our method still achieves good adaptation of these target samples some source domains, which shows its effectiveness.
Moreover, the categorical latent variable benefits the correctness the model in the case that the target samples adapted to previously unexplored regions with very large numbers of steps. Our optimization objective is to minimize the energy to adapt the sample, therefore it is possible that the energy the target samples is lower than the source data after very large numbers of steps. this case, the adapted samples could arrive in previously unexplored regions due to the limit of source data. This can further be demonstrated in Figure 4, where the performance of the adapted samples drops after large numbers of steps, reaching a low energy value. Additionally, in the unexplored regions, the classifier could not be well trained, which might also be reason for causing the performance drop. This is also one reason that we set the number of steps categorical latent variable benefits the correctness of the model in such cases also can found in Figure 4. The oracle model shows almost performance degradation even with small energy values after adaptation. The model with the latent variable p(z]x) is also more robust to the step numbers and energy values than the model without z. These results show the role of the latent variable in preserving the categorical information during adaptation and somewhat correcting prediction after adaptation.
small value, e.g., 20 and 50. The
With the categorical latent variable z, is natural to make the final prediction directly by (y(y)z) without the sample adaptation procedure. However, here would like clarify that it is sub-optimal.
20

---

Published as a conference paper at ICLR 2023
20
-10-
-10
-20-
Visualizations of
Visualizations of
Figure 8: Visualizations of the target features x; and categorical latent variables z₁. We use art prt-painting on PACS as the target domain. Different colors denote different categories. is obtained by Po (z Most data points z, are clustered according to their labels, demonstrating that z, can capture the categorical information.
Table 4: Analyses the categorical latent variable. As expected, prediction the adapted target samples x performs better than prediction directly on the categorical latent variable z.
Overall comparisons on PACS and Office-Home
PACS
Office-Home
ResNet-18
ResNet-50
ResNet-18
ResNet-50
predict directly on
82.46 0.34
85.95
10.33
64.49
10.25
70.60
10.53
predict on adapted target samples x
84.92
0.59
87.70
10.28
66.31
10.21
72.07 10.38
(b) Detailed comparisons on PACS.
Photo
Art-painting
Cartoon
Sketch
Mean
Predict directly on z
No adaptation
94,22 10.25
79.52 40.2
80.46
2043
75.63 2068
82,46 20.34
Predict with model adaptation
(Tent)
Adaptation with I sample per step
80.49
14 40.38
$1.49
30.28
51.60
Adaptation with samples per step
93.65 10.33 80,20 4024
76.90
#052
68.49
10.72
79.81
#0.31
Adaptation with 64 samples per step
96.04 10.33 81.91 +0.37
80.81
1064
76.33
2065
83.77
20.41
Adaptation with 128 samples per step
97.25 10.44 84.91 40.3
81.12
20.7
76.80
2083
85.02
=0.49
Predict on adapted target samples with our method Adaptation with I sample
96.05 82.28
81.55
79.81
84.92
The latent variable is dedicated to preserving the categorical information in x during adaptation. It still contains the domain information of the target samples. Therefore, is not optimal to directly make predictions on the latent variable z due the domain shifts between the z and the source- trained classifiers. By contrast, the proposed method moves the target features close to the source distributions address domain shifts while preserving the categorical information during adaptation. To show how it works by direct prediction on z, we provide the experimental results of making predictions only from Table 4. As expected is worse than predictions on the adapted target features x, demonstrating the analysis we provided above.
To show the advantages of our method, we also combine the prediction of the latent variable z with model adaptation methods. We use the online adaptation proposed by Wang et al. (2021), where all target samples are utilized to adapt the source-trained models in an online manner. The model keeps updating step by step. In each step, the model is adapted to one batch of target samples. As shown in Table 4b, with large numbers of target samples per step, e.g., 128, the adaptation with Tent is competitive. However, when the number of samples for online adaptation small. e.g., 1 and 16, the performance of the adapted model even drops, especially for single sample adaptation. By contrast, method adapts each target sample to the source distribution. All target samples are adapted and
21

---

Published as a conference paper at ICLR 2023
Table Training cost of the proposed method. Compared with ERM, our method has about 20% more parameters, most of which come from the energy functions of source domains. Similar to test time, the time cost of training increases along with the number of steps of the energy-based model.
Parameters
Adaptation steps
10000 iterations training time
ERM
11.18M
6.2 h
20
7.9
40
9,4
This paper 13.73M
60
10.6 h
80
12. h
100
14.0 h
Table 6: Comparisons on PACS. Our method achieves best mean accuracy with ResNet- 18
backbone and is competitive with ResNet-50.
Backhone
Method
Photo
Art-painting
Cartoon
Sketch
Mean
Dou et al, (2019)
94.99
80.29
77.17
71.69
81 11.04
Iwasawa Matsuo (2021)
81 440
Zhao et al. (2020)
96.65
80.70
76.40
71.77
81,46
ResNet-18
Wang ct al. (2021)
95.49
81.55
77.67
77.64
83.09
Zhou et al. (2020b)
96. 10
84.10
78.80
75,90
83.70
Xiao et al. (2022)
95.87
82.02
79.73
78.96
84.15
This paper
96.05 =0.37
82.28 +0.31
81.55 +0.6
79.81 #0.41
84.92 +0.59
Dou al. (2019)
95.01
82.89
80.49
72.29
82.67
Dubey et al. (2021)
84,50
Iwasawa & Matsuo (2021)
85.10
Zhao et al. (2020)
98.25
87.51
79.31
76.30
85,34
ResNet-50
Gulrajani & Lopez-Paz (2020)
97.20
84.70
80.80
79.30
85.50
Wang et al. (2021)
97.96
86.30
82.53
78.11
86.23
Seo et al. (2020)
95.99
87.04
80.62
82.90
86.64
Xiao et al. (2022)
97.88
88.09
83.83
80.21
87.51
This paper
97.67 10.14
88.00 29
84.75
80.44 =:0388
87.70 20.28
predicted equally and individually. The overall performance of our method is comparable to Tent with 128 samples per adaptation step.
Time cost with different adaptation steps. As the number of steps increases, both the training and test time cost consistently increases for all target domains. Without adaptation, the test time cost for one test batch is about 0.05 second. The 20-step adaptation will take about 0.1 extra second. This number will increase to 0.25 second with 50 steps. The test time increases by more than 0.5 seconds for 100 iterations, which is ten times that without adaptation and might limit the application of the proposal. The training time cost more than two times that for ERM for 100 iterations as shown in Table 5. Since the extra time cost mainly caused by the iterative adaptation, potential solutions can be speeding up the Langevin dynamics with some optimization techniques like Nesterov momentum (Nesterov, 1983). or exploring some one-step methods for the target sample adaptation. In other experiments on PACS in the paper we use 20 steps for all target domains considering both the overall performance and the time cost.
Detailed comparisons. We provide the detailed performance of each target domain on PACS (Table 6), Office-Home (Table 7), and PHEME (Table 8). On PACS, our method achieves competitive results on each target domain and the best overall performance with both ResNet- 18 and ResNet-50 as the backbone. Moreover, our method performs better than most of the recent model adaptation methods (Wang et al., 2021; Dubey et al., 2021; : Iwasawa & Matsuo, 2021; Xiao et al., 2022). The conclusion on Office-Home and PHEME similar to that on PACS. We achieve competitive and even better performance each target domain.
Results on single source domain generalization. To show the ability of our method of handling corruption distribution shifts and single source domain generalization, we conduct some experiments on CIFAR- 10-C and ImageNet-C. We train the model on original data and evaluate it on the data with 15 types of corruption.
22

---

Published as a conference paper at ICLR 2023
Table Comparisons on Office-Home. Our method achieves the best mean accuracy using both ResNet- 18 and ResNet-50 backbone.
Backbone
Method
Art
Clipart
Product
Real World
Mean
Iwasawa & Matsuo (2021)
47.00
46.80
68.00
66. 10
57.00
Wang al. (2021)
56,45
52.06
73.19
74.82
64.13 13
ResNet-18
Xiao al. (2022))
59.39
53,94
74.68
76.07
66.02
This paper
60.08 1:033
53,93 1:.34
74.50 00339
76.74 1:0.4
66.31 10.21
Guirajani & Lopez-Paz (2020)
61.30
52.40
75.80
76,60
66.50
Wang al. (2021)
62.12
56.65
75.61
77.58
67. 97.9
ResNet-50
Dubey al. (2021)
68.90
Xiao al. (2022)
67.21
57.97
78.61
80.47
71.07
This paper
69.33
1:.34
58.37
1:030
79.29
=0.32
81.26
1:026
72.07 110.38
Table Generalization beyond image data. Rumour detection on the PHEME microblog dataset. Our method achieves the best overall performance and is competitive in each domain.
Charlie Hebdo
Ferguson
German Wings
Ottawa Shooting
Sydney Siege
Mean
ERM Buseline
79.4 2025
77.1 #0.36
75.7 +0.12
68.2 #0.48
75.0 #0.25
75.1 #0.29
Wang et al. (2021)
80.1 10.18
76.9 10:56
74.7 10.52
72.0 10.48
75.4 10:3
75.8 10.23
Xiao et al. (2022)
81.0 2052
77.2 10.25
75.7
10.31
70.0 i1.46
76,5 1n:22
76.1 10.21
This paper
81.8 2043
77.8 10.37
75.3 10.14
71.9 10.28
75.6 10.15
76.5 10.18
Table 9: Experiments on single-source domain generalization, The model trained on original
data and evaluated 15 different types of corruption. Our method is competitive with Sun al. (2020), Rusak et al. (2020) and Hendrycks et al. (2020), and is outperformed by Wang et al. (2021). Mimicking good domain shifts during training is important for our method.
Method
CIFAR-I0-C ImageNet-C
Yun al. (2019)
31.1
Guo (2019)
25.8
Hendrycks (2020)
17.4
51.7
Rusak (2020)
50.2
Sun et al. (2020)
17.5
Wang et al. (2021)
14.3
44.0
This paper (noisy data negative samples)
21.5
55.8
This paper (corrupted data as negative samples)
17.0
51.1
Since our method needs to mimic distribution shifts to train the discriminative energy-based model during training, for the single source domain setting, generate the negative samples by adding random noise to the image and features of the clean data. We also use the other corruption types (not contained in the evaluation corruption types) as the negative samples during training, which we regard as "corrupted data as negative samples". Note that these corrupted data are only used as negative samples to train the mergy--basdd model. As shown in Table 9, by mimicking better domain shifts during training, our method achieves competitive results with Sun et al. (2020). We also compare our method with some data-augmeenatiion-baeee based methods (e.g.. Mixup (Guo al., 2019), CutMix (Yun et al., 2019) and AugMix (Hendrycks et al., 2020)), our sample adaptation is also competitive with these methods. The proposed method performs worse with single source domain, although we generate extra negative samples to mimic the domain shifts. The reason can be that the randomly generated domain shifts do not well simulate the domain shift at test time.
Analyses for ensemble prediction. We conduct several experiments on PACS to analyze the ensemble inference our method. We first provide the results of each source-domain-speccin classifier before and after sample adaptation. As shown in Table 10, Although the performance before and after adaptation to different source domains are different due to domain shifts, the proposed sample adaptation to most of the source domains performs better. Moreover, the ensemble inference further improves the overall performance of both without and with sample adaptation, where the results with sample adaptation
still better, expected.
We also try different aggregation methods to make the final predictions. The results are provided in Table 11. The best results in Table 10 are comparable, but it is difficult find the best source
23

---

Published as a conference paper at ICLR 2023
Table 10: Sample adaptation to each source domain on PACS. The experiments conducted on ResNet-18. Due to the domain shifts between the target domain and different source domains, the performance before and after adaptation are different. The results with sample adaptation to most of the source domains are better than those without adaptation. The ensemble inference further improves the overall performance, where the results with sample adaptation are still better than that without adaptation.
Photo
(b) Art-painting
Art art-paintin
Cartoon
Sketch
Ensembll
Photo
Cartoon
Sketch
Ensemblle
w/o aduptation
1ass
95.03 94.69 1421
Live
wlo adupiaaiio
78.52 10.45
79.68 Loch
79.83 Lasa 79.79 1est
w/ udapiation
95.81 #327
95.99 =144s 96.05 4937
w/ adaptation
81.49
82.19 40.s
80.81
82.28
(c) Cartoon
(d) Sketch
Photo
Art-painting
Sketch
Ensemble
Photo
Art-painting
Cartoon
Ensemble
wlo aduptation
79.05 saur
78.93 2u41
78, cus 79.15 2037
wio udaptation
78.32 au3e
76.98 2023
76.16 4002 79.28 cwa
adapiation
81.09 80.44
80.32
81.55
aduptation
79.72 79.69
79.77
79.81
Table 11: Analyses of different aggregation methods for the predictions. The experiments are conducted PACS using ResNet-18. The results with different aggregation methods are similar while ensemble inference performs slightly better.
Aggregation methods
Photo
Art Art-painting
Cartoon
Sketch
Moon
Adaptation to the closest source domain
95.41 #0.28
79.86 =0.1
79.67 #044 78,97 =0.2
83.48 #0.43
Weighted average adaptation to different source domains
95.93 20.33
82.18 20.37
81.24 #a52
79.54 =0m
84.76 10.55
Most confident prediction after adaptation
95.77
81.93
#0.11
80.67
79.25
=0.6
84.41
Ensemble (This puper)
96.05 20.37
82.28
20.31
81.55 4008
79.81
20.41
84.92 10.59
domain for adaptation before obtaining the results. We tried to find the closest source domain of each target sample by the cosine similarity of feature representations and the predicted confidence. We also tried to aggregate the predictions by weighted average according the Cosine similarity. With cosine similarity, the weighted averaged results are comparable to the common ensemble method we used in the paper, but the results of adaptation to the closest source domain are not so good. The reason can be that the cosine measure is not able to estimate the domain relationships, showing that it is difficult to reliably estimate the relationships between source and single target samples. The results obtained by using the most confident adaptation are also not good as the ensemble method, although comparable. The reason can be that ensemble methods introduce uncertainty into the predictions, which is more robust.
Benefit for larger domain gaps. To show the benefit of our proposal for domain generalization scenarios with large gaps, we conduct experiments on rotated MNIST and Fashion MNIST. The results are shown in Figure 9. The models are trained on source domains with rotation angles from 15t to 75% 30 60%, and 30 45,, and always tested on target domains with angles of 0' and 90%. As the number of domains seen during training decreases the domain gap between source and target increases, and the performance gaps between our method and others becomes more pronounced. Notably, when comparing our method with the recent test-time adaptation of Xiao et al. (2022), which adapts model to each target sample, shows adapting target samples better handles larger domain gaps than adapting the model.
24

---

Published as a conference paper at ICLR 2023
Rotated MNIST
Rotated Fashion-MNIST
80
ERM baseline
ICLR
70
Accuracy
60
80
50
40
30
607,
30", Source domains
30", 45", 60",
Source domains
Figure 9: Benefit for larger domain gaps. We train different source set distributions and evaluate on target sets with rotation angles of 0' and 90" As the domain gap between source and target sets increases, our method performs better than alternatives.
25


Reviews:
[{"review_id": "gb86AmAwVt", "reviewer": "Reviewer_yzZP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a test-time sampling based domain generalization approach based on a discriminative energy-based model formulation. The model performs modification of test-time inputs by sampling towards the source data distribution via Langevin dynamics, while simultaneously preserving label information (by augmenting their model with a categorical latent variable) to maintain the discriminative components of the adapted sample. Experiments are performed on both image and text dataset benchmarks on multi-source domain generalization, demonstrating the effectiveness of all components of their model with several ablation studies.", "summary_of_the_review": "Generally I find this paper to touch an interesting problem with a novel generative-discriminative modeling based approach. Yet, there are some unclear points that I would like to clarify and I summarize them below:\n- The model learns an unbounded energy function. It is not clarified though, at test time how does the method ensure a lower energy bound on this sampling process? To illustrate: a given test target domain sample can(?) in fact be evaluated to have a much lower energy value than observed already at the beginning of the sampling process. This would then \"adapt\" the sample (for a fixed number of SGLD steps) towards previously unexplored regions by the source domain data? How can the model correct for this and give confident decisions in such cases?\n- A somewhat implicit assumption of the model is that the categorical latent representation for the test sample will have high fidelity to the correct class. Can the authors discuss how reasonable is this? In case the target sample is represented strongly within an incorrect source domain class (through the only-source-trained backbone), wouldn't this divert the SGLD sampling process off-track, towards the wrong class?\n- Is there a deeper analysis of the ensemble classification approach? How robust does this make the model, in comparison to e.g. adapting the target sample only to the closest source domain? Similarly one could also compare the ensemble approach to decision making with respect to the most confident source domain mapping.\n- Experiments show that the proposed method can only work on learning from multiple source domains. Interesting observations are presented in the supplementary (Table 9) in that sense, where the model more or less fails on domain generalization from CIFAR-10 to CIFAR-10-C with hand-crafted sources. How is it on ImageNet-C, did the authors try? Also there many simple data augmentation methods that can achieve very high domain generalization of a vanilla classifier to CIFAR-10-C and ImageNet-C, which one should include in Table 9.\n- How sensitive are the final accuracies in Table 2 to the # sampling steps? Can the authors provide some additional rows to Table 2 with 10-steps and 50-steps for instance?\n- What was the exact feature dimensionality for the EBM modeling network per dataset? How did the authors define the feature extractor backbone? (e.g., is it a post-ReLU activation before the final classifier layer or so?)\n- Minor comment: The first sentence of the second paragraph in Sec 2.1 is unfinished.", "strength_and_weaknesses": "Strengths: 1) The paper is nicely written and the methodology is well described (considering also the clarifications in the Supplementary). 2) Proposed approach is novel and it shows compatibility to different data structures due to its capability to operate in a latent feature space.\n\nWeaknesses: 1) There are several fundamental assumptions implied in the methodology with regards to the extent of the source and target domains, which should have been clarified/discussed. 2) The proposed model is rather complex in its optimization objective since it involves many parametric components put together, and this may somewhat limit its applicability to a broad range of domain generalization problems (e.g., single-domain generalization). 3) Empirical evaluations show minimal performance gains with the proposed approach with respect to the state-of-the-art.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YeViYYCAaGu", "reviewer": "Reviewer_aESp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a domain generalization method with a new discriminative energy-base model. The proposed idea is novel and effective because they adapt the target samples to source distributions instead of generalizing the model to unseen target samples in most previous methods. Theoretically, the authors provide detailed mathematical derivations and proofs. Experimentally, the authors present extensive empirical results and conduct a thorough ablation study.", "summary_of_the_review": "Interesting paper on domain generalization at test time that proposes a discriminative energy-based model to adapt target samples to source distributions, and works well on six image and text benchmarks. I have concerns about its viability as a source-free domain adaptation method but would be willing to reevaluate based on the authors' response.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well written, the motivations for choices in the method are clear and the method is simple yet effective, ripping the benefits of a discriminative energy-based model and Langevin dynamics.\n\n2. The background of the energy-based model is clear and kindful in Section 2.\n\n3. Extensive experiments show that this method performs quite well in comparison to existing DG methods on both image and text benchmarks. Thorough insights are conducted in experiments.\n\n4. Theoretical validation and qualitative results indicate that energy-based test sample adaptation is a good solution to reduce the domain gap at test time.\n\nWeaknesses:\n1. It's a bit unclear why the authors haven't included comparisons or discussions about how this method would perform in source-free DA settings. I expect to see whether this method would work in that context too.\n\n2. During the training phase, the proposed sample adaptation framework simultaneously trains the shared backbone network, the classification model and an extra energy function. In the test phase, the target samples are adapted by Langevin dynamics of the energy function. Does this compare fairly to existing DG methods to only use source data at the training step? Depending on the method but it would be a good discussion to have. As an aside, this concern arises because the proposed framework w/o adaptation has achieved comparable performance (e.g., Table 1 and Table 2).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7ybbh4fkaY", "reviewer": "Reviewer_E7hz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed an energy-based sampling adaptations method for domain generalization. In their method, they adapt the unseen target sample to source-domains at test time. A category latent variable is used to sample update. They show the effectiveness in several classification tasks.", "summary_of_the_review": "Generally, this is a good paper with clear writing. The story is complete. ", "strength_and_weaknesses": "Strength:\n\n1. The energy-based approach of sample adaption for domain adaption is interesting. The background and proposed method (training and inference) are also clear.\n\n2. The proposed method show the effectiveness on several tasks. And the visualization of the adaptation process shows the effectiveness of the method.\n\n3. The authors also point  out the failure case or limitations of the proposed method with complex background. \n\nWeaknesses:\n\n1. Seems hard to work well in complex tasks. And the proposed rely on good feature representation (the shared backbone model). \n\n2. Somewhat unsure whether is fair during inference.  In this paper, the ensemble inference method is used. The adaption can lead the predication more confidently at each domain classifier.  The sum of log p for each domain domain seems reasonable with sample adaption. it is unclear whether without sample adaption is already good enough with voting method.  \n\n3. In Figure 5,  energy and accuracy are shown as the number of step increase. The energy is decreasing. However, there is still some space for energy. What is the sample accuracy with lowest energy?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4ALmgcwLCA", "reviewer": "Reviewer_Y7Jd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the setting of homogeneous domain generalization, and proposes a novel single-sample based adaptation scheme. In this scheme, the discriminative classifier remains constant but the respective test sample gets adapted in feature space based upon an energy-based objective. The main contribution of the paper is to adjust energy-based density modelling such that it can be employed at test-time for domain adaptation. For this, the energy-based modelling is extended to be conditioned on categorical information inferred from the test sample before adaptation such that test sample adaptation  does not loose categorical information.", "summary_of_the_review": "In summary, the paper proposes an interesting approach to a relevant problem. Clarity and reproducibility could be improved with a minor revision. At this point, my main concern is whether energy-based sample adaptation as proposed is actually helpful if it is based upon inferred categorical information (which kind of requires solving the classification problem already _before_ the adaptation). This point requires clarification and some solid empirical support before warranting acceptance of this paper.\n\n### Updated after author feedback ###\nThe authors have addressed my main criticism and have provided preliminary evidence that energy-based test sample adaptation is preferable over standard adaptation in a small batch setting. I am increasing my overall score, assuming that the authors will extend and revise their manuscript accordingly.", "strength_and_weaknesses": "Strengths:\n * Using energy-based density modelling as basis for test-time adaptation and adapting the sample rather than the model are novel ideas (to my knowledge).\n * The presentation of background and method in Section 2 is thorough and complete.\n * The proposed method is well motivated and clearly derived from first principles.\n * Useful illustrations are provided that illustrate essential parts of the problem/method\n * The empirical evaluation is thorough and conducted on diverse benchmarks, including non-image data\n\nWeaknesses:\n* The main question for me is if conditioning on the categorical information $z$ in energy-based density modelling is reasonable. This design choice is motivated by \"In order to maintain the category information of the target samples during adaptation and promote better classification performance, we further introduce a categorical latent variable in our energy-based model.\" (Introduction). However, when used within a discriminative model, wouldn't it be more natural to predict classes directly upon this categorical latent variable rather than (i) first adapting a sample to become more likely under the energy with this categorical variable and then (ii) predict based upon this sample? Directly predicting upon the latent variable would be more computationally efficient and it remains unclear why the two step approach (i) + (ii) should result in better predictive performance? Specifically, when making $p_{\\phi_i} (y\\vert z^n, x)$ unconditional of $x$, that is: $p_{\\phi_i} (y\\vert z^n)$, how would this perform at inference time?\n* The paper somehow (probably unintentional) obfuscate that adaptation is conducted in feature space and not in input space. It is briefly mentioned at the end of Section 2, but is easily missed (I missed it did during the first read). Since this is clearly an essential part of the method's design, it should be presented earlier and clearer. This also applies to Algorithm 1 where both input and its feature representation are denoted by $\\mathbf{x}$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster