Variational Invariant Learning for Bayesian Domain Generalization
Authors: Zehao Xiao
Jiayi Shen
Xiantong Zhen
Ling Shao
Cees G. M. Snoek
Venue:  (2021)
Keywords: domain generalization
variational invariant learning
Bayesian inference

Abstract: Domain generalization addresses the out-of-distribution problem, which is challenging due to the domain shift and the uncertainty caused by the inaccessibility to data from the target domains. In this paper, we propose variational invariant learning, a probabilistic inference framework that jointly models domain invariance and uncertainty. We introduce variational Bayesian approximation into both the feature representation and classifier layers to facilitate invariant learning for better generalization across domains. In the probabilistic modeling framework, we introduce a domain-invariant principle to explore invariance across domains in a unified way. We incorporate the principle into the variational Bayesian layers in neural networks, achieving domain-invariant representations and classifier. We empirically demonstrate the effectiveness of our proposal on four widely used cross-domain visual recognition benchmarks. Ablation studies demonstrate the benefits of our proposal and on all benchmarks our variational invariant learning consistently delivers state-of-the-art performance. 

principle under the Bayesian inference framework, which enables Under review as a conference paper at ICLR 2021
VARIATIONAL INVARIANT LEARNING FOR BAYESIAN DOMAIN GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain generalization addresses the out-of-distribution problem, which is chal- lenging due the domain shift and the uncertainty caused by the inaccessibility to data from the target domains. In this paper. we propose variational invariant learning. probabilistic inference framework that jointly models domain invari- ance and uncertainty. We introduce variational Bayesian approximation into both the feature representation and classifier layers to facilitate invariant learning for better generalization across domains. In the probabilistic modeling framework, principle to explore invariance across domains
we introduce a in unified way. We incorporate the principle into the variational Bayesian lay- ers in neural networks, achieving domain-invante invariant representations and classifier. We empirically demonstrate the effectiveness of proposal on four widely used cross-domain visual recognition benchmarks. Ablation studies demonstrate the benefits of our proposal and on all benchmarks our variational invariant learning consistently delivers state-of-the ofte-of-the-art performance.
INTRODUCTION
Domain generalization (Muandet et al., 2013), as an out-of-distribution problem, aims to train a model on several source domains and have it generalize well unseen target domains. The major challenge stems from the large distribution shift between the source and target domains, which is further complicated by the prediction uncertainty (Malinin & Gales, 2018) introduced by the in- accessibility to data from target domains during training. Previous approaches focus on learning features using novel loss functions (Muandet et al., 2013; Li al., 2018a) or spe- cific architectures (Li et al., 201 7a; D I'Innceette & Caputo, 2018) 8). Meta-learning based methods were proposed to achieve similar goals by leveraging an episodic training strategy (Li et al., 70;7;; Balaji et al., 2018; Du et al., 2020). Most of these methods are based on deep neural network back- bones (Krizhevsky et al., 2012; He et al., 2016). However, while deep neural networks have achieved remarkable success in various vision tasks, their performance is known to degenerate considerably when the test samples are out of the training data distribution (Nguyen et al., 2015; llse et al., 2019), due to their poorly calibrated behavior (Guo et al., 2017; Kristiadi al., 2020).
As an attractive solution, Bayesian learning naturally represents prediction uncertainty (Kris- tiadi al., 2020; MacKay, 1992), possesses better generalizability to out-of-distribution exam- ples (Louizos & Welling, 2017) and provides an elegant formulation to transfer knowledge across different datasets (Nguyen al., 2018). Further, approximate Bayesian inference has been demon- strated to be able to improve prediction uncertainty (Blundell et al., 2015; Louizos & Welling, 2017; Atanov et al., 2019), even when only applied to the last network layer (Kristiadi et al., 2020). These properties make it appealing to introduce Bayesian learning into the challenging and unexplored scenario of domain generalization.
In this paper, we propose variational invariant learning (VIL), a Bayesian inference framework that jointly models domain invariance and uncertainty for domain generalization. We apply variational Bayesian approximation to the last two network layers for both the representations and classifier by placing prior distributions over their weights, which facilitates generalization. We adopt Bayesian neural networks to domain generalization, which enjoys the representational power of deep neural networks while facilitating better generalization. To further improve the robustness to domain shifts, introduce the

---

Under review as a conference paper at ICLR 2021
us explore domain invariance for both feature representations and the classifier in a unified way. We evaluate our method on four widely-used benchmarks for cross-domain visual object classifica- tion. Our ablation studies demonstrate the effectiveness of the variational Bayesian domain-invarian invariant features and classifier for domain generalization. Results further show that our method achieves the best performance on all of the four benchmarks.
2 METHODOLOGY
We explore Bayesian inference for domain generalization. In this task, the samples from the target domains are never seen during training, and are usually out of the data distribution of the source domains. This leads to uncertainty when making predictions ference offers a principled way to represent the predictive uncertainty in neural networks (MacKay, 1992; Kristiadi et al., 2020). We briefly introduce approximate Bayesian inference, under which will introduce our variational invariant learning for domain generalization.
the target domains. Bayesian in-
2.1 APPROXIMATE BAYESIAN INFERENCE
Given a dataset (x(1) y J.V. of N input-output pairs and model parameterized by weights 0 with a prior distribution p()), Bayesian neural networks aim to infer the true posterior distribution p(0|x, x y). As the exact inference of the true posterior is computationally intractable, Hinton & Camp (1993) and Graves (2011) recommended learning a variational distribution g(0) to approximate p(0) y) by minimizing the Kullback-Leebhe (KI.) divergence between them:
0* = arg min DL (0))) |p 0/0 (0x,,))) y)].
(1)
The above optimization is equivalent to minimizing the loss function:
Bayes (o) [log ply x 0)] + (p)
which is also known as the negative value of the evidence lower bound (ELBO) (Blei et al., 2017).
2.2 VARIATIONAL DOMAIN-INVARIANE LEARNING
In domain generalization, let D = (D.) SUT T be set of domains, where S and T denote source domains and target domains respectively. and T do have any overlap with each other but share the same label space. For each domain D; D, we can define joint distribution p( p(x;,V y:) the input space X' and the output space V. We aim to learn a model f X + y in the source domains S that can generalize well to the target domains T.
The fundamental problem domain generalization is to achieve robustness to domain shift between source and target domains, that is, we aim to learn model invariant to the distributional shift be- tween the source and target domains. In this work, we mainly focus on the invariant property across domains instead of exploring general invariance properties (Nalisnick & Smyth, 2018). Therefore, we introduce a formal definition of domain invariance, which is easily incorporated as criteria into the Bayesian framework to achieve domain-invarianl learning.
Provided that all domains in D are in the same domain space, then for any input sample x, in domain D,, assume that there exists domain- function that is able to project X, to other different domains D: with respect to the parameter Ç. where Ç g()), and a different lead to different post-transformation domains Dç. Usually the exact form of 9c(-) is not necessarily known. Under this assumption, we introduce the definition of domain invariance, which we will incorporate into the Bayesian layers of neural networks for domain-invariant learning.
function gc(-) which is defined mapping
Definition 2.1 (Domain Invariance) Let X, be given sample from domain D, E D, and X. = (x(x)) be a transformation of x, in another domain Dr, where Ç 9({)- po(ylx) denotes the output distribution of input x with model 0. The model O is domain-invariant if.
Po (xx) po (xc):
JC~ g())
(3)
2

---

Under review as a conference paper at ICLR 2021
Here, y to represent the output from neural layer with input x. which either be the prediction vector from the last layer or the feature vector from the convolutional layers.
To make the domain- invariant principle casier to implement, then extend the Eq. (3) to expec- tation form:
pe xx,) E [x<(x)))
Based on this definition, DKL [pe (y, [x,]] Egg (Yç (x/x)))]]= to quantify the domain invariance of the model, which will be zero when the model domain invariant. As in most cases, there is analytical form of the domain- transform function and only a few samples from Dr are available, which makes Eng [Po (xcc)))))) intractable. Thus, we derive the following upper bound of the divergence:
the Kullback-Leibler divergence between the two terms in Eq. (4),
DKL (y (y (x/xx)))ll (yc [x] S Eeg [KL [po x. (ye xx)]
(5)
which can be approximated by Monte Carlo sampling.
We define the complete objective function of our variational invariant learning by combining Eq. (5) with Eq. (2). However, in Bayesian inference, the likelihood is obtained by taking the expectation over the distribution of parameter 0, i.e., Po (y|x) = Eyce) [p(' xx 0)], which is also intractable in
Eq. (5). As the KL divergence is a convex function (Nalisnick & Smyth, 2018), we further extend
Eq. (5) to an upper bound:
[B [x,y. I)Ipp(ysxx oi]]
Egg [KL (oooys/xx)llooy((eeas - Egg
(6)
which is tractable with the unbiased Monte Carlo approximation. The complete derivations of Eq. (5) and Eq. (6) are provided Appendix A.
In addition, it is worth noting that the expressible in reality and there are only a limited number of domains available practice. This problem is exacerbated because the target domain is unseen during training, which further limits the number of available domains. Moreover, in most of the domain generalization databases, for a cer- tain sample x, from domain D.. there is no transformation corresponding to x, in other domains. This prevents the expectation with respect to qe from being directly tractable in general.
distribution a(C) is implicit and in-
Thus, we resort to use an empirically tractable implementation and adopt episodic setting in (Li al., 2019). each episode, we choose one domain from the source domains as the meta- source domain D and the rest are used as the meta-target domains (Dr) To achieve variational invariant learning in the Bayesian framework, use samples from meta-target domains in the same category as x. to approximate the samples of g< (x.). Then we obtain a general loss function for comain-invariant learning:
N
Eg( DKL [p(ya|xe, 0))lp(y) xx) 00]]
C
where (xi) N illustration the domain-invariant loss function can be found in Appendix B.
i=1 are from D.. denoting the samples in the same category as x,, More details and an
With the aforementioned loss functions, we develop the loss function of variational invariant learning for domain generalization:
LVII. Layes + AL1-
Our variational invariant learning combines the Bayesian framework, which is able to intro- duce uncertainty into the network and is beneficial for out-t--distiibuttion problems (Daxberger & Bernández-Lobate 2019), and a domain-invariant loss function L1. which is designed based on pre- dictive distributions to make the model generalize better to the unseen target domains. For Bayesian learning, it has been demonstrated that being just "a bit" Bayesian in the last layer of the neural network can well represent the uncertainty in predictions (Kristiadi et al., 2020). This indicates that applying the Bayesian treatment only to the last layer already brings sufficient benefits of Bayesian inference. Although adding Bayesian inference to more layers improves the performance, it also increases the computational cost. Further, from the perspective of domain invariance, making both
3

---

Under review as a conference paper at ICLR 2021
the classifier and feature extractor more robust to the domain shifts also leads to better performance (Li et al., 2019). Thus, there is a trade-off between the benefits of variational Bayesian domain in- variance and computational efficiency. Instead of applying the Bayesian principle to all the layers of the neural network, in this work we explore domain invariance by applying it to only the classifier layer up and the last feature extraction layer Φb
In this case, the Bayes in Eq. (2) becomes the ELBO with respect to ab and Φ jointly. As they are independent, the Layes is expressed as:
Bayes g ( p(y/x, 1 Φ)] + DKL ( + (xLla()))lp())
The above variational inference objective allows us to explore domain-invariant representations and classifier in unified way. The detailed derivation of Eq. (9) is provided in Appendix A.
Nomain-Invariian Classifier the proposed domain-invariane principle into the last layer of the network, which gives rise to
To establish the domain-invariant classifier, we directly incorporate
=
(i(2))
1 T
E By(()) [KL [ply.lz,, ( (lp(y |z), +)]
(10)
==1
where z denotes the feature representations of input x, and the subscripts and f indicate the meta- source domain and the meta-target domains as in Eq. (7). Since p(y z, ab) is a Bernoulli distribution, can conveniently calculate the KL divergence Eq. (10).
Invariat Representations
To also make the representations domain invariant, we have
1 1 T Σ
((())
Σ
E By(
DKL [p(2, ())lp(())))))))))))))))))))))) x x)]
(11)
where ó are the parameters of the feature extractor. Since the feature extractor is also a Bayesian layer, the distribution of p(z|x, Φ) will be factorized Gaussian if the posterior of & is well. We illustrate this as follows. Let x be the input feature of a Bayesian layer Φ, which has a factor- ized Gaussian posterior, the posterior of the activation of the Bayesian layer is also factorized Gaussian (Kingma et al., 2015);
((pr)) ~N(Ns:: ofg) VoLs EΦ = p(zylx, Φ) NN(7) 03),
(12)
Tilios and
x 113
where =j denotes the j-th element in likewise for zi. and di.j denotes the element at the position (t,j) in Φb Based on this property of the Bayesian framework, we assume that the posterior of our variational invariant feature extractor has factorized Gaussian distribution, which leads to easier calculation of the KL divergence in Eq. (11). Note that with the domain-invariant representations, z in Eq. (10) corresponds samples of the feature representation distributions: 2, p((, xx, Φ) and 24 p(zz Xe, Φ).
2.3
OBJECTIVE FUNCTION
The objective function of our variational invariant learning is defined as:
LVIL = Layes + (xb) + L()),
(13)
where Ay) and A are hyperparameters to control the domain-invariant terms. We adopt Monte Carlo sampling and obtain the empirical objective function for variational invariant learning as follows:
M
LVIL=
log p(y, s xx: 26) $ (m)) + DKL [<(bb)]pp((()]] (
L
M
t=1
m
DKL [p(ysppe (2,s, 26(1)) zi, (
N
T
N
M Σ
+ A
DKL [p(zz Xx, ( & (m))))
M
10=1
(14)

---

Under review as a conference paper at ICLR 2021
where x, and z denote the input and its feature from a.. respectively, and xj and z are from D, as in Eq. (7). The posteriors are set to factorized Gaussian distributions, i.., (( = .V Nup. o and
(()) = "V oz). We adopt the reparameterization trick to draw Monte Carlo samples (Kingma & Welling, 2014) as pb(() = μ + (00) * o., where ( ~ NO 1). We draw the samples for e (fm) in similar way. In the implementation of our variational invariant learning, to increase the flexibility of the prior distribution in our Bayesian layers, we choose to place a scale mixture of two Gaussian
distributions as the priors p(ıb) and p(Φ) (Blundell et al., 2015):
TV + (1- (0, o
(15)
where d1, 0g and x are hyperparameters chosen by cross- gross-validation
RELATED WORK
One solution for domain generalization is to generate more source domain data to increase the prob- ability of covering the data in the target domains (Shankar et al., 2018; Volpi et al., 2018). Shankar et al. (2018) augmented the data by perturbing the input images with adversarial gradients gener- ated by auxiliary classifier. Qiao et al. (2020) proposed more challenging scenario of domain generalization named single domain generalization, which only has one source domain, and they designed an adversarial domain augmentation method create "fictitious" yet "challenging" data. Recently, Zhou et al. (2020) employed a generator to synthesize data from pseudo-novel domains to augment the source domains, maximizing the distance between source and pseudo-novel domains as measured by optimal transport (Peyré et al., 2019). Another solution for domain generalization is based on learning nomain-invariant features (D' Innocente & Caputo, 2018; 88; Li et al., 2018b; 2017a). Muandet et al. (2013) proposed domain-invariant component analysis to learn invariant transfor- mantions by minimizing the dissimilarity across domains. Louizos et al. (2015) learned invariant representations by variational autoencoder (Kingma & Welling, 2014), which introduced Bayesian inference into invariant feature learning. Dou et al. (2019) and Seo et al. (2019) tried to achieve a similar goal by introducing two complementary losses, global class alignment loss and local sam- ple clustering loss, and employing multiple normalization methods. Li et al. (2019) proposed an episodic training algorithm obtain both domain-invariann feature extractor and classifier.
Recently, meta-learning based techniques have been considered to solve domain generalization prob- lems. Li et al. (201 8a) proposed meta-learning domain generalization method which introduced the gradient-based method, i.e., model agnostic meta-learning (Finn et al., 2017), for domain gen- eralization. Balaji al. (2018) address the domain generalization problem by learning regular- ization function in a meta- meta-learning framework, making the model robust to domain shifts. Du et al. (2020) propose the meta variational information bottleneck learn Nomain-invarti representations through episodic training.
To the best of our knowledge, Bayesian neural networks have not yet been explored in domain gen- eralization. Our method introduces variational Bayesian approximation to both the feature extractor and classifier of the neural network in conjunction with the newly introduced domain-invarian prin- ciple for domain generalization. The resultant variational invariant learning combines the represen- tational power of deep neural networks and variational Bayesian inference.
Similar to our proposal, CCSA by Motiian et al. (2017) also aligns representations across domains in the same class. Specifically, CCSA utilizes an L2 distance between deterministic features while exploit Bayesian neural networks to learn domain-invarian representations by minimizing the distance between domain distributions. Theoretically, minimizing the distance between distributions incorporates larger inter-class variance than minimizing distance of deterministic features. More- over, we apply our variational invariant learning to both the feature extractor and the classifier, while CCSA only considers an alignment loss on the feature representations.
4 EXPERIMENTS
4.1 DATASETS AND SETTINGS
We conduct our experiments four widely used benchmarks in domain generalization:
5

---

Under review as a conference paper at ICLR 2021
PACS (Li et al., 2017a) consists of 9,991 images of seven classes from four domains photo, art- painting, cartoon and sketch. We follow the "leave-one-out" protocol in (Li et al., 2017a; 2018b; Carlucci al., 2019), where the model is trained on any three of the four domains, which we call source domains, and tested on the last (target) domain.
Office-Home (Venkateswara al., 2017) also has four domains: art, clipart, product and real- world. There are about 15,500 images of 65 categories for object recognition in office and home environments, We use the same experimental protocol for PACS.
Rotated MNIST and Fashion-MNIST were introduced for evaluating domain generalization in (Piratla et al., 2020). For the fair comparison, we follow their recommended settings and randomly select subset of 2,000 images from MNIST and 10,000 images from Pashion-MNISS considered have been rotated by 0% The subset of images is then rotated by 15t through 75 in intervals of 15", creating five source domains. The target domains are created by rotation angles of 0' and 90º. We use these two datasets to demonstrate the generalizability by comparing the performance on in-distribution and out-of-distribution data.
which is
For four benchmarks, we use ResNet-18 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009) as the CNN backbone. During training, we use Adam optimization (Kingma & Ba, 2014) with the learning rate set to 0.0001, and train for 10,000 iterations. In each iteration we choose one source domain as the meta-source domain. The batch size is 128. To fit memory footprint, choose a maximum number of samples per category per target domain to implement the domain- invariant learning, i.e. sixteen for PACS, Rotated MNIST and Fashion-MNIST MNIST datasets, and four for the Office- Home dataset. We choose X and X based on the performance on the validation set and their influence is summarized in the new Fig in Appendix C. The optimal values of X and X are 0.1 and 100 respectively. Parameters σ, and 02 in Eq. (15) are set to 0.1 and 1.5. The model with the highest validation set accuracy is employed for evaluation on the target domain. All code will be made publicly available.
4.2 ABLATION STUDY
We conduct an ablation study to investigate the effectiveness of our variational invariant learning for domain generalization. The experiments are performed on the PACS dataset. Since the major contributions of this work are the Bayesian treatment and the domain-invariant principle, we evaluate their effect by individually incorporating them into the classifier the last layer sp and the feature extractor the penultimate layer Φ. The results are shown in Table 1. The "V" and "x" in the "Bayesian" column denote whether the classifier & and feature extractor & are Bayesian layers or deterministic layers. In the "Invariant" column they denote whether the domain-invarian loss is introduced into the classifier and the feature extractor. Note that the predictive distribution is a Bernoulli distribution, which also admits the domain-invarian loss, therefore include this case for a comprehensive comparison.
In Table 1, the first four rows demonstrate the benefit of the Bayesian treatment. The first (a) serves as a baseline model, which is a vanilla deep convolutional network without any Bayesian treatment and domain-invariant loss. The backbone is also ResNet-18 pretrained on ImageNet. is clear the Bayesian treatment, either for the classifier (b) or the feature extractor (c), improves the performance, especially in the "Arrt-aintiing and "Sketch" domains, and this is further demonstrated in (d) where we employ the Bayesian classifier and feature extractor simultaneously.
The benefit of the and (f) to (b). The settings with domain invariance consistently perform better than those without it. similar trend is also observed when applying the
principle for classifiers is demonstrated by comparing (e) to (a)
principle to the feature extrac- tor, as shown by comparing (g) to (c). Overall, our variational invariant learning (h) achieves the best performance compared to other variants, demonstrating its effectiveness for domain generalization. Note that the feature distributions p(z|x) are unknown without Bayesian formalism, leading to an intractable (Φ)). Therefore, we do not conduct the experiment with only the domain-invarti loss on both the classifier and the feature extractor.
To further demonstrate the invariann property of our method, we visualize the features learned by different settings of method in Table 1. We use t-SNE (Maaten & Hinton, 2008)

---

Under review as a conference paper at ICLR 2021
Table I: Ablation study PACS. All the individual components of our variational invariant learning benefit domain generalization performance. More comparisons can be found in Appendix D
Bayesian
Invariant
PACS
ID
Φ
Photo
Art-painting
Cartoon
Sketch
Mean
a
x
x
X
X
92.85
75.12
77.44
75.72
80.28
(
V
X
93.89
77.88
78.20
77.75
81.93
cc)
X
V
X
92.81
78.66
77.90
78.72
82.02
(d)
V
93.83
82.13
79.18
79.03
83.73
(e) X
X
V
X
93.95
80.03
78.03
77.83
82.46
(f) X
95.21
81.25
80.67
79.31
84.11
g X
V
V
95.15
80.96
79.57
79.15
83.71
(h)
95.97
83.92
81.61
80.31
85.45
(a)
(b)
(c)
(d)
(h)
Figure 1: Visualization of feature representations. The eight sub-figures correspond to the eight settings Table 1 (identified by ID). Colors denote domains, while shapes indicate classes. The target domain (violet) is "art-painting". The top row shows the Bayesian treatment enlarges the inter-class distance for all domains, considerably. The bottom row, compared with the top-row figures in the same column, shows the domain- innaiiainvariant principle enlarges the inter-class distance in the target domain by reducing the intra-class distances between the source and target domains.
reduce the feature dimension into two-dimensional subspace, following Du al. (2020). The visualization is shown in Fig. 1. More visualization results are provided in Appendix E.
Figs. (a), (b), (c) and (d) show the baseline, Bayesian classifier, Bayesian representations and both Bayesian classifier and representations, demonstrating the benefits of Bayesian inference for learning domain-invariant features. The Bayesian treatment both the classifier and the feature extractor enlarges the inter-class distance in all domains, which benefits the classification perfor- mance the target domain, as shown in Fig. (d). Figs. 1 (e). (f), (g). (h) are visualizations of feature representations after introducing the domain-invarian principle, compared to the upper row. Comparing the two figures in each column indicates that our domain-invarian principle imposed on either the representation or the classifier further enlarges the inter-class distances. At the same time, reduces the distance between samples the same class from different domains. This is even more apparent in the intra-class distance between samples from source and target domains. As result, the inter-class distances in the target domain become larger, therefore improving performance. is worth noting that the domain invariant principle the classifier in Fig. (f) and the feature invariant features. Our variational invariant learning in Fig. (h) therefore has better performance by combining their benefits.
tractor in Fig. (g)) both improve the domain-invariant

---

Under review as a conference paper at ICLR 2021
Table Ablation with two variational invariant layers in the feature extractor on PACS. Bayesian ø' and Invariant ø' denote whether the additional variational invariant layer in the feature extractor has a Bayesian property and domain-invariin property. More Bayesian layers benefits the performance while excessive domain-invariant learning harms it.
Bayesian &' Invariant ø'
Photo Art-painting
Cartoon
Sketch
Mean
95.97
83.92
81.61
80.31
85,45
X
95.69
83.28
82.06
81.00
85.51
95.72
82.33
81.10
80.67
84.96
We also experiments with more layers in the feature extractor, Table 2. "Bayesian d" and "Invariant d² denote whether the additional feature extraction layer Ø' has the Bayesian property and domain-invvriant property. The classifiers have both properties in all in Table 2. The first row is the setting with only one variational invariant layer in the feature extractor. When introducing another Bayesian learning layer ø' without the domain- invariint property into the model, as shown in the second row in Table 2, the average performance improves slightly. If we introduce both Bayesian learning and domain-invariant learning into ø¹, as shown in the third row, the overall performance declines bit. One reason might be the information loss in feature representations caused by the excessive use of domain-invariant learning. In addition, due to the Bayesian inference and Monte-Carle sampling, more iriiationlliivariian layers leads to higher memory usage and more computations, which is also one reason for us to apply the variational invariant learning only to the last feature extraction layer and the classifier.
Table 3: Comparison PACS. Our method achieves the best performance on the "Cartoon" domain, competitive on the other three domains and obtains the best overall mean accuracy.
Photo
Art-painting
Cartoon
Sketch
Mean
Baseline
92.85
75.12
77.44
75.72
80.28
JiGen (Carlucci et al., 2019)
96.03
79.42
75.25
71.35
80.51
Epi-FCR (Li et al., 2019)
93.90
82.10
77.00
73.00
81.50
MetaReg (Balaji et al., 2018)
95.50
83.70
77.20
70.30
81.68
MASF (Dou et al., 2019)
94.99
80.29
77.17
71.69
81.04
CSD (Piratla et al., 2020)
94.10
78.90
75.80
76.70
81.38
DMG (Chattopadhyay et al., 2020)
93.35
76.90
80.38
75.21
81.46
L2A-OT (Zhou et al., 2020)
96.20
83.30
78.20
73.60
82.83
DSON (Seo et al., 2019)
95.87
84.67
77.65
82.23
85.11
RSC (Huang et al., 2020)
95.99
83.43
80.31
80.85
85.15
VIL (This paper)
95.97
83.92
81.61
80.31
85.45
4.3 STATE-OF-THE-ART COMPARISON
In this section, we compare our method with several -ate-offthheart methods on four datasets. The results are reported in Tables 3-5. The baseline on PACS (Table 3), Office-Home (Table 4), and rotated MNIST and Mushion-MNIST (Table 5) are all based on the same vanilla deep convolutional ResNet- 18 network, without any Bayesian treatment, the same as row (a) in Table
On PACS, as shown in Table 3, our variational invariant learning method achieves the best overall performance. On each domain, our performance is competitive with the state-of-the-art and we exceed all other methods on the "Cartoon" domain. On Homee as shown in Table 4, we again achieve the best recognition accuracy. It is worth mentioning that the most challenging "Art" and "Clipart" domains, our variational invariant learning also delivers the highest performance, with a good improvement
previous methods.
L2A-OT and DSON outperform the proposed model on some domains PACS and Office- Home. L2A-OT learns generator synthesize data from pseudo-nove novel domains augment the source domains. The pseudo-novel domains often have similar characteristies with the source data. Thus, when the target data also have similar characteristics with the source domains this pays off as the
8

---

Under review as a conference paper at ICLR 2021
Table 4: Comparison Office-Home. Our variational invariant learning achieves the best perfor- mance on the "Art" and "Clipart" domains, while being competitive on the "Product" and "Real" domains. Again we report the best overall mean accuracy.
Art Clipart
Product
Real
Mean
Baseline
54.84
49.85
72.40
73.14
62.55
JiGen (Carlucci et al., 2019)
53.04
47.51
71.47
72.79
61.20
CCSA (Motian et al., 2017)
59.90
49.90
74.10
75.70
64.90
MMD-AAE (Li et al., 2018b)
56.50
47.30
72.10
74.80
62.68
CrossGrad (Shankar al., 2018)
58.40
49.40
73.90
75.80
64.38
DSON (Seo et al., 2019)
59.37
45.70
71.84
74.68
62.90
RSC (Huang et al., 2020)
58.42
47.90
71.63
74.54
63.12
1.2A-OT (Zhou et al., 2020)
60.60
50.10
74.80
77.00
65.63
VIL (This paper)
61.81
53.27
74.27
76.31
66.42
Table 5: Comparison Rotated MNIST and Pashion-MNIST. In-distribution performance is eval-
uated on the test sets of MNIST and Fashion-MNIST with rotation angles of 15", 1,,, 45", 60% and 75, while the out-of-distribution performance is evaluated on test sets with angles of 09 and 90%. Our VIL achieves the best performance on both the in-distribution and out-of-distribution test sets.
MNIST
Fashion-MNIS:
In-distribution Out-of-distribution In-distribution Out-of-distribution
Baseline
98.4
93.5
89.6
76.9
MASF (Dou et al., 2019)
98.2
93.2
86.9
72.4
CSD (Piratla et al., 2020)
98.4
94.7
89.7
78.0
VII (This paper)
99.0
96.5
91.5
83.5
pseudo domains more likely to cover the target domain, such "Product" and "Real World" in Office-Home and "Photo" PACS. When the test domain is different from all of the training domains the performance suffers, e.g., "Clipart" in Office-Home and "Sketch" in PACS. Our method representations and classífiers, resulting in competitive results across all domains and overall. DSON mixtures batch and instance normalization for domain generalization. This tactic is effective on PACS, but less competitive on Office-Home, We attribute this to the larger number of categories on Office-Home where instance normalization is known to make features less discriminative with respect to object categories (Seo et al., 2019). Our Nomainninvariian network makes feature distributions and predictive distributions similar across domains, resulting good performance on both PACS and Office-Home,
generates domain-invarisn
On the Rotated MNIST and Fashion-MNISS datasets, following the experimental settings in (Piratla et al., 2020), we evaluate our method on the in-distribution and out-of-distribution sets. As shown in Table 5, our VIL achieves the best performance on both sets of the two datasets, surpassing other methods. Moreover, out-of-distributton sets, demonstrating its strong generalizability to unseen domains, which is also consistent with the findings in Fig. 1.
method especially improves the classification performance the
CONCLUSION
In this work, we propose variational invariant learning (VIL). variational Bayesian learning frame- work for domain generalization. We introduce Bayesian neural networks into the model, which is able to better represent uncertainty and enhance the generalization to disto--disttibuttinn data. To handle the domain shift between source and target domains, we propose a comainn-nvarianl principle under the variational inference framework, which incorporated by establishing a domain-invariant feature extractor and classifier. Our variational invariant learning combines the representational power of deep neural networks and uncertainty modeling ability of Bayesian learning, showing great effectiveness for domain generalization. Extensive ablation studies demonstrate the benefits of the Bayesian inference and domain-invarinn principle for domain generalization. Our variational invariant learning sets state-of-the-art four domain generalization benchmarks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Andrei Atanov, Arsenii Ashukha, Kirill Struminsky, Dmitriy Vetrov, and Max Welling, The deep weight prior. International Conference on Learning Representations, 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using netaaregularizztion. In Advances in Neural Information Processing Systems, pp- 998-1008, 2018. 1,5,8
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisti- cians. Journal of the American Statistical Association, 112(518)855--87, 2017.
Charles Blundell, Julien Cornebise, K. Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. ArXiv, abs/55505,0424, 2015. 1, 13
Fabio M Carlucci, Antonio Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In IEEE Conference on Computer Vision and Puttern Recognition, pp. 2229-2238, 2019. 6, 8,
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. arXiv preprint arXiv:2008. 12839, 2020.
Erik Daxberger and José Miguel Hernández-Lobato. Bayesian variational autoencoders for unsu- pervised out-of-distribution detection. arXiv preprint 1922:511,,,0551,, 2019. 3
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain general- ization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, pp. 6450-6461, 2019. 8,9
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. arXiv preprint 2xXiv:2007.07645, 2020. 5,7
Antonio D' I'Innocente and Barbara Caputo. Domain generalization with domain-specinn aggregation modules. In German Conference on Pattern Recognition, pp. 187-198. Springer, 2018. 1,
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126-1135, 2017. 5
A. Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. 1,
Geoffrey E. Hinton and Camp. Keeping the neural networks simple by minimizing the description length of the weights. In COLT '93, 1993. 2
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. arXiv preprint arXiv:2007.0244 022454 2020. 8,
Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. arXiv preprint 19...990512221 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412. 144266988, 2014.
10

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. Advances in neural information processing systems, pp. 2575-2583, 2015,
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just a bit, fixes overconfidence in relu networks. arXiv preprint arXiv:2002.101 18, 2020. 1,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012. 1
Da Li, Yongxin Yang. Yi-Zhe Song. and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp- 5542-5550, 2017a. 1,5,6
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. arXiv preprint arXiv:1710.03463, 2017b.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI Conference on Artificial Intelligence, 201 8a. 1,
Da Li, J. Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M. Hospedales. Episodic training for domain generalization. IEEE/CVF International Conference on Computer Vision, pp- 1446-1455, 2019. 3, 4, 5, 8
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400- 5409, 2018b. 5, 6, 9
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pp. 2218-2227, 2017.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. arXiv preprint arXiv: 151.00000 2015. 5
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9NNvv):55792260) 2008.
David JC MacKay The evidence framework applied classification networks. Neural computation, 4(5):720-736, 1992. 1,2
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks, In Advances in Neural Information Processing Systems, pp. 7047-7058, 2018.
Sacid Motian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. In IEEE International Conference on Computer Vision, pp. 5715-5725, 2017. 5,9
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013. 1,
Eric Nalisnick and Padhraic Smyth. Learning priors for invariance. International Conference on Artificial Intelligence and Statistics, pp. 366-375, 2018. 2,3
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Cuong V Nguyen, Yingzhen Li. Thang D Bui, and Richard Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
11

---

Under review as a conference paper at ICLR 2021
Gabriel Peyré, Marco Cuturi, al. Computational optimal transport: With applications data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-677, 2019.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specific low-rank decomposition. arXiv preprint urXiv:2003./2815, 2020. 6, 8,
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556- 12565, 2020. 5
Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. arXiv preprint arXiv:1907.0422 2019. 5, 8,
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXiv: 10745, 2018. 5,9
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In JEEE Conference on Computer Vision and Puttern Recognition, pp. 5018-5027, 2017. 6
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in neural information processing systems, pp. 5334-5344 2018. 5
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. arXiv preprint arXiv:2007.033000 2020. 5, 8, 9
A DERIVATION
A.1 DERIVATION OF THE UPPER BOUNDS OF DOMAIN-IVVARIAN LEARNING
As in most cases the Eyg (pe (yclxx)] is intractable, we derive the upper bound in Eq. (5), which is achieved via Jensen's inequality:
DKL. [pocppol. (x.)I (po(y [x<]] 1x.) po Epo(ys/xx)) llog Eng (yclxe))
I (x., Po , Bg [log Po (xc1xx)]
=1Egc [KL (potlpooy.... (x))o (yclxe))
(16)
In Bayesian inference, computing the likelihood po (y|x) So p(y|x, 0)d0 E [p(y|x, 0)]
notoriously difficult. Thus, as the fact that KL divergence is a convex function, we obtain the upper bound Eq, (6) achieved via Jensen's inequality similar to Eq. (16):
Egg DKL
= Egg
DKL (Egtt) xx. 0) E O1]]
(17)
DklyyDKulppy o0 0) ])]
< Eg
A.2 DERIVATION OF VARIATIONAL BAYESIAN APPROXIMATION FOR REPRESENTATION (Φ)
AND CLASSIFIER (1) LAYERS.
We consider the model with two Bayesian layers & and z the last layer of feature extractor and the classifier respectively. The prior distribution of the model is p(Φ, ab), and the true posterior distribution is p(Ø, ub y). Following the settings in Section 2.1, need to learn variational dis- tribution g(0, 1b) to approximate the true posterior by minimizing the KL divergence from g( b)) p(O, vb Xx y):
Φ' 2/ arg min DEL (g(0) ())p((() (1x,)))))))))))))
(18)
12

---

Under review as a conference paper at ICLR 2021
Ds
X5
Zs
1P
Ys
S
4/0
D
{x1) N
N
D1
i= i=1
[1 (=1
Disi i=1
T
D2
(x N
{x2) N
fr
(=1
:
:
Dr
XT
N
N ( i==
{yr Jy
1=
(D:3==1
Figure 2: Illustration of the domain-invariant loss in the training phase of VIL. S denotes the source domains, T denotes the target domains, and DD S U D. x, and y denote inputs, features and outputs of samples in each domain. Li(Φ) and (z(+)) denote the domain-invariant loss functions for representations and the classifier.
By applying the Bayesian rule p(Φ, y)x x ply|x, Φ. b)p( $)) the optimization is equivalent to minimizing:
q( vb)
Layyes
= g( 2)) log p(c bb) x, Φb ah)
dodp
(19)
DeL (9( [KL[9((0 <))p((( +)] (.( (ogp(y(x,,,.)))
With Φ and &p are independent,
Bayes - Egi Fuc(s) [logrly p(y) bb, +)] + [xr.[(((>))lp(bb + mKa.
(p(d))
(20)
B DETAILS OF DOMAIN-INVARIAN LOSS IN VIL TRAINING
We split the training phase of VIL into several episodes. In each episode, as shown in Fig. 2, randomly choose source domain as the meta-source domain D., and the rest of the source domains {De) are treated as the meta-target domains. From D., we randomly select a batch of samples x,- For each x,, then select N samples [x{] which are in the same category as x,, from each of the meta-target domains D.. All of these samples are sent to the variational invariant feature extractor & to get the representations z, and {zi ) which are then sent to the variational invariant classifier 2b obtain the predictions y. and {y]]- We obtain the domain-invvriann loss for feature extractor L) by calculating the mean the KL divergence of z, and each z as (11). 1). The domain-invariant loss for feature classifier (+(b) is calculated in similar way ys N
and {yi] yifi= as (10).
C ABLATION STUDY FOR HYPERPARAMTERS
We also ablate the hyperparameters >q. Xup and π on PACS with cartoon the target domain. Results are shown in Fig 3 (a), (b) and (c). We obtain Fig (a) by fixing A as 100 and adjusting A Fig (b) by fixing X as and adjusting Ay and Fig (c) by adjusting и while fixing other settings as in Section 4.1. À and A balance the influence of the Bayesian learning and Nomain-invarian learning, and their optimal values are and 100. If the values are too small, the model tends to overfit to source domains as the performance on target data drops more obviously than on validation data. contrast, too large values of them harm the overall performance of the model as there are obvious decrease of accuracy components of the scale mixture prior of our Bayesian model. According to Blundell et al. (2015), the two components cause prior density with heavier tail while many weights tightly concentrate around zero. Both of them are important. The performance is the best when π is 0.5 according to Fig 3 (c), which demonstrates the effectiveness of the components in the scale mixture prior.
both validation data and target data. Moreover, balances the two
13

---

Under review as a conference paper at ICLR 2021
Aocraky
81.
05.0
6000
800
Target diica
750
Vasdacion data
750
750
Axp
(a)
(b)
(c)
Figure 3: Performance on "Cartoon" domain in PACS with different hyperparameters Aøs Ay and m. The red line denotes the accuracy validation data while the blue line denotes accuracy target data. The optimal value of X Ag. and are 100 and 0.5 respectively.
Table 6: More detailed ablation study on PACS. Compared to Table we add three more settings with IDs (i), (i) and (k). All the individual components of our variational invariant learning benefit domain generalization performance.
Bayesian Invariant
PACS
ID
+b
Photo
Art painting
Cartoon
Sketch
Mean
a
x
x
92.85
75.12
77.44
75.72
80.28
(b)
V
X
93.89
77.88
78.20
77.75
81.93
c
X
V
X
X
92.81
78.66
77.90
78.72
82.02
(d)
V
V
93.83
82.13
79.18
79.03
83.73
(e) X
X
X 93.95
80.03
78.03
77.83
82.46
(1)
95.21
81.25
80.67
79.31
84.11
g X
Y
X
V
95.15
80.96
79.57
79.15
83.71
1) Y
95.33
81.20
80.97
80.07
84.39
1 V
V
X
95.87
81.15
79.39
80.15
84.14
1K X
95.39
82.32
80.27
79.61
84.40
(h)
V
95.97
83.92
81.61
80.31
85.45
D DETAILED ABLATION STUDY ON PACS
In addition to the aforementioned experiments, conduct some supplementary experiments with other settings on PACS to further demonstrate the effectiveness of the Bayesian inference and domain-invariant loss shown in Table 6. The evaluated components the For better comparison, we show the contents of Table again Table 6, and add three other settings with IDs (i), (i) and (k). Note that as the distribution of features unknown without Bayesian feature extractor Φ the settings with Lr(Φ) and non-Bayesian feature extractor is intractable.
in Table 1.
Comparing (i) with (d), find that employing Bayesian inference the last layer of the feature extractor improves the overall performance and the classification accuracy three of the four do- mains. Moreover, comparing (j) and (k) to (g) shows the benefits of introducing variational Bayes and the domain-invarian loss to the classifier on most of the domains and the average of them.
E EXTRA VISUALIZATION RESULTS
To further observe and analyze the benefits of the individual components of VII. for domain-invarian learning, we visualize the features of all categories from the target domain only in Fig. 4, and features of only category from all domains in Fig. 5. The same as Fig. 1, the visualization is conducted on the PACS dataset and the target domain is "art-painting". The chosen category in Fig. 5 is "horse".
Fig. 4 provides more intuitive observation of the benefits of the Bayesian framework and domain- invariant learning in our method for enlarging the inter-class distance in the target domain. The conclusion is similar in Fig. 1. From the figures in the first row, it is clear that the Bayesian framework whether in the classifier ((b) or the feature extractor ((c) increases the inter-class dis- tance compared with the baseline method (()). With both of them ((d), the performance becomes
14

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
(c)
(e)
(h)
Figure 4: Visualization of feature representations of target domain. Different colors denote dif- ferent categories. The sub-figures have the same experimental settings the experiments in Table and Fig. Visualizing only the feature representations of the target domain shows the benefits of the individual components to the target domain recognition more intuitivell. The target domain is hurttpainting".. as Fig. 1. We obtain a similar conclusion to Section 4.2, where the Bayesian infer- ence enlarges the inter-class distance for all domains and the domain-invarinn principle reduces the intra-class distance of the source and target domains.
(a)
(b)
(c)
(d)
(e)
(1)
(g)
(h)
Figure 5: Visualization of feature representations of one category. All samples are from the "horse" category with colors denoting different domains. The target domain is "art-painting" (violet). The top row shows Bayesian inference benefits domain generalization by gathering features from differ- ent domains to the same manifold. The figures in each column indicate aomain-invarian learning reduces the intra-class distance between domains, resulting better target domain performance.
better. Further, comparing two figures in each column, the inter-class distance is also enlarged by introducing the domain-invariant principle into the setting of each figure in the first row. VIL (()) achieves the best performance by combining the benefits of both the Bayesian framework and the domain-invariant principle both feature extractor and classifier.
Fig. 5 provides deeper insight into the intra-class feature distributions of the same category from different domains. By introducing the Bayesian inference into the model, the features demonstrate the manifold of the category shown in the first row ((b), (c) and (d)). This makes recognition easier. Indeed, the visualization of features from multiple categories has similar properties as shown in Fig. 1. As shown in each column, introducing the domain invariant learning into the model leads to a better mixture of features from different domains. The resultant comain-invarinn representation makes the model more generalizable unseen domains.
15

---

Under review as a conference paper at ICLR 2021
We also visualize the features in rotated MNIST and Fashion MNIST datasets, as shown in Fig. 6. Different shapes denote different categories. Red samples denote features from the in-distribution set and blue samples denote features from the out-of-distribution set. Compared with the baseline, method reduces the intra-class distance between samples from the in-distribution set and the out-of- distribution set and clusters the out-of-distribution samples of the same categories better, especially in the rotated Fashion-MNNI dataset.
MNIT
Fashion-MNIST
Baseline
VIL
Baseline
VIL
Figure 6: Visualization feature representations in rotated MNIST and rotated Fashion-MNIST datasets. Samples from the in-distribution and out-of-distribution sets are in red and blue, respec-
tively. Different shapes denote different categories. Compared to other methods, our VIL achieves better performance both the in-distribution and out-of-distribution sets in each dataset, and espe- cially on the -uttt--diitiiuutiin set from the Mashion-MNISS benchmark.
16


Reviews:
[{"review_id": "bCeFSKfymgw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- The paper proposes a Bayesian inference framework for domain generalization (DG) that deals with both the uncertainty and domain shit. \nThe proposed method treats the uncertainty effectively by applying the variational Bayesian inference to the last two layers of the model (feature representations and classifier layers).\nTo make the model domain-invariant, the proposed method introduces a domain-invariant principle that makes distributions of the model's outputs (representations) in the same class similar across domains. \nThe paper experimentally demonstrated the effectiveness of the method with four visual datasets.\n\nPros:\n- The problem tackled by this paper is very important because accurate prediction is generally difficult in DG due to the lack of target training data and therefore considering the uncertainty is especially important.\n- Experimental results with four widely used datasets show the effectiveness of the method.\n\nCons:\n-  The technical novelty of this method seems a bit low. This is because this method is a relatively simple combination of Bayesian modeling of neural networks and some existing techniques for DG. \nIn previous works for DG, techniques to bring distributions of model's outputs in the same class closer together across domains have been proposed.\nFor example, CCSA [1] learns the domain-invariant representations by matching the representations in the same class across domains.\n\n[1] Motiian et al., Unified deep supervised domain adaptation and generalization, ICCV2017\n\n- More analysis of hyperparameters such as $\\lambda_{\\phi}$, $\\lambda_{\\psi}$, and $\\pi$ would improve the quality of this paper.\n\nReasons for Scores:\n- Although I like the research direction of this paper as described in Pros, I have some concerns about the technical novelty of this method as described in Cons.\n\nMinor comments: \n- definition 2.1 (domain invariance) is a little confusing for me although I can see what the method wants to do by seeing Eq. (7). For example, what is the formal definition of domain-transform function?\n- Although the loss function for domain-invariant learning (7) seems to bring the distributions of the same class across domains closer together, can the method incorporate a\nmechanism to enhance dissimilarity of the different classes across domains? I think that it might improve performance.\n- There are some ambiguous statements. In Eq. (10), are $z_{s}$ and $z_{t}$ the means of distribution of $q(z|x,\\phi)$?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nf-0ZWZlFN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a variational Bayesian approach to domain adaptation.  The goal is to achieve flexibility in specifying domain invariance as well as modeling uncertainty. The variational approach could also help generalization.\n\nThe key idea is simple and straightforward — a distribution over domains is first specified and variational approximation is then used for parameter estimation. In particular, the authors decompose the variational distribution in terms of the domain and classifier parameters, applying the approximation to the last two network layers. The objective function is given as a weighted sum of the KL divergence corresponding the two component distributions.\n\nIn the empirical evaluation, the authors compare the performance on in-distribution and out-of-distribution data. They conclude that the proposed method achieves generally improved accuracy on several image datasets, including PACS, Office-Home, as well as MNIST variants. \n\nStrengths:\n- The presentation is clear. The proposed Bayesian formulation is natural and the resulting learning problem is computationally practical.\n\nWeaknesses: \n- Some parameters such as the weights \\lambda_{\\phi} and \\lambda_{\\phi} do not have clear Bayesian interpretations.\n- In the experimental evaluation, different baseline methods are used for each datasets. \n\nQuestions:\nHow do the parameters \\lambda_{\\phi} and \\lambda_{\\phi} impact the performance? Could we interpret these parameters from the MLE perspective?", "extra_scores": {}, "extra_text": {}}, {"review_id": "LCahpY941o", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes variational invariant learning (VIL) as a framework for probabilistic inference that jointly models domain invariance and uncertainty. The approach exploits variational Bayesian approximation in both feature encoding and classifier layers in order to facilitate domain generalization and invariance. The paper evaluates VIL on benchmarks for cross-domain visual object classification.\n\nPositives:\nThe paper presents their methodology and definitions very clearly. Comparisons to state-of-the-art methods are well presented. The ablation study is very clear to provide a thorough understanding of the approach. Overall I like this paper.\n\nConcerns:\nThe authors state: \"the Bayesian approach has not yet been explored in domain generalization\" or \"this is the first work to adopt variational Bayes to domain generalization\". I may not strongly agree on these statements in their current form. There has been some previous work on extending variational autoencoding frameworks for domain generalization from several perspectives (e.g, adversarial inference). This type of variational inference based models ideally exploit a Bayesian approach for domain generalization as well. I would expect the authors to rephrase their claims in a more specific manner to make this distinction. Regarding this concern, one example to consider for instance is \"The Variational Fair Autoencoder\" (Louizos et al., ICLR 2016) approach for invariant representation learning with variational autoencoders for domain generalization.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2UG1dHNNbX5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "########################\n\nSummary:\n\nThe paper introduces a variational invariant learning approach with Bayesian approximation for domain generalization.\n\n####################\n\nReason for score:\n\nOverall, the paper is above the borderline. I like the idea of utilizing the Bayesian variational learning to address the domain generalization problem, which is very novel and promising. My major concern is about some unclear parts described in the paper and insufficient experimental comparison (see cons below). Hopefully, it would be grateful that the authors could address my concerns during the rebuttal period.\n\n########################\n\nPros:\n\n(1) The proposed variational Bayesian learning framework in the paper to represent uncertainty and enhance the generalization is reasonable and interesting, which can inspire other researchers in related fields.\n\n(2) The introduced domain-invariant principle to establish a domain-invariant feature extractor and classifier seems promising, which can lead to an end-to-end framework with CNN and a Bayesian network.\n\n(3) Extensive experimental results on four domain generalization benchmarks show the proposed method obtains a new state-of-the-art performance, which is appealing and convincing.\n\n########################\n\nCons:\n\n(1) In the paper, the authors claim that they only apply the domain invariance to the last feature extraction layer. In order to examine the relevance between the number of layers and the performance, an ablation study about the number of feature extraction layers should be added. In addition, the efficiency discussion about the model layers will be better to denote the setting chosen in the work.\n\n(2) In Table 1, what happens if without Bayesian and with both invariant, i.e., only introducing invariant loss into both classifier and the feature extractor? I notice that such a certain situation is missing in the experiments.\n\n(3) As shown in Table 2 and Table 3, DSON and L2A-OT can outperform the proposed model in terms of certain metrics, it would be better if the authors can analyze the results and give some comparisons about that.\n\n(4) As for the uncertainty estimated by the proposed Bayesian network, is there any threshold being set in the model to determine which samples can be used in the training？ Moreover, how to utilized such estimated uncertainty should be discussed further.\n\n########################\n\nQuestions during the rebuttal period: \n\nPlease address and clarify the cons above. Thank you!\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although the proposed method shows sota results, it is a simple combination of two existing methods, a bit of Bayesian + domain generalization.  It seems that the total improvement by the proposed method is just the sum of improvements by Bayesian and by domain generalization.  No synergy between Bayesian and domain generalization is observed.

I personally doubt that the Bayesian treatment of the domain generalization loss is not essential.
The derivation in Sec. 2 is unnecessarily complicated.  In derivation from eq(3) to eq(5), the authors first "extend" (3) to (4), which is not appropriate ((4) can hold even if $p(y_{\zeta}|x_{\zeta})$ is highly diverse. ).  After that the authors apply Jensen to come back to an appropriate form (5), which is a weighted sum of distances (which is an appropriate criterion).
If they start from Eq.(5), the proposed objective is simply the sum of the standard ELBO (2) and a natural domain invariance loss (5).  For non-Bayesian treatment of the domain invariant loss, you could simply replace the KL by Lp norm between $y_s$ and $y_\zeta$.  I expected that by answering to the question by Reviewer 1 the authors would prove synergy between Bayesian and domain generalization.  But the authors just excused that
'' The feature distributions  are unknown without Bayesian formalism, leading to an intractable $L_I$. Therefore, we do not conduct the experiment with only the domain-invariant loss on both the classifier and the feature extractor.''
I don't really understand what the authors mean, but the authors should have explained why you cannot replace the KL with non-Bayesian Lp loss.



Decision: Reject