LASSO: Latent Sub-spaces Orientation for Domain Generalization
Authors: Long Tung Vuong
Trung Quoc Phung
Toan Tran
Anh Tuan Tran
Dinh Phung
Trung Le
Venue: ICLR 2022 Submitted (2022)
Keywords: Domain generalization
Image Classification
machine learning
deep learning

Abstract: To achieve a satisfactory generalization performance on prediction tasks in an unseen domain, existing domain generalization (DG) approaches often rely on the strict assumption of fixed domain-invariant features and common hypotheses learned from a set of training domains. While it is a natural and important premise to ground generalization capacity on the target domain, we argue that this assumption could be overly strict and sub-optimal. It is particularly evident when source domains share little information or the target domains leverages information from selective source domains in a compositional way instead of relying on a unique invariant hypothesis across all source domains. Unlike most existing approaches, instead of constructing a single hypothesis shared among domains, we propose a LAtent Sub-Space Orientation (LASSO) method that explores diverse latent sub-spaces and learning individual hypotheses on those sub-spaces. Moreover, in LASSO, since the latent sub-spaces are formed by the label-informative features captured in source domains, they allow us to project target examples onto appropriate sub-spaces, while preserving crucial label-informative features for the label prediction. Finally, we empirically evaluate our method on several well-known DG benchmarks, where it achieves state-of-the-art results.

Under review as a conference paper at ICLR 2022
LASSO: LATENT SUB-SPACES ORIENTATION FOR DOMAIN GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
To achieve satisfactory generalization performance on prediction tasks in an un- seen domain, existing domain generalization (DG) approaches often rely on the assumption of the existence of fixed domain-invariant features and common hy- potheses learned from set of training domains. While it a natural and important premise to ground generalization capacity on the target domain, we argue that this assumption could be overly strict and sub-optimal. It is particularly evident when source domains share little information the target domains leverages informa- tion from selective source domains in a compositional way instead of relying on unique invariant hypothesis across all souree domains. Unlike most existing ap- proaches, instead of constructing single hypothesis shared among domains, we propose LAtent Sub-Space Orientation (LASSO)) method that explores diverse latent sub-spaces and learning individual hypotheses on those sub-spaces. More- over, LASSO, since the latent sub-spaces are formed by the label-informative features captured in source domains, they allow us to project target examples onto appropriate sub-spaces while preserving crucial label-informative features for the label prediction task. Finally, we empirically evaluate our method on several well- known DG benchmarks, where achieves state-of-the-ard results.
INTRODUCTION
One of the most challenging problems in applying machine learning to real-world problems is to address the domain shift encountered when test data at inference time come from different distribu- tions compared training data, often causing unexpectedly imperfect generalization performance. To handle this issue, many out-of-distributiin learning settings have been investigated, notably do- main adaptation (DA) and domain generalization (DG). In particular, DA setting (Mansour et al.) 2009 Ben--aavdd et al. 2010 Zhao et al. 2019) takes the assumption that both labeled source data and unlabeled target data are available at the training phase, while DG setting (Blanchard et al., 2011 Muandet et al. 2013 Ganin al. (0166) is much more challenging due the complete ab- sence of any target data at training time, and the learned model is expected to perform a zero-shot prediction on test samples. While being more challenging than DA, DG is arguably and applicable to real- world scenarios where there is need to rapidly deploy a prediction model on a new target domain without any access target data.
versatile
A common premise on which most of the existing DG approaches rely to address the domain shift problem is learn shared set of features among different source domains, e.g., Nomain-invarian
features Muandet et al. 2013; Ganin et al. 2016 Motiaan et al., 2017 Ghifary et al. 2015) Xie al. 2017 7 et Pir9Piaaatla Piratla et al.] 2020 Zhao al. 2020). By learning
features, these approaches identify latent representation from multiple source domains which is also expected encompass unseen target domains, hence reducing the domain shift. However, since this latent space is often high-dimensional and unseen target domains could vary greatly and un- expectedly, these domain invariance approaches could become ineffective. We argue that using a single hypothesis which requires to be optimal for all source and target domains is rather strong assumption and might not have an effective algorithmic solution, hence presenting a key weakness to these existing approaches.
In this paper, we propose a new approach to efficiently reduce the domain shift. Our idea to optimally decompose the high-dimensional latent feature space to lower-dimensional sub-spaces,

---

Under review as a conference paper at ICLR 2022
after which an individual hypothesis is learned (optimally) for each latent sub-space. In particu- lar, these latent sub-spaces are formed by babel-informative features which are learned from source latent representations that sub-space latent representations information to effectively train sub-space hypotheses. We then perform extensive experimental eval- uations to demonstrate the merits of our methodology and framework. In sum, our contributions as follows:
capture sufficient and necessary
We propose LAtent SubSpace Orientation (LASSO), novel approach for DG, which aims to learn label-informative latent sub-spaces from source latent representations, preserving crucial label information for training qualified individual sub-space hypotheses.
We develop a rigorous theoretical analysis to explain how our approach can learn mean- ingful sub-spaces by learning an indicator function I that helps reduce latent domain shift. Moreover, by linking to information theory (ef. Theorem [}, we theoretically jus- tify the rationale behind learning the sub-space indicator function as a mechanism for element-wise selection of latent features to gather sufficient label information.
Finally, we empirically demonstrate that our proposed method can achieve favorable results when evaluating our model on domain generalization benchmarks in the comparison with state-of-the Ee-thh-an art methods.
2 RELATED WORK
Domain generalization (DG) approaches can be categorized into several groups: domain-invariant representation learning, meta-learning, and augmmnntation//elf-sungg wttp:///ssssuuprrssioiooo The works in the first group aim to learn a domain- invariant representation with the hope of transferring well to unseen domains. Notably, Muandet et al. (2013) construct shared components by minimizing the discrep- ancy of the source domain marginal distributions using kernel-based algorithm. Xiie et al. al.] (2017) employ an adversarial training strategy preserve the desired invariance and eliminate variations of domain factors from features. Seoo et al. (2020) combine batch normalization and instance nor- malization to remove domain speeiffc styles while preserving semantic category information. Other
works (Ghifary et al.) 2015) Li et al.] [2088b llse et al.) 2020) employ auto-ence muto-encoders to support ionainninvariant feature extraction.
Another efficient approach for DG is meta-learning. For example, the studies in (Li et al. 201 8a, Balaji al. 2018) use source domains to simulate meta-train/mmeta-ees.. which encourages the trained model to generalize better on the meta-test data, leading to a more plausible global gen- cralization performance on the unseen target domain. Dou et al. (2019) then extend the work Li
cet al. (2018a) Sa by combining it with metric learning loss encourage domain-independenn semantic feature space.
Self-supervised learning and data augmentation have been also applied to DG. Typically, Carlucci cct al. (2019) propose to solve the pretext task of Jigsaw Puzzles to improve the generalization per- formance on unseen domains. [Shankar al. (2018) augment training data with instances perturbed along with directions of domain change. In addition, Zhou et al.) (2020a) employ a classifier that can learn the generalization on additional augmented samples diversity pseudo-novel domains by leveraging optimal transport theory. Zhou et al.] (2020b) augment the original training data of source domains with synthetic data from unseen domains such that augmented data are correctly classified by the classifier while fooling the domain classifier to make the task model intrinsically more domain-gent geraaaiigneerllzzbbbl.. Zhou et al. (202I) propose mix the styles of different source do- mains based on normalization-based style-transfer technique to effectively increase the diversity of domains during training.
Recently, learning domain-specing information to boost classification performance in domain gen- cralization has attracted more attention. For example, [Huang et al. (2020) iteratively discard the dominant features to exploit all useful features (including both invariant and specific features) that highly correlate with labels. Chattopadhyay et al. (2020) propose a -spaii- mask learned from the domain label to balance domain- domain-invariant invariant and domain-specifi features. As result, source domain classification can benefit from the specialized features while retaining the generalizing ap- plicability of domain-invariant features.
2

---

Under review as a conference paper at ICLR 2022
3 PROPOSED METHOD
3.1 PROBLEM SETTING
Let D = ( v( ) I..., K be the source labeled datasets, where k: is the domain index, Ng is the number of examples in D and y E > := (1,... C} is the set of C' classes. Let P& and Ph be the (marginal) data distribution and corresponding density of the source domain k. Similarly, let p (y x) denote the probabilistic labeling distribution of this domain. A domain is then defined by pair of data distribution and labeling function, in which its data ( is generated by first sampling xf ~ p (.) and subsequently y ~ P (- xf)). With a little abuse of notation, let us also denote k-th domain by Dy and the data generation process is (xt,) ~ DDg For target domain D=, p? and p" are its data distribution and density respectively, while p" (y ax) is the target labeling distribution.
....
A mixture of the source domains is denoted by D :- Σk= TA In order to sample data from this mixture, one first sample a domain index from the categorical distribution k Cat(z), where и specifies the mixture weights, and then sample the actual data from the corresponding domain k. K
The empirical datasets obtained could hint to estimate Th
N
with N
=k N
N
k=1
Finally, mixture of data distribution is defined by -P. = > k- TRPS
hypothesis ʃ : + åg map from the common data space to the the C-simplex label space Ac := {a E RC ||a||] 1 A 2 0}. Let & (ʃ (z:, .y) be the loss incurred by using this hypothesis to predict æ E X. given its ground-truth label y E V. Therefore, the general loss of the hypothesis f w.r.t. the joint distribution IB is:
C DD) E (f) ,)]
(1)
We examine the composite hypothesis f g where g X + is the feature extractor
mapping the data space to a latent space and h Z + V V is the classifier on this latent space. With respect to the latent space, given a probabilistic labeling distribution p (y r). let denote p (y as the probabilistic labeling distribution induced by (v x) and g. that is, for any 2 g (X),
1 (yl =) fg- plul a)p(a)de
JJohansoon et al. 2019). Basically, pd (y 2) can be regarded
J (c)) p(ar)de
weighted sum of p (y x)) f (2). We further define - (y 2) and T (y
2)
with 2 E 9 (X) C 2 as the induced conditional labeling distributions source and target domains on the latent space via the feature extractor g. Finally, we consider the DG setting, in which we only possess the source labeled data and do not have any prior information of any target data during the training process.
In what follows, we develop a rigorous theory to promote our LASSO, whose proof's can be found in the Appendix A
3.2 SINGLE HYPOTHESIS AND FULL SPACE FOR DOMAIN GENERALIZATION
Most existing works in DG use single hypothesis acting the entire latent space 2 for predicting unseen target domains. In what follows, we develop an upper-bound which relevant to the data shift on the latent space of the target loss C (f, D² with f = h a g.
Theorem 1. If the loss function l is upper-bounded by a positive constant L. for any hypothesis f X Ae where = h g with g X Z and h 2 - Ac, the target general loss is upper-bounded by:
C(J,DT) S exp A SGPFT.9PP9 a-1 Lb ((I. max Epg Ap (y) m)]l
(2)
where 9 and 9 PT are forward distributions induced by applying on Ps and PT, R is the a-divergence (a 1), and Aps (y x) [P (y x) - p" (y = =)]]e= represents the label shift between the labeling assignment mechanisms of an individual source domain and target domain on the input space.
3

---

Under review as a conference paper at ICLR 2022
The bound developed in Theorem I supports in characterizing the factors influencing the loss on the target domain of a hypothesis f: (i) the label shift: Egg [llAps (y 1 (ii the source loss: L (f. IDF)), and (iii) the latent data shift: t (gy P gy Based on this theorem, we aim find the hypothesis ʃ such that the upper bound of target loss is minimized. Note that the label shift term is natural characteristic of domains, hence it is almost unchangeable. Therefore, one seeks to minimize the source loss, or the discrepancy between latent distributions, or both of them. Certainly, if we know target data (e.g., samples from P²'), we can minimize the divergence between g PS and g#IP" directly. However, this is almost impossible for the DG setting because the target domain is unknown beforehand when training.
To address this problem, a large number of works propose learning domain invaariint features on a full high-dimensional latent space together with single hypothesis on top of these domain- invariant features. Nonetheless, due to the great variance of unseen target distributions on the full high-dimensional latent space, the latent data shift is possibly high in many cases, which hurts the generalization ability of the single hypothesis on unseen target domains.
Evidently, given latent representation z with label y in the high-dimensional latent space, only a small portion of its features known as babee-informative features is highly relevant to the label y, while the remaining ones are redundant. By eliminating irrelevant features and grouping the latent representations of data examples across multiple domains with the same set of label-informative features, can form latent inboomiiform sub-spaces to reduce the latent data shift, whereas preserving sufficient label-information for training good hypotheses on those latent sub-spaces. This can be explained from the fact that unseen target and source examples with the same label- informative features are projected onto the same latent sub-space on which the latent data shift between mixture of source domains and target domain becomes smaller due the compactness of this sub-space compared to the full high-dimensional latent space.
Furthermore, by learning multiple hypotheses, each of which corresponds to a sub-space, LASSO allows sub-spaces to -iitanye-wisely explore different groups of label-informatis features. hence encouraging the diversity of latent representations for achieving better generalization ability, which concurs with the principle in (Huang et al., 2020 2 [hhttoppdhyyy et al. 2020 Blanchard et al., 2021) This intuitively boosts the generalization ability of hypotheses. The reason that given source examples with their labels, LASSO aims to explore possible compact groups of label-informative features which can predict accurately the labels. In the inference time, given a target example, if the feature extractor
successfully activate group of aabel-informative features, this target example is projected and matched with corresponding source examples on a sub-space in which a good hypothesis is used to predict label for the target example.
3.3 LATENT SUB-SPACES WITH INDIVIDUAL HYPOTHESES FOR DOMAIN GENERALIZATION
For a data sample (x. y) ~ DDS. we consider latent representation == g(x) E RD and propose to learn a sub-space indicator I which renders T(2) E {0, Specifically, r(=) specifies the sub-space which we project = onto via (=) = @ [(2), where @ is the element- deement-wise product operator. Our intuition is that I' is able to keep the most label informative features of latent vector
(i.e., the th element of I(()): T=((=) 1 means that is label-informative and should be selected when projecting 2 onto its sub-space).
Given a sub-space index index m as Am {x : [(g(x))= c X. Let IpS be the distribution restricted by p& over the set Am and PT as the distribution restricted by p7 over An. Eventually, as the probabilistic labeling distribution on the sub-space (Am, IPS). meaning that if x ps S (y 12) = ΣK MEPÉ (y z). Similarly, we define if x ~ Ipf PTT (y =x==p"= (y x).
10, we denote the region on data space which has the same
define pi (y xx)
m
Due to this construction, any data sampled from P.S m or PT have the same index m = (2)) [[(())), which specifies the set of relevant feature elements for classifying data from Am. This hints us to employ an individual hypothesis fm = hm og" where fm specializes on (Am; Ipss and PS with := og for each m. We note that h.m.
hm specializes the sub-space (o" (Am) operates on the sub-space indexed by m, while fm operates on its induced data space. Additionally, since each data point E X corresponds to only single (()(z)), the data space is partitioned
4

---

Under review as a conference paper at ICLR 2022
into disjoint sets, i.e., X Um Am. where Am n A,, 0, Vm & Therefore, these different hypotheses hyn make use of different features, as reflected in the different sub-space index m.
We now present the way to efficiently formulate sub-space hypotheses. Let F h E H be a hypothesis class of f : X + Ac. where f f= h o with g X + 2 E 9 and h 2 Ac E H. Each hypothesis f h induces the sub- space hypotheses fM [5mlme for which fm (x)) = hm (T(g(x)) o g (x)) for x E Am with hm (m @ (z)) = h(m @ g (z)) can be viewed as a hypothesis on the sub-space indexed by m. We define the general loss of fM [fmlmEM on the target domain as
ff=hop ho:ggg
C (fM.DD)
c (fm, DT). Σ
(3)
MI
MEM
where M c {0.1) is the set of all feasible sub sub-space indices conducted from the observed data, MM specifies the cardinality of M, and D/ is the joint distribution of (It, y) where x ~ IP and y pm (. x)).
Indicaio x labe--infimmaiive
Smartee
Clantier
Emerasy
Ris
Input space
Latent space
Dony ot Feakunt ds
(a)
(b)
Figure 1: (a) Latent Sub-space Orientation Framework and (b) An example of reducing data shift between source domains and target domain in the latent sub-spaces.
Figure La depicts the generative process of our LASSO in which the sub-space indicator I' specifies the sub-space to project
g(x) onto and the projected sample is then predicted by a sub-space hypothesis. Figure lb we intuitively demonstrate why projecting onto appropriate sub-spaces is useful to reduce latent data shift. For the red points on the latent space, the dimension is commonly label-informative, hence projecting onto this dimension is helpful to preserve label information for constructing a good hypothesis on this sub-space. The same observation occurs for the blue points for the dimension 2. The green points symbolize target data for which we observe that the latent data shift on the full latent space is significantly greater than that on the sub-spaces. In addition, Figure Tb also represents some notions in our developed theory, i.e., Am=(1.0] with the distribution projected onto the sub-space [1, 0] and similar notions for another sub-space. Additionally, each sub-space index m associates with two equivalent hypotheses fm. on Am and hm on the sub-space m.
Pm=]11,] includes the data samples whose latent representations indexed by
The following theorems explain the intuition that matching source and target domains on good sub- spaces helps reduce the latent data shift w.r.t a general unseen target domain; and offer us the training process of the sub-space indicator I, which extracts the most label informative features of latent representations z.
Theorem 2.
Given sub-space indicator T, if the loss function is upper-bounded by positive constant L, the sub-space target general loss is upper-bounder by:
1
C (fM, S
Σ exp {R (gJ PT
p
[C IAPm
(4)
IMI
MEM
where I.S. is the joint distribution of (x.y) with x Ps and y p ( (.)x),, and Apm m Egape Ilpp (-1x)-10 p.r PTu
Theorem 2 can be viewed as an extension of Theorem in the context of LASSO with multiple sub-spaces. This theorem shows an upper-bound of the sub-space target loss which consists of three terms: (i) the sub-space label shift: Apms (ii) the source sub-space hypothesis losses: L (fm, D&
and (iii) the sub-space latent data shift: R (g PTT
# Tho gf PS).
5

---

Under review as a conference paper at ICLR 2022
Since the sub-space label shift depends on the natural characteristics of domains and the source sub- space hypothesis losses are trainable, the sub-space latent data shift (i.e., the divergence between the mixture of source domains: gº # PP and target domain: g # on sub-space) is essential to lower the upper-bound in 4 As demonstrated in Figure [b, by projecting corresponding latent representations onto low-dimensional sub-spaces conducted from lahel-informative features, can reduce the latent data shift on the sub-spaces due to the compression effect when projecting to the sub-spaces. Moreover, if can appropriately choose the good sets of label-informative features for the sub-spaces, we can preserve sufficient label information on the sub-spaces for training good hypotheses fm hm with low source sub-space hypothesis loss L (f D with the aim lower the upper-bound in
The next arising question regarding how to train a qualified sub-space indicator for choosing good sets of label-informative features to conduct low-dimensional and tabel-preserving sub-spaces addressed in the following theorem.
Theorem 3. Let X is a random variable of source sample (i.e., drawn from P.) and Y is a random variable of ground-truth labels. Denote N Em'EM P. (Am'). we then have
FS (Am)
Σ MEM
L (fm, + const,
(5)
N
where C (fm, Dm) is defined based on the cross-entropy loss and ll denotes mutual information.
hint of how to train sub- sub-space space indicator I together with the sub- hypotheses fM = [fmlmEM- As suggested by Theorem 3 we minimize the source sub-space hypothesis losses L (Sm, Dm)) by
Theorem 3 gives
S (Am)
ps (Am)
min Σ Tigh MEM
L (Jm, Dmm
= min
Σ
Eog, 16th (
N
P.g.
N
ME.M
min Eng [e (h (m (z)) v)l.
F.g.h
Furthermore, Theorem t indicates that by solving the optimization problem in 6 we implicitly learn r,g to maximize the mutual information I [I (g(X)) @ (X), YY), which enables the sub- space indicator I' to preserve the label-informative features of = - (z).
3.4 LATENT SUB-SPACES LEARNING FRAMEWORK
In what follows, we present the technical details of our proposed method which solves the optimiza- tion problem in equation 6 In particular, we discuss the way to formulate and learn the sub-space indicator I, the feature extractor X + 2, and the classifier h Z + Ac, in which I and (g. h) are updated alternatively.
3.4. SUB-SPACE INDICATOR
We employ a probabilistic sub-space indicator I' (z) E J0. 1]" in which (2) [0, represents the probability that 2d is a label-informative feature. Additionally, in our implementation, we con- sider a 2D tensor 2 E D' x whereas each za E D' known as an attribute consists of D' features which are simultaneously selected unselected the dimensions of the projection sub-space. The probabilistic sub-space indicator is formulated as I' (2) plement-wise product r (z) O is defined as zdiag (T (z)), wherein we multiply each element in (2) column of z to select or unselect the group of features this column. Consequently. (2) aa (za) is computed based solely on the group of D' features in the attribute za itself rather than full latent z. That means attribute-based Ia depends on attributes which are shared across domains instead of domains, hence, becomes more independent from domain information. Moreover, by Theorem 3 model can still learn meaningful attributes as long as the performance on source domains is guaranteed. In unseen domains, the target of sub-space indicator I' is to detect label-informative attributes which are learned in source domains instead of identifying all lahel- informative attributes. For further discussion about attribute-based Ta please refer Appendix B. Finally, we minimize the following loss for updating I::
[Tis (2)), ..., .'.,,'p I'D (æp)] [0, 1]" The
L, (T]g, h) B (e (h (T (2) 2) )=)===()))

---

Under review as a conference paper at ICLR 2022
3.4.2 LUB-SPACE HYPOTHESES
To encourage the sub-space exploration and strengthen the sub-space hypotheses, given a data-label pair (z, v) DD, we sample mask indicators from the Berboulli distribution with parameter I'a (ad),
ie.. ma Ber (I' (za)) to gather [ma]d=1- and then project onto sub-space m m@2. Then, are updated by minimizing the following loss function:
Lu (g, hT) = (a [ (er(tB)(T(s)) [E (m + 2))l)]] |-=g(>))
(8)
where m Ber (T' (z)) means ma Ber (T' (za)) d 1,... D. For further discussion about mask sampling please refer to Appendix B.2
Finally, the pseudcode of our LASSO is summarized in Algorithm
Algorithm 1 LASSO: Latent Sub-space Orientation for DG.
1: Initialize: encoder g. classifier h, Sub-space indicator I and dataset ID:
2: for epoch = + epochs do
3: for ite in iterations do
4: Sample Mini-batchs IB = {z,, Vi) D
5: Optimize L1 in Eq. (7) w.r.t I on B.
6: Optimize L1 in Eq. (8) w.r.t. h and y on BB.
7: end for
8: end for
9: Return: The optimal: g", and l
3.4.3 INFERENCE PROCESS
At the testing phase, we use two inference strategies: threshold and ensemble strategies
Ensemble:
for each target example x, we could compute (z), then sample T masks m,1,...; My Ber (T (=)), and ensemble as hys - h (m, 2). However, for efficient computation, we approximate the ensemble prediction by: hen = h (z) @ 2).
Threshold: we employ a threshold r E [0, 1] to compute the mask. Specifically, for each target example I, we compute
(z)) then evaluate the mask [_ru(>)><]ddn
where is the indicator function, and predict h (m @
4 EXPERIMENTS
4.1 EVALUATION ON BENCHMARK DATASETS
We report the empirical results on PACS Li ct al. 2017). VLCS (Toralla & Efros 2011). l), and Office-Home (Venkateswara al. 2017 datasets using standard backbones such AlexNet, ResNet18, and ResNet50 for the feature extractor. Due to the paper space limit, we leave the details of evaluation protocol, experimental settings and implementation in Appendix
The results on VLCS using AlexNet are reported in Table | the results on Office-Home using ResNet18 and ResNet50 are reported in Table 2 and the results on PACS using three backbones reported in Table 3 respectively. Those empirical results clearly show that our LASSO provides competitive classification accuracy compared to the baselines. Especially, the average results on Office-Home and PACS in Table
consistently show that our proposed method LASSO outperforms the ERM baseline with large margins on both backbones ResNet18 and ResNet50. Although some other baselines also get comparative or better gains than ours with ResNer18, the gains shrink with ResNet50 since larger ResNet backbones are known to generalize better (Gulrajani & Lopezz Paz, 2021). Additionally, the threshold variant LASSO-T aims to select and retain the top most relevant and inforraifoee features (the most appropriate sub-space), while the ensemble variant LASSO-En aggregates the predictions of possible sub-space hypotheses. Therefore, the slight superior of the threshold variant to the ensemble one is possibly due to the fact that the former with appropriate can select compact and lahel-informative sets of features.

---

Under review as a conference paper at ICLR 2022
Table I: Classification Accuracy on VLCS.
Method
Backbone
PASCAL VOC
LahelMe
Caltech
Sun
Average
ERM Vapnik:
AlexNet
70.5810.00
59.72.0.00
96.2510.00
64.51:0000
MASF
[Doi cl 8 2019
AlctNet
69.14:2.000
64.90:10.00
94.78:0000
67.64:00.00
74.11
IGGnn
Cariage
2019
AlexNet
70.6210.00
60.90.1.00
96.9310.00
64.30.0.000
73.19
SEA.A
TTer
AlexNet
70.40+0..0
62.00+0..00
97.20+-0.00
66.00+0.00
74.00
LASSO
AlexNet
71.5510.21
62.42.1..22
97.33.10.22
65.15::0.22
74.15
LASSO-Fa
AlexNet
21.37+0.06
62,26+0.19
97.17+0.31
65.25-+0:26
74.01
Table 2: Classification Accuracy on Office-Home
Method
Backhone
Art
Clipart
Product
RealWorld
Average
ERM (aanik
1999
ResNet18
52.15:0.000
45 05.0000000
70.86:0.00
73.15::0.00
60.51
HGen Carmee
ResNer18
53.04+0.000
47 47.51+0.00
71.47+0.00
72.79+00000
RSC Frmaang 01 20.0
ResNet18
58.42:20.00
47 47.90=0.000
71.63:20.00
74.54:20.00
63.12
L2
20
ResNet18
60.60-10.00
50.10:10:00
7..00
77.00:10.00
65.60
LASSO-
ResNer18
59.29+00.92
50:77+0.22
73,28+0.18
74.48-+0.30
64.46
LASSO-Ea
ResNet18
58.63-20.27
0.948:20.66
72.99:0.13 15
74.46:20.30
64.39
ERM (Vapnik 1999
ResNet50
61.30:10.700 70
52.40:10..8
75.80&:0.10
76.50:100.30
66.50
MI L La di 201
ResNer50
61.50+00.00
53 13.20++0.66
75.00=
77.50=0.40
RSC
oi 2030
ResNet50
60.70::1.40 1.0
51 51.40:20.30
74.80=1.10
75. 15.10:1.3300
65.50
2019
ResNet50
60.40:10.70
52.70.111.00
75.00
76.00:1:.70
66.00
(Blanefard arar M041
ResNer50
61.50++0.70
52 40-00.60
74.90+0.40
76-80+0044
SagNer NNam el 2014
ResNer50
63.40:20.20 20
54 14.80020.00 4D
75 03,800::0444
78. 30:3000:300 30
68. 68.10
LASSO-Y
ResNet50
66.56:10.04
58.2110.199
78.6910.01
09.27:00.4
70.70
LASSO-Fa
ResNer50
67.07-0.388
13+03-
78.27+00.0
79. 79.19:#0:41
20.66
Table 3: Classification Accuracy on PACS.
Method
Backbone
Photo
Art-painting
Cartoon
Sketch
Average
BRM Vapnik
1999
AlexNet
68.47:0063
67.21::0.7
66. 66.12:0.51
55.32:+0.44
69.28
MLDO La
2018a
AlexNet
85.0010.000
66:2310.00
66.88-0.00
58.9611.000
70.01
MASF Deu a 2019
AlcxNet
90.6810 12
0.335
72.46::0.99
67.35:10.1 12
75.21
JiGen
2019
AlexNet
89.0010.00
67.63100000
71.71:0.000
65.18:10.00
73.38
MetaR
70.35-0.63
72.62
DMG Chanopoonyy vy
2020
AlexNer
87.3111::00
65055000
69.88-0000
71.42:00.00
73.32
LASSO
AlexNer
89455D005
69. 64
70.1-0025
69.01+00.54
74,68
LASSO-Ea
AlexNer
89.1240.06
69. 19:1994..8
70.29-0.21
68.92+00.77
74.43
ERM [Vagnik 1999
ResNe118
95.81110.27
77.861: 1.29
76.91=0.61
75.43:10.37
81.51
MLDO Lirel ml 201 Ka
Re
94 0403000000
79. 00.
77 0..30:00000
71. 71.50:20.00
80.70
MASF
RedNe118
91.9910.09
$0.2910.118
07.17=0.08
71.69110.22
$1.04
ResNed18
96.05+00.00
79.4240.0
75.25=0..0
71.35-+0.00
MetaRax
[o 2013
ResNec18
05.5010.24
83.70.0.19
0..20-0.31
70.30:10.28
$1.70
DMG Chalemnannyy
2020
ReaNe188
93 93.35+0.000
76.90-0000
ML38 4.000
75. 21.00000.00
81.46
1.2A-G1
ResNec18
96.20:10.00
03.30=:0.00
08.20=0.00
73.60:00.00
82.80
RSC
(Humit k0Ai
ResNer18
03.52.20.00
78 00
80=00000
79.78:00.00
82.65
SPA-X
2114
ResNer18
93.90+0.000
81:20++0.00
77 77.80-0000
73.70+00.00
81. 70
MaschDDr (Mazssonn
2021
ReaNe118
95.9310.000
79.77.0.00
$0.03:0.00
17.1110:00
83.21
LASSO-r
ResNec18
94.7610.03
82.1710.60
75.37=0.55
07.34
LASSO-Ea
ResNet18
91.7610.03
82.02:0.755
07.65=0.76
77.0710:42
82.87
ERM Vapniik 1999
ReaNer50
97.2+0.30
84.7=0.40
8088=0.60
79. 3-1-3-111
$5.50
MLD
RedNec50
97.440.30
15
10.1
76.6
$4.90
MetaRe (aaline
ResNer50
97.6+0031
87.2+0013
79.2=00.7
DMG
2000
ResNe150
95.0=0.00
82.9=0.00
$0.5=0.00
72.3++0.00
82.67
RSC Fiiing P07-0
ReaNe100
97.62:0.34 30
85
79.7=1180
78.2:::.20
85.20
(Sigonir 2119
ReaNec50
96.7+-0
83.5+0.0
79.1-0060
74.3+-2.00
MTL
Blandiat Fol P01
ResNa50
96.4-10.00
87.5
77.1=0.50
77 77.3:::1.80
84.60
SagNer Iuasee 20463
ReaNec50
97. 07.11:0.10
$7.4:1000
$0.7=0060
$0.01:0.4
$6.30
MatchDC (1taar 2021
ReaNet50
97.9410.000
85.611000
82.12=0..00
78.76110.00
86.11
LASSO-r
ResNec50
96.62440.56
87.81:0.69
82.33-10.20
LASSO-Ea
ResNec50
96.9311..55
87.23.0.57
$2.55 1.09
81.95.1..77
87.15
4.2 ABLATION STUDY
Sub-space representation: To better understand the benefits of learning latent sub-spaces, we vi- sualize the distribution of the learned features to analyze the latent space generated by ERM and LASSO-T using t-SNE (van der Maaten & Hinton 2008) in Figure 2 Furthermore, we decide to choose "sketch" which contains only colorless images as the target domain. This domain can be considered as the most distant domain from the others and hence results in the largest source-target divergence. Figure [2a] shows feature distributions samples are colored accordingly to domain label, with source domains are in red, green and back, while target domain is in blue. As it can be clearly seen, ERM training results in representations with high cross-domain separation for both target and source domains. On the other hands, the do- main representations on a sub-latent space of LASSO is more homogeneously mixed, since samples from different domains are hardly distinguished. Figure 2h further shows that features from differ-
different domains on latent space, in which
8

---

Under review as a conference paper at ICLR 2022
ERM
LASSO-T
LASSO
(a) Latent visualization of all four domains, differ- ent colors represent different domains.
(b) Latent visualization of target domain only, dif- ferent colors represent different classes.
Figure 2:
The t-SNE feature visualization of two methods ERM and LASSO-7 for experiment on PACS dataset with "Sketch" (represented by blue in 2a as target domain using ResNet50.
classes are more compact and clearly separated on LASSO's sub-latent space than on full single latent space. These results indicate that feature distributions are better aligned on latent sub-space with our proposed LASSO approach.
Additionally, we further verify the data shift reduction on sub-spaces by measuring the divergence between the source and target distributions ERM and LASSO-r via Jensen-Shame Shannon (JS) diver- gence ss We employ a domain discriminator ha which distinguishes the source domain:
C(ha) Ex-pg [log(ha
+ Em-PT [log(1 ha(G(z))))
where is the feature extractor, which is for ERM and gf for LASSO-T. is well-known that we search ha in a family with infinite capacity then Ris (G TP + (TT) max,a 2(h,)+2log
log
2
Goodfellow et al. 2014). The empirical results R = 0.247 and RJ
0.164
show that the latent distribution divergence between sources and target domains is decreased on sub-spaces. This observation once again provides supporting evidences for our approach.
Effect of 7: we present the performance on vali- dation set (Val) of source domains and target do- main with different thresholds to analyze its ef- fect along with the average number selected fea- tures. The Table 4 demonstrates that by choosing reasonable т, the model performance can im- proved since latent is mapped to appropriate sub- space (i.e., sub-space being compact to reduce the data shift while preserving sufficient informative features for label prediction). Alternatively, in the case of large т, the performance can be dropped since very few features remain. It also can be seen that with 0.6 and F 0.7, the model utilizes
Table 4: Average number of selected features (SF) based on t and its performance (a trial on PACS dataset with "sketch" as target domain using ResNet50).
val-SF
Val
target-SF
Target
0.5
2048
97.24
2048
82.86
0.6
64.8
97.63
59.9
83.38
0.7
20.9
96.75
15.4
83.63
0.8
7.8
84.23
3.4
79.30
0.9
2.7
61.52
1.4
45.58
only a small number of features (i.e., 59.9 and 14.4 on target domain respectively) but nonethe- less still yields good results. This indicates that sub-indicator I' is able to identify essential label- informative features.
5 CONCLUSION
In this paper, we first theoretically analyze the upper bound of target loss on latent space in domain generalization settings and the main factors that affect the performance. Then we point out that learning appropriate latent sup-space can derive better generalization performance for domain generalization task. Motivated by this, we have proposed a novel LASSO framework which can: (I) leamn diverse latent sub-spaces and corresponding individual hypotheses and (2) project source and target examples onto appropriate sub-spaces preserving crucial label-informative features for label prediction, while reducing the latent data shift due to the sub-spaces compactness compared to the entire latent space. Experiments on benchmark datasets verify that LASSO achieves competitive and even otate-of-the-art performances.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regularizztion. In Advances in Neural Information Processing Systems, pp. 998-1008, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning, 79(1-2):151-175 2010.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in neural information processing systems, pp-
2178-2186, 2011.
Gilles Blanchard, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. J. Mach. Learn. Res., 22:2-1, 2021.
Fabio M Carlucci, Antonio D I'nnnceete, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2229-2238, 2019.
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference on Computer Vision, pp. 301-318. Springer, 2020.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain general- ization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, pp. 6450-6461, 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The Journal of Machine Learning Research, 17(1):2096-200 2030, 2016.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generaliza- tion for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV, 2020.
Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pp. 322-348. PMLR, 2020.
F. D. Johansson, D. Sontag, and R. Ranganath. Support and invertibility in domain-invarian repre- sentations. In Proceedings of Machine Learning Research, volume 89, pp. 527-536, 2019.
Da L., Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Da Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 201 8a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400-5409, 2018b.
10

---

Under review as a conference paper at ICLR 2022
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Confer- on Computer Vision, pp.
2021.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, pp. 7313-7324, PMLR, 2021.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3433 2009.
Saeid Motiian, Mareo Piccirilli, Donald Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5715-5725, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Hyeonseob Nam, HyunJae Lee, Jongehan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8690-8699, 2021.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specific low-rank decomposition. In International Conference on Machine Learning. pp. 7728-7738. PMLR, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: 191 1.08731, 2019.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In Computer Rision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pp. 68 83. Springer, 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint 1rXiv: 10745, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):129-1158, 2014.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, 2008.
Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural net- works, 10(5):888-99)) 999, 1999.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Haohan Wang, Zexue He, Zachary Lipton, and Eric Xing. Learning robust representations by projecting superficial statistics out. arXiv preprint arXiv:1903.06256 2019.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In Advances Neural Information Processing Systems, pp- 585-596, 2017.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. International Conference on Machine Learning, pp- 7523-7532. PMLR, 2019.
11

---

Under review as a conference paper at ICLR 2022
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generaliza- tion via entropy regularization. Advances in Neural Information Processing Systems, 33, 2020.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European Conference Computer Vision, pp. 561-578. Springer, 2020a.
Kaiyang Zhou, Yongxin Yang, Timothy M Hospedales, and Tao Xiang, Deep domain-adversarial image generation for domain generalisation. In AAAI, pp. 13025-13032, 2020b.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
A THEORETICAL DEVELOPMENT
In this Section, we present all proofs relevant to theory developed our paper. We begin with crucial proposition for our theory development.
Proposition A.1. Let f 2' + VA where ʃ = h o with + Z and h 2 + Va.
i) We have
((h()) (v
de = Σ (() (z) ( (2)
where p" (v) 2) Lo b (())))
1 a p(z)dz for any g (X) and pe (z) Sg- P (2) de is the
density of the push-forward distribution pa 0
ii) We have
Σ
f 1o 2 12. (v 2)) p (2)
10 (yl x) 1² (v (x)
where p (z) ( (x) ( dr is the density of the push-forward distribution P Iw and
pig (y 2) J J p² (x)))) and 22 (y z) Sa-i( n² (wl=))(=))d= for any E g (X).
lca) p(a)dz
Jo 1co) p(z)dr
Proof. i) We first prove for any E [C] that
/ [(h((z) v2) (y 11)2)p) (z) dz = f ((F()) (x)
12

---

Under review as a conference paper at ICLR 2022
In fact, we have
ʃ ((((())))) v2) (2))) (z) dz
1 8 (1( [g (c)) p(u) 2)) (z) dx
p (x) drdz
1g P=) de
(z), 9) / 9-)
p (ula)) x) (x) dods
e(h(2), z)p (z) drdz
1(2)
(s) (1) (z) 4) pl p) (x) dæds
Jz
11
LEED (x) v) x) (x) dzdr J
( { 1 ( p dzdr
xx
/
())
/ (f) v) p (x) dz.
where note that condition the indicator function which returns if condition is true and 0 other- wise, the equality in (1) due to Fubini theorem.
Finally, we reach
Σ S ((( (u) 2)p (2)
Σ / y=
((0(0)))))(()))ppp) dr.
ii) We then prove that
f 1p 1 (y (2)2) - p22- (u 2) p² (z) 10 (2(1)))) pp- (v (z) vy e [C].
We derive
1. (2)2) 2) 22. (y 2) pa (z)
/
[g 1()) P. (y) x)) (x) de
[g D² (y x) (x)) dx
/ (())
p (x) drdz
1 (x)
I (x)
(2) dr
p² ()12))(())
---1()) p² 1w12)p(0)) dr dz
(01 x)] (z) dx dz
/ 1(x)
<
(2(12)) (v1l x)) (z) drds
(2) 1 (v) x) p² (v) x (x) drdz
2.. .
11
Jz 1==g(()
1p (y x) - p² (u) x)) (x) dzdr
/ 10 ())2)) (2 x)) (z) dr.
where in the derivation of Fubini theorem again.
13

---

Under review as a conference paper at ICLR 2022
Theorem 4. (Theorem 1 in the main paper) If the loss function is upper-bounded by positive constant L, the target general loss is upper-bounded by:
C(f.D") S exp { R (f+F", {*p L (f, + L max Bug [p (y [x][
(10)
where I# S and 9# PT are pushed-forward distributionss induced by applying
Ps and PT, R
is the o-divergence (o > 1), and Aps (y x) == [Pg (y cl x)) - (y = e x) c= the label shift between the labeling assignment mechanisms of an individual source domain and target domain on the input space.
Ie= represents
Proof. Let ID be the joint distribution including the pairs (y, z) where Cal (T), Ph 8.9 and y ~ Pk (y 2). The density p (y, 2) this distribution is as follows:
(z),
K
K
p (g. 2)
8.9 T&Pk (2)
S.g (1
Σ
TaPk
S.9
2).
Let LD be the joint distribution including the pairs (y, 2) where ~ pT (2) and y pT (y 22)
The density pT 2) of this distribution as follows:
(y, (2) T (1)
Eventually, we denote D as the joint distribution including the pairs (y, 2) where 2 (z) =
Σk_ TaPk 5.9
(z) and y ~ a (y 2). The density ph. (y, of this distribution is as follows:
K
r (v.2) p (2) (8) 12 = Σ
8.9 TAPK
(T's (y 1).
1=1
With the above equipment, we have
/ ((((z) ,1) v (z.1) v) dzdy
C Dh,
C (h, + ((h h. 1. (2,2)- - - (z,y) v) dady
S ( + (z), 1p n.g (2.1) - - (z, y) Idady
S C(h,DS + /
11 (z. 3)
8.0 (2. v) Idady
K
Ch, + L f
Σ
TAPE
Sia (z)
Ta (y 2) -P
5.9 (y 2) Idzdy
C
K
Th /
= ( +
1p
(y 2) - (y 2) lp
8.9 (z) dz.
14

---

Under review as a conference paper at ICLR 2022
We further note that
/
C ( =
E (h(2)) ph 5.9 (v,2)
y=1
c
* / ((M())) 8.9 (y =) (z) dz
K
((((3)))))
Pk 8:9 (y 12) pk (2) dz
T
y=1
K
8 ( (x) = C (f)
y=1
where we have (1)
by using Proposition
A.I (i).
C
K
1 ( 1)) Px
8.9 (y 2) 1p%
S.9 (z) dz
Tk
K
p T (y 2) Pk
3. (y 2) Ip
S.9 (z) dz
TK
y=1
K
T (y (1) 2 wv 1x) lo (x) dr
TT
y=1
K
C
p0 (y x) 2 wv | x)) lo (x) dr
y=1
K
Tg
Ty
[|| Ape (y x)]1 S max Egg Ap (y xx)l
(2) where we have by using Proposition A.1 (ii) Therefore, we obtain
L (h, (h.0) < C (J) + max Egg pp (v x)]l
(11)
Finally, we manipulate L (h, DT as
/
C (h,
E (1)))) ())))))) T.o (z. y) dzdy
yxz
p² (2,1)
1 من ue (z, v) 3-
e e(h (2), 19) dzdy-
yxz pho (z.y)
The Holder inequality gives
TT (z. LImE phonl (2. - dzdy
U phe (z,y) ( -= dzdy
L (b, <
Referring to the definition of the Rényi divergence and note that e(h (2),y) S L, we obtain
L (1 & [exp { R (D L [A.]] 11 1.d_
(12)
15

---

Under review as a conference paper at ICLR 2022
We further derive
1 1
p Th (2.3)
a-1
Ra (T.
a a- log
T (2, v) dzdy
ph (2.0)
1
log
/
1 (=) pf (y 2)
a-1
1 (2. dzdy
a
p (2) T. (y
2)
1
pT (z)
log
T (z, v) dzdy
a
D (z)
1
log
/
pTa (z)
Σ p F (2, v) dz
a 1
Pr 5.9 (z)
1 log
pT (z)
DT: (x) dz
1
Pr 5.9 (=)
R (g pd (1gx
(13)
Therefore, combining 12) and 13 reach the following incquality:
L (h, T9 S (exp { TRth (g* P" 11* p & (h1 [DMss]] L
Subsequently, by noting that
/
L (h, D. = Σ1
(y) p (=)
y=1
!
((f(z) (11=)p² (x) dx = C (f.D"),
==1
reach
Cf. < exp {
R (g+ P 19xp))) (h,
Finally, referring to the inequality in we reach the conclusion.
We would like to highlight that our Theorem T despite developing in a general context of DG is also novel due to two following reasons: (i) it proposes the target loss upper-bound in general setting of multi-class classification with a sufficiently general loss, which can be viewed as non-trivial generalization of existing works in DA and DG (Mansour al., 2009 Ben-David al., 2010 Zhao et al.) 2019) and (ii) our bound is interweaving both input and latent spaces, which is novel and appropriate for theoretical analyses in deep learning, while the bounds in previous works only involve input space.
Remark:
Theorem 5. (Theorem the main paper) Given deterministic sub-space indicator I, if the loss function & is upper-bounded by a positive constant L, the sub-space target general loss is upper- bounded by:
C (JM.D"") <
Σ exp { (o PT
gf # p
--1 [f D2x) + LAPm]] (14)
M
mEM
where DS is the joint distribution of (x, v) with x ~ ss and ~ phd (. ()),), and Apm =
Expps [lpp (.1x)
-ppppp p. (:1 [))]]]] sub-space indices m.
Proof. Given sub-space index e M, by noting that fm (z) h (g"(x)) (x)) with PS over Ams using the same proof for a single space in Theorem 4 we obtain
L (Jm, DD?') 1 exp ( R (g
PT m _f ) [ (Jm, man) + T.Apm
16

---

Under review as a conference paper at ICLR 2022
Finally, taking average
M, we reach
(JM, v
L
Σ MEM
# TT exp { R (g"
m" # p)))
a-1 [ (D) + I.Pm
M
Theorem 6. (Theorem in the main paper) Let X is random variable of source sam- ple (i.e., drawn from P and Y is a random variable of ground-truth labels. Denote N Em'EM pa (Am'), ) then have
P (Am) E
((T (0())
L (Jm: + const,
(15)
N
MEM
where the loss L. (sms Dmm) is defined based on the cross-entropy loss and denotes the mutual information.
Proof. Denote T = [g O (X). we have
I(TY)) = / p(l,y) log
p (t..v)
dldy
(()) p
/ p(t.u) log
( !) dtdy
p()
/ p(t.v) logp (y t) d + (Y)
f p(l,z) logh (y 1) P (y t) didy + (Y)
h 1)
/ p(t, v) log h (t.v) v) dtdy + D (p t) |/ t)) + -HO (Y)
z / p(t, y) log h (t. 8) dtdy + const,
where H specifies the entropy, DRLis Kullback-Leibler (KL) divergence, (y ()) h (1,
h (T (g (x)) @ g (z) v) for any h: Z + Va, and h (t, y) returns the y-th element of (t).
We further derive
/ p(t, y) log h (t, v) dtdy
Exto (p = (t. i)]
E - Tag (x())
Σ Eng py== T (x)) g (x)) log (T (g ( (z)
Note that we have according to our definitions: psS. = EmEm
because I' (g (x)) Og (x)
pushes forward X ~ (²) to T ~ (t). Morcover,
pas m. we hence obtain
N
/ p(l, v) log h (1. y) didy
ps (Am)
Eps b ( (mg
N
IPS (Am)
Σ Epg [- (y = x)) log fm (z. i)]
N
MEM
SSS (Am)
Σ MEM
L (Jm; D&))
N
17

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL EXPERIMENTS
B. EFFECT OF THE ITTRIBUTE-BASED INDICATOR I'a
We consider a feature vector [Z1.,.,,: DD'xD, whereas each 2a E DR known as an attribute and model sub-space indicator as I' (z) = [T, (z1). I'd (ZD)] E [0. I'a (z) = Ta (za) is computed based solely on the group of D' features the attribute 2a itself rather than full latent z. Consequently, attribute-basee Ta depends on attributes which are shared across domains, hence, becomes more independent domain. Moreover, by Theorem 3, model can learn meaningful attributes long as the performance source domains is guaranteed. In unseen domains, the target of sub-space indicator
to detect label-informative attributes which is learned in source domains instead of identifying all label-informative attributes.
More specifically, in training phase, since attributes za come from different source domains, corresponding Fa plays domain invariant hypothesis which output {0, 1} identifying label- informative attributes across source domains. unseen domain, I'a (zd) only predict 2d is label- informative attribute which are learned in source domains not.
Table 5: Performance LASSO-7 with different Fa settings.
Method
Backhone
Photo
Art Aat-ptinting
Cartoon
Sketch
Average
LASSO-7 Da(=)
ResNet50
97.0110.16
87.30=0.52 87.81
81.9110.46
80.02:10.45
86.56
LASSO-r Ta(sa)
ResNet50
96.6240.56
82.9910.71
82.33+0.200
87.43
In order to analyze the benefit of attribute-based Ia examine the performance the setting of [a(z) which uses full latent as input for la and I' l'a(za) which uses only attribute zd as input for Ta As can be seen from Table both [=a(za) and [=) achieve competitive performances on target domain:
"Photo", "Art- "Art-painting" and "cartoon" domain but I' l'a(aa) setting outperforms on "sketch" domain. Remind that using "sketch" as target domain leads the largest source-target divergence. The results of "sketch" domain demonstrate that ra(=) becomes less efficient when source-target divergence. In contrast, T2()) shows its better generalization as not be interfered by other noisy attributes.
B.2 MASK SAMPLING FOR MODEL TRAINING
In training phase, sampling spaces. The motivation is that target domains might have or share small sub-set of learned features in source domains e.g. in PACS dataset, many label-informative features can be learned in rich in- formation domain such as "Photo" domain while "Sketch" domain only has edge/shape information for classification. Therefore, in training phase, would like to train hypotheses different sub-set of learned features (i.c., sub-spaces of sub-space) which would strengthen sub-space hypotheses on arbitrary target domains..
Ber (T (2)) would help hypotheses explore more useful sub-
Table 6: Classification Accuracy on PACS with ResNet50
Training
Backhone
Photo
Art-painting
Cartoon
Sketch
Average
Deterministic-r
ResNet50
95.99=0.42
1.70
80.09:1.54
79.01=4.44
85,34
Deterministic-En
ResNet50
93.05=0.03
1.018
1..08:1.62
77.73=2.65
84.26
LASSO-T
ResNet50
96.62-0.56
87.81 0.66
82.99+00771
82.33-0.20
87.43
LASSO-En
ResNet50
96.93=0.65
87.23=0.557
1..55=1.09
81.95=0.37
87.15
To demonstrate the benefit of sampling, we conduct additional experiments using deterministic man- which uses threshold determine mask in model training. The results in Table 6 indicate sampling significantly enhance performance.
B.3 DISCUSSION AND COMPARISON WITH RANDOM DROPOUT
We would like to report additional experimental results in comparison with "Random Dropout" Baseline (Srivastava al. 2014) for different dropping rate follows:
18

---

Under review as a conference paper at ICLR 2022
Table 7: Classification Accuracy PACS with ResNet18
Feature Dropping
Backbone
Art paintng
Sketch
Average
0%
ResNet18
96.17.10.27
78.42.11.29
76.73.10.64
75.4010.37
81.68
10%
ResNet18
96.29++0.66
81.01
76.98+00.67
75.60+0.81
82.47
30%
ResNet18
96.41+0.60
79.7340.69
77.68-0.28
75.22+1.07
82.26
50%
ReaNet18
96.59:0.144
81.20:10.02
77.92=20.44
75.98:0.17
82.92
70%
ResNet18
96.29+t014
79.93+0.72
78.26:+0.67
74.89+0.09
82.34
LASSO-1
ReaNet18
94,76:10.03
82.17:00.00
78.37=00.55
77.3420.37
83.16
LASSO-Ea
ResNei18
94.7610.03
82.02.10.75
77.07-10442
Table 8: Classification Accuracy PACS with ResNet50
Feature Dropping Rate
Backbone
Photo
Art-painting
Cartoon
Sketch
Average
0%
RexNet50
97.20110.30
84.7011.40
80.80:10600
79.301
85,50
86,9610.37
30%
ResNet50
98.08++0.32
87.26++0.63
80.11+1.100
81.31+0.800
86.69
50%
ResNet50
97.60:0.04
87.16+0.11
80.46+0.24
80.80+0.77
86.51
70%
ResNet50
97.08-+0.47
86,33-40.73
81.78++0.98
80,57++1001
86.44
LASSO-7
ResNer50
96.624:0.56
87.8140.69
82.99=0071
82.33++0.20
87.43
LASSO-Ea
ResNet50
96.931:0.65
87.23:10.57
82.55:11.09
81.9510.37
87.15
The results in Table 7 and Table & indicate that ERM with appropriate dropping rate (i.e., 50% for ResNet18 and 30% for ResNet50) is able to achieve comparative performance to most current baselines.
In fact, "Dropout" has quite similar effect to the representation as LASSO, which increases the diversity of latent representations. However, LASSO goes step further compared to "Dropout". Specifically, representation used as it is during inference time, without making the use of infor- mation in the target data, e.g., using the full network for inference, or average over a set of randomly "Dropout" networks (MC dropout). On the contrary, the sub-space indicator in LASSO additionally takes into account the information from the target domain select the suitable masked network which is likely to generalize better to this particular target domain. This is the motivation behind the selection of appropriate sub-space.
C EXPERIMENTAL SETTINGS
C.I DATASET DETAILS
To evaluate the effectiveness of the proposed method, we utilize three datasets: PACS (Li et al. 2017]. VLCS Torralba & Efros 2011), and office-Home (Venkateswara al. 2017). which are the common DG benchmarks with multi-source domains.
PACS (Li et al. 2017): 9991 images of seven classes total, over four do- mains:Art.pinnting (A), Cartoon (C), Sketches (S), and Photo (P).
VLCCS (Torralba & Efros 2011): five classes over four domains with total of 10729 samples. The domains are defined by four image origins, i.e., images were taken from the PASCAL VOC 2007 (V), LabelMe (I.), Caltech (C) and Sun (S) datasets.
Office-Home (Venkateswara et al. 2017): 65 categories of 15500 daily objects from domains: Art, Clipart, Product (vendor website with white-background) and Real-World (real-object collected from regular cameras).
C.2 EVALUATION PROTOCOL
For PACS and VLCS, we follow the setting of (Li et al. (017)) as these two datasets provide specific "train" and "validation" splits for each domain ensure a fair comparison. We use the validation subsets from all training domains to create an overall validation set. On the target domain, we evaluate the entire dataset. Office-Home also follows the jet al., 2017). addition, the generalization our method is based on the accuracy of the sub- space indicator r. Therefore, we choose the model when the sub-space indicator I' converged. In
protocol in (Li)
19

---

Under review as a conference paper at ICLR 2022
particular, train all models for the same fixed number of steps and consider only the final check- point. Finally, we report the average performance with error bars over 5 runs for all experimental settings and compare two variants of our proposed method LASSO-7 (the threshold variant) and LASSO-En (the ensemble variant) with the recent domain generalization baselines.
C.3 BASELINES
We compare with the following recent domain generalization baselines (with reproducible public codes):
ERM Vapnik miniiiisss the sum of errors across domains and examples. For our
experiments, we employ the implementation from (Gulrajani & Lopez-Par 202I), a strong baseline that can achieve competitive accuracies on DG benchmarks.
MLDG (Li ct al.
8a); Learning to generalize: Meta-lcarning for domain generalization.
MetaReg (Balaji et al. 2018): Metareg: Towards domain generalization using meta-
regularization.
MASF (Dou al.
2019): Domain generalization via model-agnostic learning of semantic
features.
JiGen (Carlucci et al.
2019) : Domain Generalization by Solving Jigsaw Puzzles.
DMG (Chattopadhyay al. 2020): Learning to balance specificity and invariance for in
and out of domain generalization.
L22--T (Zhou et al.
2020a): Learning to Generate Novel Domains for Domain General-
ization.
RSC (Huang et al.
)
Self-challenging improves cross-domain generalization.
Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization.
GroupDRO (Sagawa et
2019):
SFA-A (L al.
2021):
A Simple Feature Augmentation for Domain Generalization.
MTL Blanchard et al,
2021;
Domain Generalization by Marginal Transfer Learning.
SagNets (Nam 2021):
Reducing Domain Gap by Reducing Style Bias.
MatchDG (Mahajan et
2021): Domain generalization using causal matching.
C.4 IMPLEMENTATION DETAILS
We detail our algorithm's 's implementation sub-space Indicator P: given latent vector g(z) DxDD' (i.e. on each attribute: r'(-) = (Ta(-)) where each I'a is neural network. In current work, we model Pa as a Full-Con Connected (FC) layer [D' x 1] prior sigmoid function a and Ia takes zg as input. (Code for experiments is available in Supplementary Material.)
[21, 2D]124 RR'),, we model 1'(.) as a composition of independent indicator
The following are adapted versions on backbones, in which the last FC layer is used as the classifi- cation network. The summary of the additional parameter is presented in Table
AlexNet: For AlexNet backbone, the extracted feature dimensions are E R-4096 In current imple- mentation, we partition z into 256 features E IR 16 as input for la-
For ResNet18 & ResNet50 backbones, instead of partitioning the feature tensor, we utilize direcly feature tensor before avepooling layer. In particular, the extracted feature are z E 512x7777 for ResNet18 and E R2048x777 for ResNet50. We use zd E 7=7 as input for la.
ResNet18 & ResNet50:
Optimization: We adopt the SDG optimizer for training with: the model learning-rate (Irs and sub-space indicator I' learning-rate (Irr) for different bacbones and datasets presented in Table I0 mini-batch size of 64 for 100 epochs. The learning rate (Ira,g and Ity) decayed by 0.1 after every 80% epochs. The values of parem--rrsseterss Ira,g> Irg and chosen based on the performance on source validation set. Additionally, in order to the sub-space indicator quickly adapt to features induced by h. g, Irr is set much larger than Irn.g.
20

---

Under review as a conference paper at ICLR 2022
Table 9: Additional parameters summary
Backbones
Partitions
Original
Additional
AlexNet
256
61M
4352 (0.007%)
ResNet18
512
11M
25600 (0.230%)
ResNet50
2048
23M
0.1M (0.440%)
Table 10: Hyperparameters.
Dataset
Backbone
In
Irr
VLCS
AlexNet
1.0c
1.0e -2
Office-Homm Home
ResNet18
6.5e -
1.0e
ResNet50
6.5c
1.0e-0
PACS
AlexNet
1.0c
1.0e
ResNet18
1.0e
1.0e
ResNet50
1.0e-2
1.0e-0
The number of sampling times in Eq. 8 also hyper- hyyper-parameter our LASSO. When trying with several possible values of this haper-parameter, we observed that (i) when increasing the number of sampling times, the performance slightly improves but the training time more significantly creases, and (ii) when setting this hyper-parametts to 1, the performance consistently is satisfactory for all experimental datasets. Therefore, all experiments, we sample once per mini-batch for com- putation efficiency. Additionally, inference time, conveniently choose
0.6 for ResNet18 ResNet50 and v = 0.5 for AlexNet on all datasets. However, from the results Table 4 it is reasonable to choose based the performance of the validation data of source domains although sometimes the best performance on validation data does not reflect the best performance on target domain.
Data augmentation: We follow the image augmentation protocol introduced Gulrajani & Lopez-Paz 2021) which is increasingly standard in state-of-the-ar DG work: crops of random size and aspect ratio, resizing to 224 x 224 pixels, random horizontal flips, random color jitter, grayscal- ing the image with 10% probability, and normalization using the ImageNet channel statistics.
Finally, we use system of GPU NVIDIA Tesla V100 with dual CPUs Intel Xeon E5-2698 v4 to conduct experiments.
21


Reviews:
[{"review_id": "wswCemsseYE", "reviewer": "Reviewer_aksw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the abstract, the author claimed that exploring diverse latent sub-spaces for individual hypotheses learning is superior compared with learning single hypothesis shared among domains, however, it is not clear to me what is the significance of doing so.  The authors try to give an explanation by delivering an upper bound target general loss, which to me is quite questionable since the target domain is not available during the training phase. Simply using the mixture sources domain may not reflect the true distribution of the target one.\nBesides there are some other weaknesses as follows:\n1. Is Theorem 2 strictly lower than Theorem 1? I can only find the author’s explanation rather than strict prove. This part is essential for this paper, besides, this is closely related to my given score. \n2. In Section3.4.2, the author proposes to use sampling for model training. I wonder about the performance of doing this in a deterministic manner, just like using threshold at the testing phase.\n3. Experiments are somewhat lacking, 1) challenging datasets (e.g., medical imaging for CV, reinforcement learning in MLDG) should be considered since the proposed method is more related with general machine learning,  2) only RSC is considered as the latest stoa baseline, which is not desired.\n\nMinor:\n4. For the experimental part, why not keep the same manner to select the validation set? It would be better to keep this unchanged. \n", "summary_of_the_paper": "This paper critically re-examines the rationality of domain-invariant based method for DG (generalize to unseen domains without re-training). It highlights that, tackling all source domains equally without taking the underlying relationship between them to learn domain-invariant features can lead to a sub-optimal solution. To deal with this, it introduces a latent space decoupling-based method called Latent Sub-Space Orientation (LASSO) to eliminate some irrelative representations for prediction. By this, the relationship between latent space and label space can be strengthened, thus the model can generalize better. The results show that LASSO (especially with threshold) can commonly lead to a marginal performance boost for popular DG benchmarks. ", "summary_of_the_review": "Though I am not an expert in this area, I have discussed my comments with my colleagues with rich experience in this area. Thus, I am still confident in my comments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "ZMi_PFcaueK", "reviewer": "Reviewer_crRo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) Authors correctly point out overly strict and sub-optimal assumption that is popular in the literature currently. \n\n2) Authors’ claims are backed both by theory and experimental evidence. \n\n3) Some benchmark papers in the literature (which don’t have an assumption of domain-invariant features) are not considered [1-2]. Both these papers don’t have an assumption of domain-invariant features and are similar in spirit as the current paper.  \n\n4) Theoretical analysis in the given paper helps to come up with a loss but not actually analyze the method. Like in [1-2], can authors comment on the learning theoretic study? How does risk bound change with the number of available domains and number of examples in the training? \n\n5) As we have a large number of domains and a large number of training examples then one should be able to have optimal error or loss.  I could not draw this conclusion from the theory that the authors provided. Can authors comment more on this? \n\n6) How were hyperparameters tuned for the baseline method? \n\n[1] Blanchard, Gilles, Gyemin Lee, and Clayton Scott. \"Generalizing from several related classification tasks to a new unlabeled sample.\" Advances in neural information processing systems 24 (2011): 2178-2186.\n\n[2] Blanchard, Gilles, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. \"Domain Generalization by Marginal Transfer Learning.\" arXiv preprint arXiv:1711.07910 (2017) J. Mach. Learn. Res. (JMLR) 22 (2021): 2-1.\n", "summary_of_the_paper": "\nAuthors point out that the assumption of existence of fixed domain-invariant features and common hypotheses learned from a set of training domains could be overly strict and sub-optimal. Authors propose a new method which doesn’t have a single hypothesis shared among domains and give theoretical analysis of the proposed method. Authors also give results on benchmark datasets. \n", "summary_of_the_review": "Paper has rigorous set of experiments and is backed by clearly explained motivation. Addition of suggested literature could further improve the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TJmADpL5Zax", "reviewer": "Reviewer_43Db", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strength\n  - The proposed method is solid and novel.\n  - The empirical results show solid improvement over baselines with a large network backbone. \n\n- Weakness\n  - Theoretical arguments are not particularly strong. For example, Theorem 1 and 2 are rather for domain adaptation when one has access to the target domain, so it is unclear how it is relevant to the domain generalization context. While formulating a bound with latent space might be new, but it does not seem to add any better understanding for domain generalization.\n  - \\Gamma is optimized based on Equation (7) and it is unclear what authors mean by \"our principle is to encourage \\Gamma_{d} to becomes more independent\". Are there any part of the optimization problem that promotes independence?\n  - Might be good to add a baseline without updating \\Gamma (i.e., adding a dropout layer).\n\n- Misc\n  - It is unclear what it means by \"but the gains shrink with ResNet50 since larger ResNet backbones are known to generalize better\" in Section 4.1.3.\n  - The trend observed in the paper on the performance w.r.t. the network size seems interesting. Might be good to add more experimental results with deeper networks to see if trend holds true.", "summary_of_the_paper": "The paper presents LASSO, a Latent Sub-Spaces Orientation to tackle domain generalization problem. The paper involves both theoretical and empirical results. \n\nTheorem 1 says the target domain loss is bounded by terms involving source domain losses, label shift, and the latent data shift (which reminds me of the theorems from the seminal work by Ben-David et al. for domain adaptation). Theorem 2 extends Theorem 1 but with latent subspaces. Theorem 3 says that for a given subspace indicator \\Gamma, the mutual information between the sub-space and label is lower bounded by the sum of negative source domain losses.\n\nThe proposed method essentially introduces the subspace-indicating binary variables in the neural network. This seems to work similarly to the dropout, but with more structure and with learned and data-dependent drop rates. Inference is also done by generating multiple subspace-indicating binary masks and ensemble them. \n\nExperimental results are provided on standard DG benchmarks. The trend seems clear that for shallow network (e.g., AlexNet, ResNet-18) the performance is not necessarily better than previous methods, but with deeper network (e.g., ResNet-50) the improvement is  somewhat significant. ", "summary_of_the_review": "While there is a question on the significance of the theoretical results, overall the paper proposed a new domain generalization algorithm that is both technically and empirically solid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KXWOXYiInCb", "reviewer": "Reviewer_gyy1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Instead of using a single consistent latent space, the authors propose to employ multiple latent spaces by capturing the label-informative features from the available source domains. Some theoretical analyses are given to support their learning scheme, and a very simple latent subspace orientation algorithm is proposed to tackle the domain generalization challenges. Experiments validate the usefulness of the proposed method on different datasets.\n\nSome concerns should be considered for rebuttal.\n1. Generally, the proposed method alternatively optimizes an indicator function and subspace learning. Although there are three theorems to support your claim, the reviewer still has concerns about the standpoint and update of the subspace indicator. Apart from these claimed theoretic analyses on the subspace indicator \\Delta, the authors should give a more intuitive explanation of why it is useful, which can help the readers understand your work, rather than lots of equations. Moreover, the reviewer is wondering if we fix these indicator updates after some iterations, what will happen for the performance because the review doubt such an update is meaningless for this task in practical systems.\n2. Why is maximizing the mutual information useful in your framework? From the reviewer's view, using other adversarial learning or metric learning also can achieve your objective.\n3. The presented performance is unsatisfactory when compared to other works published in 2021, although this paper is very simple and easy to implement. \n4. The authors should clarify why the performance of using Threshold outperforms those of using ensemble. Intuitively, using an ensembling strategy has a tendency to have better performance. Moreover, how could you set the value of \\tau? Is it a hyper-parameter? Is there any adaptive learning strategy to determine it or empirically setting?\n5. Many grammar mistakes are shown in the work, and the authors should pay attention to your presentation.\n", "summary_of_the_paper": "This paper proposes a latent subspace orientation algorithm for domain generalization, which is built on diverse latent subspaces and individual hypotheses with label-informative features. Extensive theoretic and experiments show the effectiveness of the proposed method with two variants.", "summary_of_the_review": "This paper proposes a simple latent subspace orientation algorithm for domain generalization based on some theoretical and experimental observations. The technical parts with some complex theoretical analyses are satisfied, yet the model construction and experimental parts could be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DhJ3ZLU3oIG", "reviewer": "Reviewer_H9Wv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the paper presents lots of theorems, this reviewer could not find a clear justification for the proposed method. \n\n(1) How the diversity of latent representations could reduce the latent data shift issue: \n\nTheorem 1 and the first paragraph of page 4 suggest that to achieve domain generalization one has to reduce the latent data shift, which is unfortunately not straightforward since no target data is available in the task. The second paragraph argues that learning a single domain-invariant model is not enough to resolve the issue, but such an argument is not theoretically proven. Moreover, in the third paragraph, the authors claim that modeling and utilizing diverse latent representations are the key to the success, but the connection between the diversity of multiple latent representations and the latent data shift issue is totally missing. (Further, considering the technical details in Sec. 3.4, there is no guarantee of the diversity in the proposed method.)\n\n(2) How the sub-space hypothesis could reduce the latent data shift:\n\nIn page 5, the authors argue that the compactness of the sub-space can reduce the latent domain shift, which is however not proven theoretically either.\n\n(3) How the mutual information maximization is linked to reducing the latent data shift:\n\nTheorem 3 derives a lower bound of the mutual information between ground-truth label and data projected to the sub-space (i.e., masked feature representation), and accordingly, the masking module is learned to maximize the mutual information in Eq. (6). However, there is no justification for this approach nor the connection between the mutual information and the latent data shift (or domain generalization in general).\n\n(4) How the independence of the masking weights leads to better generalization:\n\nThe argument in page 6 about the advantages of such independence is not rigorous; the reason following the argument seems indecipherable.\n\nDue to these, this reviewer feels the theorems derived in this paper are not very useful to justify the proposed method, and cannot find clear reasons for its advantages. \n\n\n[Other comments]\n- Do not understand why z is a 2D tensor and what each dimension stands for.\n- Would recommend drawing a figure to better illustrate the overall pipeline.\n- The first theorem resembles the well known theorem in literature of domain adaptation. It would be useful if their differences are well explained in the paper.\n", "summary_of_the_paper": "This paper introduces a new method for domain generalizable classification. Given a feature representation of input, the proposed method first samples binary masks from a Bernoulli distribution whose parameters are determined by the feature, projects the input feature to multiple sub-spaces by using the masks, and applying classifiers associated with each of the sub-spaces. The parameters of the Bernoulli distribution are predicted by a module that is learned together with the feature representation and classifiers. \n\nAlthough the paper tries to provide mathematical backgrounds and underlying theories of the proposed method, unfortunately, this reviewer finds no clear justification for learning and using the sub-spaces. Also, the improvement by the method seems marginal (or it even underperforms sometimes) when coupled with small backbone networks.\n", "summary_of_the_review": "This reviewer feels the theorems derived in this paper are not very useful to justify the proposed method, and cannot find clear reasons for its advantages. Also, the practical value of the method seems limited as its performance is not impressive or even inferior to previous work when it is incorporated with small networks. \n\n-- post-rebuttal --\nI greatly appreciate the kind responses and the update of the manuscript and would like to upgrade my rating accordingly, but since my major concern has not been resolved yet, I am still leaning towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for improving domain generalization based on the idea of learning different subspaces for each domain. Authors provide theoretical analysis related to their proposal and further evaluate their proposed method on a subset of DomainBed benchmark.
 
 
**Strong Points:**
 
- The paper is well-written.
 
- The proposed method is novel.
 
- Authors provide theoretical analysis in support of their proposal.
 
- The theoretical results seem to be correct.
 
- Empirical evaluation shows that the proposed method improves over baselines on a subset of datasets included in the DomainBed benchmark.
 
**Weak Points:**
 
- The complexity of the theoretical results makes it very difficult for the reader to get any intuition about the underlying mechanisms at play.
 
- The theoretical analysis is disconnected from the proposed algorithm. It is hard to see how one could end up proposing such an algorithm following the theoretical results. I suggest that authors would consider reorganizing the paper with less emphasis on the theoretical part, perhaps simplifying the theoretical results and pushing the rest to appendix.
 
- The empirical evaluation can be improved significantly. Domain generalization is a very well-established area at this point. WILDS is a carefully designed and well-known benchmark and showing improvement in that benchmark would be very convincing but unfortunately authors do not discuss or even refer to it. They instead report their results on a subset of datasets used in DomainBed benchmark. The DomainBed benchmark is less challenging than WILDS but even following DomainBed closely and reporting the 3 evaluation metrics on all 7 datasets would have been satisfying. However, authors only report the results on 3 datasets. Reporting the results on a diverse group of datasets is particularly important in the case of Domain Generalization because we know that many methods are able to show improvements on a few datasets but it is challenging to beat the baselines on a significant majority of datasets.
 
**Final Decision Rationale**: 
 
This is a borderline paper. On one hand, the proposed method is interesting and novel. On the other hand, the theoretical contributions are very limited and the empirical evaluation is not strong enough for acceptance. Given that all weak points mentioned above can be addressed, I recommend rejection and I sincerely hope that authors would strengthen their paper by addressing them before resubmitting their work.
Decision: Reject