Exploiting Domain-Specific Features to Enhance Domain Generalization
Authors: Ha Manh Bui
Toan Tran
Anh Tuan Tran
Dinh Phung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representation Learning
Domain Generalization

Abstract: Domain Generalization (DG) aims to train a model, from multiple observed source domains, in order to perform well on unseen target domains. To obtain the generalization capability, prior DG approaches have focused on extracting domain-invariant information across sources to generalize on target domains, while useful domain-specific information which strongly correlates with labels in individual domains and the generalization to target domains is usually ignored. In this paper, we propose meta-Domain Specific-Domain Invariant (mDSDI) - a novel theoretically sound framework that extends beyond the invariance view to further capture the usefulness of domain-specific information. Our key insight is to disentangle features in the latent space while jointly learning both domain-invariant and domain-specific features in a unified framework. The domain-specific representation is optimized through the meta-learning framework to adapt from source domains, targeting a robust generalization on unseen domains. We empirically show that mDSDI provides competitive results with state-of-the-art techniques in DG. A further ablation study with our generated dataset, Background-Colored-MNIST, confirms the hypothesis that domain-specific is essential, leading to better results when compared with only using domain-invariant.

Exploiting Domain-Specific Features to Enhance Domain Generalization
Manh-Ha Bui' Toan Tran'
Anh Tuan Tran' Dinh Phung
1 VinAI Research, Vietnam 2 Monash University, Australia (v. habm1, v. v.toantm3, v. anhtt152, v. v.dinhpq2)@vimaii.t
Abstract
Domain Generalization (DG) aims to train a model, from multiple observed source domains, in order perform well unseen target domains. To obtain the gen- eralization capability, prior DG approaches have focused on extracting domain- invariant information across sources generalize target domains, while useful domain-specific information which strongly correlates with labels in individual domains and the generalization to target domains is usually ignored. In this paper, we propose meta-Domain Specific-Domaa Invariant (DDSDI) a novel theoreti- cally sound framework that extends beyond the invariance view further capture the usefulness of domain-specific information. Our key insight is to disentangle features in the latent space while jointly learning both domain- invariint and domain- specific features in unified framework. The domain domain-speciftt specific representation is optimized through the meta-learning framework to adapt from source domains, targeting robust generalization on unseen domains. We empirically show that mDSDD provides competitive results with state-of-the-art techniques DG further ablation study with our generated dataset, Background-Coloree MNISS,,-MNIST confirms the hypothesis that -ppeafi is essential, leading to better results when compared with only using domain-invariant.
Introduction and Related work
Domain Generalization DG has recently become an important research topic in machine learning due its real-world applicability and its close connection the way humans generalize learn in new domain. In a DG framework, the learner is trained on multiple datasets collected under different environments without any access any data on the target domain (T). One of the most notable approaches to this problem is to learn the "domain-invarian"" features across these training datasets, with the assumption that these invariant representations are also held in unseen target domains (2) 3 5 (). While this has been shown to work well in practice, its key drawback is completely ignoring flomain-specific information that could aid the generalization performance, especially when the number of source domains increases (7).
For instance, consider the problem of classifying dog or fish images from two source domains: sketch and photo. While the sketch contains conceptual drawing of the animal, the photo includes their taken picture within a background. In this case, sketch domain-invariant is kept across domains, while domain-spenn speciiic, e.g., a dog in a house or fish in the ocean. will be discarded due only existing in the photo domain. However, this background information, when present, could lead to an improvement of the classification performance in target domains due common association between the objects of its background, and when negligent sketches are hard to distinguish. From theoretical standpoint, there has also been strong recent evidence to indicate the insufficiency of learning domain- invariant representation for successful adaptation domain adaptation problems (8.9). For example,
"Correspondence to Manh-Hi Ha Bui: <hb. bummbumianhhaa@aa....... com>.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Zhao et al. (8) has pointed out the degradation in target predictive performance if domain-invarian representations are forced while the marginal label distributions on the source and target domains are overly different.
Utilizing domain-speciff features in DG has been widely studied in recent works (e.g.. 1T0 Ding and Fu (10) introduce multiple domain-specitff networks for each domain, then use the structured low-rank constraints to align them with innaiinnnv While this encourages the better transfer of knowledge, its main problem is the requirement of too many domain-specific networks. More recently, Chattopadhyay et al. (7) proposed a masking strategy to disentangle nomain-invariant and domain-specific to further boost domain-specinn cettaaamdddmminsppeeiii representations might not be disentangled since the learning and inferring procedures are performed implicitly (i.e., without any theoretical guarantee) through mask general- ization process. That means it lacks a clear motivation as well as theoretical justifications.
learning, but its key drawback is that domain-
Regarding meta-learning related work, a typical approach involving meta-learning in DG is MLDG (1I that is based on gradient update which simulates train/test domain shift within each mini-batch, mainly to learn transferable weight representations from meta-source domains to quickly adapt the meta-target domain, and so improve generalization ability. However, their task objective adapts for all representation features which include domain-invariant, since low effectiveness because domain-invarian stable across domains, pushing to adapt those features might affect the stability of those domain-invariamt invariant, leading to lower generalization performance on the target domain.
To handle these domain-invariant shortcomings, in this paper, we propose a novel theoretically sound DGJ approach that aims to extract inhhrminformttiv domain-specific and then explicitly disentangles the domain-invarianl and domain-specifie representations in an efficient way without training multiple networks for domain-specific. Following the meta-learning idea and mitigating previous work's s drawbacks, we apply meta-learning technique specifically to exploit domain-specific quality which should need to be adapted to unseen domains from source domains. Our contributions in this work summarized as follows:
We provide a theoretical analysis based on the information bottleneck principle to point out the limitation of only learning invariant and the importance of domain-specific representation by a certainly plausible assumption.
We then develop a rigorous framework to formulate elements of iammainnivariaan/dommiin specific representations, in which our key insight is to introduce an effective meta- optimization training framework (IT) to learn domain-specinf representation from multiple training domains. Without accessing any data from unseen target domains, the meta-training procedure provides suitable mechanism to self-learn domain-specific representation. We term our approach meta--omaan Specific-Domain Invariant (DDDDI) and provide necessary theoretical verifications for it.
To demonstrate the merit of the proposed mDSDI framework, we extensively evaluate MDSDI on several state-of-the-arn DG benchmark datasets, including -olore--MNIST, Rotated-MNIST, VLCS, PACS, Office-Home, Terra Incognita, DomainNet in addition to our newly created Background-Colle -oceroound-ooored---ee MoloST for the ablation study to examine the behavior of our DDDD]
2 Methodology
2.1 Problem setting and Definitions
Let X' C CR be the sample space and y IR the label space. Denote the set of joint probability distributions on X' x y by Pxxy, and the set of probability marginal distributions on X' by Px. A domain is defined by joint distribution P(x, y) E Pxxx, and let P be measure on Pxxx, i.e., whose realizations are distributions on X' x V
Denote source domains by Sti {(x,) 11 D i i= I...,, N, where n; is the number (6) (4)
of data points in S(A), i.e., (af0) )) std p(0 y) where (() ~ P; and z p(O) in which p(( Px. In typical DG framework, a learning model which is only trained on the set of source domains [ J without any access to the (unlabeled) data points in the target

---

domain, arrives at good generalization performance on the test dataset S = =1x2] 03 13 where
(ef.uf) std pI (ar, y) and pT (xr, y) P.
First, we present the definition of domain Nomain-invarian invariant representation a latent space Z under covariate shift assumption (i.e., the conditional distribution P(y(æ)
(z) is unchanged across the source domains): Definition A feature extraction mapping Q X + 2 is said to be domain-invariant if the distribution (x()(X)) is unchanged across the source domains, i.e, Vi,j -1...... N, i have
= Pg (Q(X)), where (Q(())) = Pa(Q(X)IX ~ p == 1...., N. In this case, the corresponding latent representation Z Q(X) is then called the representation (see (3) also).
As mentioned in the example in the introduction part, the definition reveals that the extracted nomain-invariian latent Z₁ could be the conceptual drawing of the animal which is shared in both sketch and photo domains. However, when existing background information is taken by a picture such as house ocean, is crucial take these backgrounds into account because the domain domain-invariant feature extraction Q might ignore them by only existing in the photo domain. Therefore, we next introduce the definition of domain domain-specifer specific in latent space as follows:
Definition 2. A feature extraction mapping R X Z is said to be domain-specific if Vi,j I...,,N, N, * j such that p(d (R(X)) + p() pl) (R(X)). where (R(()) (x([(X)]X P()), i= i=1...,, N. In this case, given X ~ P the corresponding latent representation 2.5 R(X) is then called the domain-specific representation w.r.. the domain S(3)
Definition
2 states that for any domain and j. the distributions of the domain-specifie latent pk (R(X) and p(5) (R(X)) must be completely different. For instance, following our mentioned example, given fish images from i and that are sketch and photo domain. the mapping R(X) should extract specific information that only belongs to the domain including the shadow of the fish drawing in the sketch and ocean background information in the photo domain.
To this end, this paper aims to show that only learning domain-invariant will limit the prediction per- formance and generalization ability. Hence, we provide formal explanation for the motivation of learning domain-specifec by showing the potential drawback of only learning domain-invariance in terms of predicting class labels.
2.2
A theoretical analysis under the Information bottleneck method
Discard redundant
Domain Specific
Domain Specific x(X=
Domain Specific
Domain Specific Redundant
min ((xxiX²)
[( [
X
22
Zx
((Zx)))
Domain Invariant x(7;
Domain Invariant x
y
Relevant Domain Specific
2x:
Extract relevant into
(c)
Figure Venn diagram showing relationships between source domains represented by X X² target domain represented by X" and label Y. (a) The learning procedure of minimal and sufficient lahel-related representation in definition 3 (b) Explaining the theorem T where the conain-invarian based method provides an inferior prediction performance our proposed method that incorporates both domain-invarian I[Zr-; Y) and label- related Nomain-speciffi values made by our assumption [ (c) A case when the unseen (target) domain different x and y", while -nnvaratn information is still stable across domains, some Lomain-speciffh in souree domains become redundant information.
Given three arbitrary random variables A, B, and C, let us use I(A; B) to represent mutual information between A and B: (A; B C)) to represent conditional mutual information of A and B given C: ((A) to represent entropy of A; and H(AB) random variables A given B. For simplicity, we consider the case with two source domains 5 S² (the results with multiple source domains can be naturally extended from there). We also define two
Notations.
represent conditional entropy for
3

---

corresponding random variables X ~ p() the domain S" i= 1,2.
x that are sampled from the marginal distribution Px in
Figure illustratts all the definitions and assumptions above used for our theoretical verification (in Theorem D. In particular, in that figure, each of the four colored rectangles represents an individual entropy: (x²) for domain S is in blue, H (X²) for domain S is in red, H (X") for the target domain is in black, and (Y) for the class label is in green border rectangle.
We first show the ineffectiveness of only learning domain-invariant information when compared with incorporating Jomain-specific in source domain S (and similarly with domain S²). Our justification partly relies the following assumption about the correlation between the domain- specific representation and the class label:
Assuming that there exists domainn-spcciff (1), correlates with label in domain 5 such that T(ZZ)) Y|X²) ((X1;1(x3 E1, where & 0 is a constant.
Assumption 1. (Label-correlated domain-specifee
representation z extracted by the deterministic mapping - = =R(X+) in definition which
Assumption indicates that, for the source domain St, we can learn -.²¹ = R(X!) such that 1 ;Y(x?) is strictly positive and equals to I (Xx)x where , is the specific information that correlates with the label in the domain 81, but not in the domain 5² (12). For instance, in the example mentioned in the introduction, if domain St is "photo" while S is "sketch", the value of e should be positive because the background information such as a house, the ocean also provides information to predict whether the object is dog fish without considering its conceptual drawing. This assumption is particularly valid and practically plausible and is demonstrated by several examples observed in our experiments. For instance, for the DomainNet benchmark dataset, in the real-world domain, many bed pictures contain a bed in the room or bike pictures that have bicycles parked on the street. Other examples are in PACS such as dogs in the yard or guitars lying on a table in photo and art domains. These examples are strongly related to assumption I in which specific information correlates with labels in particular domain.
We next present supervised learning frameworks under the umbrella of the information theory [3) and the information bottleneck method (5) that generalizes minimal sufficient statistics to the minimal (i.e., less complexity) and sufficient (i.e, better fidelity) representations. The learning process of such representations is equivalent to solving the following objectives:
Definition 3. (Minimal and sufficient representations with label (14)). Let Zx1 = G(X+) is the output of deterministic latent mapping G. A representation sup is said to be the sufficient label-related representation and Zzup' is said to be the minimal and sufficient representation if:
sup (2x:; Y) and Zsup- argminI X²) s.t. [(Zaup Y) is maximized. G
Zzup
The learning procedure for definition 3 is illustrated in Figure D (a). The method is equivalent to employ compressed representations reduce the complexity (redundant information) of I (Zxx; X²) by minimizing and providing sufficient representation to class label Y by maximizing / 1(2x1; x Y). Similarly and motivated by multi-vier view information bottleneck settings (16), present the objective of learning sufficient (and minimal) representations with domain- domain-invarian invariant information in the below definition:
Definition 4. (Minimal and sufficient representations with Comain-invariane
(16)). Let Zx
Q(X") is the output of deterministic domain invariant mapping in the definition ! Then Z₁
is said to be the sufficient domain-invarian representation and 7, is said to be the minimal and sufficient representation if:
Z1 I and Z₁- (ergmnn((2;;X)) s.I. T(Z;; X2) is maximized. 2,
Q
Definition introduces a learning strategy for
across domains (or views) that preserves shared information across two domains by maximizing I I(Zx:: X²); and also reduces
specificity (redundant information) of the domain 5 by minimizing I (Zx X1). We next present a lemma about the conditional independence between the latent representation Zxi and both the label Y and the random variable X when Q, R, and G are deterministic functions of the random variable -

---

x)) is a Dirac delta function. then the following con- ditional independence holds: Zxi(X and X² 1T Zx: [x' inducing a Markov chain X² 4 Y ++ X xxx.
Lemma 1. (Determinism (14)) If P(Zxı
The proof of Lemma i provided in Appendix A.1
Lemma simply states that Zx1 contains no more information than X Now, we show the ineffectiveness of only learning domain-invariant approach, based on the existence of the label-related domain-specific in the following theorem:
Theorem 1. (Label-related information with Nomuin-specificity Assuming that there exists domain- specific value Eι > O in domain St (see Assumption [). the label-related representation based learning approach (i.e., using 7spu and Zssup) ) provides better prediction performance thun the domain-inwe domain-invariant invariant representation based method (i.e., using 7, and 7 Formally,
I(x+:Y)= [usupi Y) I [uup
I() + I(Zr,;Y).
The proof of Theorem T mainly based on the result of Lemma] T and is provided in Appendix A.2
The visualization of Theorem [ depicted by Figure [ (b), where the domain-specific value for domain S E2 is obtained in the same way as S1. It indicates that if an existing domain-specific representation has the positive corresponding information value e, the domain-invariance-boe learning method provides an inferior prediction performance to our proposed method that incorporates both domain-invariant and label-related domain- speciffc.
Now, Theorem D suggests that besides optimizing a comainn-nvariian mapping Q as usual, we should jointly optimize domain-specific mapping R to achieve better generalization performance. However, in domain generalization, we are not allowed to access the target domain for training and must use Q and R from source domains. As pointed out in (0) 7), although domain-invariant might be the same because it is unchanged across source domains, there is no guarantce whether this domain-specific information on the source domain is relevant to the target domain while making the prediction. Figure 1 (c) illustrates this case when the target domain has different X and Y then some extracted comain--peciffe from source domains become redundant information. Therefore, the next raising question is how to learn domain Nomain-invariant and domain-specinf effectively.
To handle these shortcomings, we next propose unified framework that jointly optimizes both Q and R by disentangling their feature representation. In particular, the deterministic mapping Q is optimized by adversarial learning to extract useful domain-invariane features across domains. Meanwhile, by leveraging the transfer weight representations from meta-source domains to adapt to a meta-target domain, we apply meta-learning deterministic mapping R to force it to extract relevant domain-specific features of the target domain to improve generalization ability.
2.3 Algorithm: meta-- Specific-Domain Invariant MDDSDI)
So far, we have discussed the main ideas of our proposed method. Here we discuss implementation details for our MDSDD apSooaaprocchh
Figure shows the graphical model and overview of our murDDDD] framework. In particular, our unified representation Z, Q9 (X); a domain-specific representation Zg = Re,, (X); ; a domain discriminator Den, Zi - I,N; a domain classifier Dons Zg + 1, N and classifier Fer Z, @ Zs y. We also denote domain random variable by D, sample space by DD and outcome by d.
network consists of the following components: a
Domain-Invarian and Domain-Specifie Extraction. The defined in Definition I is obtained by using an adversarial training framework (3). in which the domain discriminator DoD, tries to maximize the prediction probability of the domain label from the latent Zr, while the goal of the encoder Qit cannot discriminate the domain of X. This task can be performed by solving the following min max game:
representation Z
map the sample X the latent Z,, such that Døp,
min max [Tz, EE,,s-x. X (d log Dr(((=))))
89 8D,
To extract the domain- domain-specifie 7s defined Definition 2 we propose the use of the domain classifier Depg that is trained to predict the domain label from Zs. The corresponding parameters 0D and
5

---

Z1
D1
D
d
Adversaral
X
0Q
X
F
Meta
7
PR
n
R
Zg
Dg
D
(8)
[b]
including: domain- invariant Z is optimized via adversarial training with domain discriminator D,, domain-specific 2 is optimized via domain classifier Ds. these latent Z and Zy disentangled by using covariance matrix. To push them to contain label information, these latents integrated into classifier F which is optimized via cross-entropy with the label Y. To make the model able adapt specific information from source unseen domain while still remaining
Figure The graphical model (a) and overall architecture (b) for our proposed mDSDI I
information across domains, we additionally push Zş through meta-learning
procedure.
O are, therefore, optimized with the objective function below:
min (Lzg = -r,,d~x log Ds(R))))]
Bpg Ap.
Disentanglement between InvarnnInvariant and Domain-Specifie. The disentanglement condi- tion between two random vectors Z₁ and Zs can be solved by forcing their covariance matrix, denoted by Cov(Z,, Zs) close to 0. detailed discussion of disentangled two representations is provided in Appendix B.3 The related parameters (fq On) then updated in the following optimization
problem:
min (LD = Ex~x [l]Cov(Q(#)) R(x)))) 8q
where || 1l2 is the L2 norm.
Sufficiency of domain-specifie and comain-invariian w.r.t. the classification task. The goal of the classifier F parameterized by OF is to predict the label of the original sample X based the domain-invariant Z, and -ppaifi Zg, i.e.,
y= Fop (Zi @ 7s):
(4)
where denotes the concatenation operation. Then, the training process of F is then performed by solving
min 0gg 9p
(LT = -E log F(Q(z),
(5)
Meta- Training for Domain-Specific Information. To encourage the domain-specific representation Zs adapt information learned from the source domains to the unseen target domain, introduce the use of meta-learning framework (II) targeting a robust generalization. Note that the domain- invariant feature Z, remains during the meta-learning procedure. In particular, each source domain Sm. E 1, N is split into two sub-domains, namely meta-train Smr and meta-test Sme- The domain- specifcc parameters 0R and the classifier parameters 0F are then jointly optimized as follows:
min {LTm == f (w -f (w, Smr) Sme)}.
(6)
where w (0R, 0F) and
(w, (ww) - Ez.g log F(Z1, R(x))]
Training and Inference. The pseudo-code for training and inference processes of our proposed
mDSDI framework is presented in Algorithm Each iteration of the training process consists of two steps:
i) First, we integrate the objective functions function La defined as follows:
and to construct objective
min max (LA = L2, + Leg + +DDLD + Lr), 0g. 0g.0 0n On,
(8)
where λz, Azo and Ap are selected as the balanced parameters.

---

ii) The second step is to employ meta-training to adapt task-related domain- specific from source domains to unseen domains. In each split, then the gradient transformation step from meta-train domains to the meta-test domain is performed by solving the optimization problem (6).
the meta-train and meta-test are
Algorithm 1: Training and Inference processes of MDSDD
Training Input: Source domain S encoder Q8 Røg. domain classifier Dep, Deps for Zr, Zs, task classifier Før, batch size B, learning rate 1). Output: The optimal: Qea Rig Fōps
for ite + iterations do Sample Sy with a mini-batch B for each domain S(9; Compute LA using Eq. (8) and perform gradient update Veg.0n.0d, Ong Le with f).
jor = 1 + N (number of source domains) do
Split Meta-train Sz/j. Meta-test Sj:
Meta-train: Perform gradient update 0F by minimizing Eq. with SByj and yi
Meta-test: Compute Irr,, using Eq. (6) with Sj and updated gradient from Meta-train; Meta-optimization: Perform gradient
gradient update V Ven.0p Io.,. with vy:
end end
Inference Input: Target domain S" optimal: Qao Rig. Før Output: Y using Eq. (4)
3 Experiments
3.1 Experimental settings
Dataset. To evaluate the effectiveness of the proposed method, utilize 7 commonly used datasets
including: ColOred-MNIST (18): includes 70000 samples of dimension (2, 28, 28) in binary clas- sification problem with noisy label, from MNIST over 3 domains with noisy rate d E {0.1. 0.3, 0.9}. RotASed-MNISST [9); contains 70000 samples of dimension (1, 28, 28) and 10 classes, rotated from MNIST over 6 domains E {0, 15, 30, 45, 60, 75]. VLCS (20): includes 10729 samples of dimension (3, 224, 224) and 5 classes, over 4 photographic domains d (Caltech111, LabelMe, SUN09, VOC2007). PACS (2): contains 9991 images of dimension (3, 224, 224) and 7 classes, over 4 domains fartpaint, cartoon, sketches, photo). Office-Home 21: has 15500 daily images of dimension (3, 224, 224) and categories, over domains d E art, clipart, product, real]. Terra Incognita (22:: includes 24778 wild photographs of dimension (3, 224, 224) and 10 animals, over camera-trap domains d (L100, L38, L43, L46). and DomainNet (23): contains 586575 images of dimension (3, 224, 224) and 345 classes, over domains E (clipart, infograph, painting, quickdraw,
real, sketch}. The detail of each dataset provided in Appendix C.1
Baseline. Following DomainBed (24) settings, we compare our model with 14 related methods in DG which are divided by 5 common techniques, including: Standard Empirical Risk Minimization: Empirical Risk Minimization (ERM (25)); domaiin-peecfic-tearniig Group Distributionally Robust Optimization (GroupDRO (26), Marginal Transfer Learning (MTL (7))), Adaptive Risk Min- imization (ARM (28)); Meta-learning: Meta-Leernin for DG (MLDG (I);; domain-invariant- learning: Invariant Risk Minimization (IRM (18), Deep CORrelation ALigmment (CORAL (9h).
Maximum Mcan Discrepancy (MMD (30)), Domain Adversarial Neural Networks (DANN (II).
Class-conditional DANN (CDANN (32)) Risk Extrapolation (VREx (33); Augmenting data: Inter-
domain Mixup (Mixup 34
35 36)). Style-Agnostic Networks (SagNets (37)), Representation Self
Challenging (RSC 38) The detail of each method is provided in Appendix
We use the training domain validation set technique as proposed in DomainBed (24) for model selection. In particular, for datasets, we first merge the raw training and validation, then, we run the test three times with three different seeds. For each random seed, we randomly split training and validation and choose the model maximizing the accuracy on the validation set, then compute performance on the given test sets. The mean and standard deviation of classification accuracy
7

---

from these three runs are reported. We evaluate generalization performance based on backbones MNIST-Come Cet (24) for MNIST datasets and ResNet-55 (39) for nON-MNIST datasets to compare with the mentioned methods. Data-processing techniques, model architectures, and changes of objective functions during training are presented detail in Appendix C.3 C.5 All
source code to reproduce results are available at https:/gghhub.ccom/nnnnReeeeee
3.2 Results
Table 1: Classification accuracy (%) for all algorithms and datasets summarization. Our mDSDI method
achieves highest accuracy average when comparing 14 popular DG algorithms across 7 benchmark datascts.
Method
CMNIST
RMNIST
VLCS
PACS
OtticeHome
TerraIne
DomainNet
Average
ERM (5
51.5.0.1
98.0:1000
77.510.44
85.5+0.2
66.520.3
16
40.9=0.1
66.6
IRM [8
52,0#0.1
97.740.1
78.5:0.5
83.540.8
64.34222
47.6:0.8
33.912.8
65.4
GroupDRO 26
52.110.0
98.0+0.00
76.720..
84.4-0.8
66.0:0.7
43.2:11.1
33.3-0.2
Mixup R4 553 36
52.1+002
98.0+0.11
77.4+0.6
84.6+0.6
68.1+0.3
47.9+0.8
39.2+0.1
66.7
MLDG 11
51,5++0.1
97.9+0.0
77.2+0.4
84.9+110
66.8#0.6
47.7+0.9
41.2=0.1
66.7
CORAL 29
51.5+0.11
98.0++0.1
78.8++0.6
86.2+0.3
68.7+0.3
47.6=110
41.5++011
67.5
MMD
51.5+002
97,910.0
77.510.9
84.620.5
66.320.1
42.2-11.6
23.419.5
63.3
DANN (31)
51.5++0.3
97.8+0.1
78.6+0.4
83.6#0.4
65.9=0.6
46.7=0.55
38.3=0.1
66.1
CDANN (32
51.7.0.1
97.910.1
77.510.1
82.620.9
65.8.113
4..8-11.6
38.320.3
65.6
1
51.4+0.1
97.9+0.0
77.2+004
84.6+0.5
66.400.5
45.6+11.2
40.6+0.1
SugNets (37
51.7=0000
98.0:10.0
77.8+0.5
86.3:0.2
68.120.1
48.6=11.0
40.320.1
67.2
ARM 28
56,2+0.2
98.2+-0.1
77.6+0.3
85.1+0.4
64.8+0.3
45.5+0.3
35.5+0.2
66.1
VREx (33
51.8+0.11
97.9+0.11
78.3+00.
84.940.6
66.420.6
46.4-0.6
33.6=2.9
65.6
RSC (38
51.7+0.2
97.6+0.11
77.1=0.5
85.2+0.9
65.5=0.9
46.6=110
38.9=0.5
mDSDI Ours)
52.240022
98.010011
79.0
86,2#0.2
69.2=0.4
48.1
42.8-0.1
67.9
Table 1 summarizes the results of our experiments on 7 benchmark datasets when compared with mentioned methods. The full result per dataset and domain is provided Appendix C... From these results, we draw three conclusions about our ourImDSDI model:
Our MDSDI still preserves domain-invariane information. We observe in some target domains which have background-les images and assume only contain invariaan information such as -NISS, Notated-MNISS or Terra Incognita (similar observation in cartoon or sketch in PACS, clip-art or product in OfficeHome, and quickdraw in DomainNet. Full results in Appendix C.4). our DDSDI model still achieves competitive results with other baselines (e.g., 52.2% in Colored- MNIST, 98.0% in MNNIS, and 48.1 3..1% % in Terra Incognita) which are based on domain- invariant-learning techniques such as DANN, C-DANN, CORAL, MMD, IRM, and VREx. Those results demonstrate the effectiveness of our adversarial training technique for extracting domain- invariant features. Furthermore, due to considering disentangled domain-ine iiraainnniaarant and domain- specific latent, even in situations where the samples do not have domain-specifie, our model still performs well by retaining informative -ovaiininvariaace.
Our DSDI could capture the usefulness of domain-specific information. In contrast, we ob- serve that in some target domains that have relevant domain-specitf with source domains such as landscape background of the object class from photographic pictures in VLCS (similar observation in PACS such as dogs in the yard or guitars lying on a table photo and art domain, bed in the room or bike parked on the street in the Art and Real-world domain of OfficeHome. Full results in Appendix C.4). DDSDI VLCS, 86.2% in PACS). This means that our tion better but also our domain-specific ones cover helpful information in special scenarios such as backgrounds and colors related to objects in the classification task. Moreover, when comparing with other domain-specific based techniques such as GroupDRO, MTL, and ARM, the results showed that domain-specific features learned by meta-training from our model are more helpful than theirs and have captured useful domain domain-specific features from those object- background relations.
significantly higher results than other methods (e.g.. 79.0% in features not only support generaliza-
Extending beyond the invariance view to usefulness domain-specific information is important. As shown in Table [ our [DDDDI has the highest average number with 67.9% (highlighted with statistically significant according to a t-test at a significance level a 0.05). The reason why our model outperforms other baselines could be explained by the fact that their domain-invariant methods are not able to capture domain-specing information, and so have poor performance. Meanwhile, when comparing with other domain-specific based methods, their models only concentrate on domain- information to our techniques in some
specific techniques, and provide inferior
8

---

background-les images. In contrast, due to considering disentangled domain-invann invariant and domain- specific features, and having the right strategy to learn each latent, our model captures both this useful information, hence, outperforms their results. Not only has the highest average number, but our method also dominates other methods on known large-scale dataset such as 69.2% in Office-Homme or 42.8% in DomainNet. This implies that besides the essential combination between domain- invariant and domain-specific, when the number of datasets increases, our method can extract more relevant information for complex tasks, such as classifying 345 classes in DomainNet. These results also mean that our model has balance between informative domain-invariant and domain- specific features adapt better different environments than others, therefore showing the highest average in all settings.
3.3 How does m D work?
Domain-invariant (Clesses)
Domain-invariant (Domains)
Domain-speoltic (Classes)
Domain-specific (Domains)
Figure 3: Feature visualization for domain-invariant: (a): different colors represent different classes; (b): different colors indicate different domains. Feature visualization for domain-specific: (c): different colors represent different classes; (d): different colors indicate different domains. Source domain includes: art (red), cartoon (green). sketch (blue) while target domain photo (black) in domain plots. Best viewed color (Zoom in for details).
To better understand our framework, visualize the distribution of the learned features with t- SNE (40) to analyze the feature space of comain-invariaat and domain-specifie. As shown in Figure on PACS Dataset, our of the domains (see Figure 3 (b)). However, these domain-invariant features still make mistakes on
extractor can minimize the distance between the distribution
the classification task, indicated by mixture of points from different class labels in the middle (see
Figure 3 (a)), and many of these points are from the target domain (black color in the Figure 3 (b)).
Meanwhile, the domain-specific representation better distinguishes points by class label (see Figure 3
(c)). More importantly, the photo domain's 's specific features (black) are close to the art domain (red)
(see Figure 3 (d)). This is reasonable because only these two domains include backgrounds related
to the object class. It implies that meta-training in our model well learns specific features that can be adapted the new unseen domain.
3.4 Ablation study: Important of mDDDI on the Background- Colored-MNoEE dataset
This section examines our system design by checking its performance under different settings in the real scenario with our generated dataset (a similar experiment with PACS benchmark dataset is in Appendix C.6).
dataset. Figure 4 illustrates our Baakground-Colored--aa erated from the original MNIST. We assume the domain-invarian is the digit's 's's sketch and design the dataset so that the background color is domain-specific. As result, on the unseen domain, domain-specifé will be useful for the classification task. Specifically, the dataset includes three source domains di,, different by digit's color {red, green, blue }. generated from a subset with 1000 training images for MNIST per each domain. In each source domain, the background color is the same for intra-class images but different across classes. the target domain, 10000 testing images of MNIST are colored for one target domain die with digit color (orange), ). In this domain, each class's background color is similar to the same class's background color in one of three source domains.
Background-Colordd-MNNN
gen-
Importance of DDDD We aim to prove the combination of learning disentangled representation Nomain-specific, domain-invariant, and meta-training on domain-specific are important in this sce- nario. To do compare model under nine settings: learning domain domain-invariin invariant only (DI),

---

learning comainnsspecinn only (DS), meta- mheta-training on Comminn-ivvrriin (DI-Meta), meta-training on domain-specinf (DS-Meta), combination of tanglement loss Lp (DSDI-Withon) I.D). combination of domain-invarian and domain-specitfe without meta-training
and domain-specifia without disen-
Meta), meta-training on both representation Z, and Zs (DSDI- Meta), meta-training on domain-invarian without domain-specinn (DSDI-Meta DI) and our proposed framework (mDSDi-Meta DS), which is meta- vecta-training on domain-specific without domain-invariant Table2showw that our model is the best setting with 89.7% It proves that combining domain-invarinn and domain-specific is crucial by dominating the settings with only domain-invariant or domain- specific (DI, DI-Meta, DS, DS-Metal Regarding disentangling worth noting that without disentanglement loss Lp. the model only achieves 81.4%. which is lower than mDSDI It reveals adding disentanglement loss Lp is essential to boost model performance. Compared with meta-training on both, which only reaches 82. 1% and on domain-invarisn it implies that meta-training is necessary but only for the domain-specifie, and our arguments reasonable. It also shows that our combination with domain-invariant and domain-specifie is not easy: deep ensemble between two neural networks, but existing deep-down insight in framework in DG when compared with MDSDI HIDSDD-Without Without Meta which can be seen as a type of ensemble, is only around 80.4%.
representations Z₁ and Zs, it is
are 79.0%,
Table 2: Classification accuracy (%) on Background- Colored-MNIST. Ablation study shows impact of domain-invariant when combined with meta-training on domain-specific our method.
Source domains:
Blue
S
Green
Method
Accuracy
Red
DI
65.714.6
DI-Meta
63.611..1
DS
70.7=4.88
Target domain:
DS-Meta
75.3
Orange
DSDI-Without LD
81,4+2.6
DSDI-Without Meta
80.4
Figure 4:
Beckground-Coooced-NNii Dataset,
DSDI-Meta
82,1+114
where source domains include (rdd, green, blue) digit colors and target domain has (orange) color.
DSDI-Meta DI
79.0+2.3
mDSDD Meta DS (Ours)
89.7++0.8
Conclusion and Discussion
Despite being aware of the importance of domain-specifie information, little investigation into the theory and a rigorous algorithm to explore its representation. To the best of our knowledge, our work provides the first theoretical analysis to understand and realize the efficiency of domain-specific information in domain generalization. The domain-specing contains unique characteristics and when combined with domain-invariant information can significantly aid performance on unseen domains. Following our theoretical insights based on the information bottleneck principle, we propose mDSDI algorithm which disentangles these features. We next introduce the use of the meta-training scheme to support domain-specitn to adapt information from source domains to unseen domains. Our experimental results demonstrate [DDDII brings out competitive results with related approaches in domain generalization. In addition, the ablation study with our further illustrated and demonstrated the efficiency of combining domain-invariant and domain-specific via our proposed MDDD Our theoretical analysis and proposed MDSDD framework can facilitate fundamental progress in understanding the behavior of both domain fomain-invariant and domain-specnt representation in domain generalization.
Toward robustness algorithm that can effectively learn both features, there are certainly many challenges that remain in our paper, for example, theorem to explain when domain- sppeciii may hurt performance in the unseen domain, a stronger connection between theory in implementation, a method to make two representations to be non-linearly in- dependent as well as lower computational cost of the covariance matrix. In future, we plan to continue tackling these challenges to provide better understanding and learning framework in domain generalization then extending to broader settings of transfer learning. A detailed clarification, discussion, and plausible methods in the future work are provided in Appendix B
and domain-specitfe
10

---

References
[1] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifi- cation tasks to new unlabeled sample. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24, pages 2178 2186. Curran Associates, Inc., 2011.
[2] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 1342-5550 2017.
[3] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018.
[4] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multido- main discriminant analysis, 2019.
[5] Maximilian Ilse, Jakub M. Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders, 2019.
[6] Ching- Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via ionainninvariant representations, 2020.
[7] Prithvijit Chattopadhyay, Yogesh Balaji. and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization, 2020.
[8] Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On learning invariant representation for domain adaptation. CoRR, htss/1901.09443 2019.
(9] Fredrik D. Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant representations, 2019.
[10] 7. Ding and Y. Fu. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, 27(1):304-313, 2018.
[11] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learnig for domain generalization, 2017.
[12] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368-377, 1999.
[13] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc, of the 37-th Annual Allerton Conference Communication, Control and Computing, pages 368-377, 1999.
[14] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from multi-view perspective. In International Conference on Learning Representations, 2021.
[15] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations, 2018.
[16] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. CoRR, abs/2002.07017, 2020.
[17] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specific low-rank decomposition, 2020.
[18] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization, 2020.
[19] Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. CoRR, 508.07680, 2015.
11

---

[20] Chen Fang. Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web imgges for softening bias. In 2013 IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[21]
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation, 2017.
[22] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. CoRR, htss/180704975, 2018.
[23] Xingchao Peng, Qinxun Bai, Xide Xia, Zijum Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation, 2019.
[24] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. CoRR, abs/2007.0144 01444,, 2020.
[25] Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
[26] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020.
[27] Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning, 2021.
[28] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group distribution shift, 2021.
[29] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation, 2016.
[30] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018
[31] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversariy training of neural networks, 2016.
[32] Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representation, 2018
[33] David Krueger, Ethan Caballero, Heem--enrikk Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. -ut-of-distributtin generalization via risk extrapo- lation (rex), 2021.
[34] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup, 2019.
[35] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. 2020.
[36] Yufei Wang, Haoliang Li, and Alex C. Kot. Heterogeneous domain generalization via domain mixup. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020.
[37] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias, 2021.
[38] Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross- domain generalization, 2020.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
12

---

[40] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):557--266 2008.
[41] Thomas M. Cover Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006.
[42] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):223-77, 2012.
[43] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[44] Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. CoRR, abs/ las//9977002755 2019.
13


Reviews:
[{"review_id": "z-Qu6IV2vN", "reviewer": "Reviewer_d6mf", "summary": "This paper proposes a framework to learn both domain-invariant (DI) and domain-specific (DS) features for the purpose of domain generalization. The DI features are learned using an existing adversarial method which seeks a representation from which the domain *cannot* be predicted. The DS features are learned through a meta-learning framework, where the outer loop seeks to do well on a held-out validation-set (in-distribution examples). The main contribution is the unified framework in which to learn both DI and DS features. Experiments are conducted on selected datasets of DomainBed, and the results on these selected datasets are strong. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Summary**: Novel and interesting approach to exploiting DS features. DS features clearly boost performance on datasets where the source confounders (spurious correlations) correlate with *the same class label* in the target/test domain. E.g. if, in a source domain, grass strongly indicates cow and not camel, then using grass in the target/test domain will improve predictions if and only if grass still indicates cow. If instead grass correlates with camels in your target/test domain, then the DS features will *hurt* performance. This was the entire motivation behind Invariant Risk Minimization (IRM, [1]) -- discard DS features as we cannot trust them in the test domain -- and it seems to be completely ignored throughout this paper. While the direction is interesting and the results impressive, I find it inexcusable to ignore (and perhaps bury -- see below) this important caveat/limitation. The severity of failing to explicitly state assumptions/limitations is compounded by the authors’ claim of a “theoretically sound” framework. Thus, I believe the paper is just below the acceptance threshold in its current form, but am *very* willing to increase this score if the authors: (i) add a thorough discussion of the limitations of using DS features; and (ii) explicitly define their assumptions (i.e. explicitly define the situations in which DS features help and hurt performance in the test domain).\n\n**Strengths**: \n- *Interesting and important direction*: combining DI and DS features.\n- *Strong empirical results*: on the selected datasets of DomainBed.\n- *Very clear writing*. I particularly like the Venn diagrams in Figure 1, the simplicity of Figure 2, and the insights of Figure 3.\n\n\n**Weaknesses**: \n- *Limitations are ignored, and perhaps even buried*. As described in the summary above, DS features can hurt performance if they correlate with a different class label in the test domain than they did in the source domain(s). This point is mentioned in lines 24-25, but then ignored throughout the rest of the paper. This issue can be divided into two points:\n    - Most importantly, the *Color-MNIST* dataset of [1] is not only left out of DomainBed evaluations without explanation (clearly it is left out as DS features will hurt test performance), but the authors introduce their own dataset with the same name, and design it such that the DS features in the test domain *do* help test performance. Specifically, the test domain contains *seen* background colours that are *unique to the class label*, e.g. a particular kind of green always indicates ‘5’ in training domains. This overloading of the name Color-MNIST only serves to conflate two very different datasets, and further buries the limitation of using DS features in the test domain. I am left wondering why:\n        - The authors didn’t call it e.g. Background-Color-MNIST to distinguish it from the already-existing Color-MNIST.\n        - The authors did not discuss this overloading/conflation -- i.e. why they left out the original Color-MNIST [1] and created their own dataset with the same name.\n    - Secondly, there are several points throughout the paper where the authors make grand claims about domain generalization without acknowledging the aforementioned caveat (or, equivalently, the required assumptions for these claims to hold true). For example on lines:\n        - *93*: \"will limit generalization ability\": *only if DS features from the source domains correlate with the same class label in the target domain*.\n        - *154*: “better prediction performance\": *in domain S^1*.\n        - *160*: “inferior prediction performance\": *in the source domains*.\n        - *163*: “better generalization performance”: under what assumptions? See first point. Clearly this doesn’t hold for e.g. the ColorMNIST dataset of [1].\n        - *166-169*: the authors *do* acknowledge that DS features may not be relevant to the target domain. However, they speak about redundancy (present or not) rather than the true limitation of DS features hurting performance by indicating the wrong class.\n\n\n**Additional feedback/questions** (in order of importance): \n- Adding baselines of ERM (on the pooled source-datasets) and IRM to *your* Color-MNIST dataset would really help put the numbers into context. ERM might do quite well since, in the test domain, the background colours are seen and unique to the class label.\n- What hyperparameters were fitted using the validation set for all methods (yours included)? You say you selected the best model using a validation set, but your Table 3 of Appendix B.3 does not contain a final column showing the random search distribution for hyperparameter sweeps (as in Appendix D.2 of [2]).\n- Why was the Terra Incognita dataset of DomainBed not included?\n- How long did it take to train your model? \n- Is the calculation of the Covariance matrix expensive for large feature dimensionality? Does it require 12 CPUs?\n- Figure 3 -- best view in color *on a computer*! Nice figure nonetheless.\n- Do you use a *shared* feature-extractor for learning DI features? Is this optimal? See e.g. [3]\n\n[1] Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint arXiv:1907.02893.\n\n[2] Gulrajani, I., & Lopez-Paz, D. (2020). In Search of Lost Domain Generalization. In *International Conference on Learning Representations*.\n\n[3] Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017). Adversarial discriminative domain adaptation. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition* (pp. 7167-7176).\n\n\n***Update after response***: I have raised by score by 1 – see reply to authors' comment.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the authors have not adequately addressed the limitations of their work (see point under \"weaknesses\" above). Authors selected \"N/A\" for limitations in the checklist (1b).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "NbnJ591zkI", "reviewer": "Reviewer_YJwo", "summary": "This paper studies the Domain Generalization (DG) task in which multiple source domains are observed and target domains are assumed to be unknown. The authors proposed a new meta-Domain Specific-Domain Invariant (mDSDI), going beyond the invariance view to highlight the Domain Specific information. The key contribution is to disentangle features in the latent space and jointly learning both domain invariant and domain specific features jointly.\nThe theoretical analysis is proposed for the method. Extensive experimental results have been conducted on a few datasets.   ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe domain specific assumption is not new to the domain generalisation task. The novelty of the proposed meta-training is not clearly indicated. Here are a few comments. \n1. There are a lot of the definitions and assumptions, which can be shortened since it is quite standard for the domain generalisation task. \n2. No supplementary material is provided, and hence cannot check about the correctness about the proposed Lemmas and Theorem.\n3. The Ablation Study has been conducted in ColoredMNIST, which is good but the authors may need either compare all the baseline algorithms on this dataset, or use one of the 5 benchmark datasets.\n\nOne more concern is about the clarity the paper and the presentation:\n1. The writing needs to be improved. \nFor example, the authors defined be abbreviations at the very beginning, such as  Domain Invariant (DI) and  Domain Specific (DS). After that, the abbreviations are used throughout the paper. However, sometimes it is quite challenging to memorize these abbreviations. My suggestion is that the authors can still use Domain Invariant and Domain Specific terms in the main paper.\n2. Some of the reference papers are not in the standard citation format.\nFor example: \n[42] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.\n[43] Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. CoRR, abs/1907.04275, 2019.\n\n[42] has been published in ICLR while [43] has been published in ECCV. \nI suggest the authors to carefully check all the references, and cite them in a proper way.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Buga2WI9wZJ", "reviewer": "Reviewer_88Lv", "summary": "The paper studies the problem of Domain Generalization (DG), where a model trained on single or multiple source distributions / domains is expected to generalize to data from unseen / novel distributions at test-time. The authors build on top of the observation that a subset of prior approaches rely on learning representations utilizing domain-invariant information and tend to ignore domain-specific information (which correlates with task at hand) and that ignoring domain-specific information could unintentionally harm generalization performance. Given this, the authors formalize the above concepts of invariance and specificity under a probabilistic framework and propose a practical realization of the same named meta Domain-Specific Domain-Invariant (mDSDI) representation learning. The paper claims that mDSDI is capable of explicitly disentangling domain specific and domain invariant information. In practice, key components of mDSDI include a domain-invariant encoder (trained to fool a domain discriminator), a domain-specific encoder (trained to aid a domain classifier) and a classifier (for the downstream task) that takes representations from both encoders as input to predict the semantic category. Additionally, mDSDI encourages minimizing the covariance of domain invariant and specific representations and proposes using intra-domain meta-learning to ensure representations are capable of utilizing information from source domains in the target. On the DomainBed benchmark, mDSDI outperforms 14 algorithms on 5 datasets. Additionally, the paper claims that mDSDI retains domain invariant information and captures useful domain-specific information.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will now highlight the strengths and weaknesses of the submission, which will address quality, clarity, originality and significance of the submission.\n\nWith the exception of the points covered under weaknesses, I believe the following are the strengths of the submission in its current form:\n\n- The paper is generally well-written and easy to follow. With the exception of a few points under weaknesses, the authors generally do a good job of motivating different components of the proposed approach -- formalizing domain invariance, specificity and label-correlated domain specificity, and the subsequent domain specific and domain invariant encoders. I generally like that the authors chose to formalize the ideas of domain specificity and invariance from an information bottleneck perspective. mDSDI being grounded in the proposed formalism is definitely a positive point in support of the paper.\n\n- While adversarial techniques to learn domain invariant representations have been explored in the past (see [A]), the novelty of the proposed approach lies in learning an additional domain specific encoder, minimizing covariance between domain specific and domain invariant representations and additional meta learning on domain-specific features to encourage more generalizable representations. The added novelty in terms of modeling (which is grounded in the proposed formalism) is a positive point in support of the paper.\n\n- mDSDI seems to be either competitive with or outperform prior approaches on the benchmarks (5 datasets under DomainBed). Additionally, feature visualizations in Section 3.3 indicate that domain-specific components are more capable of separating out semantic categories as opposed to domain-invariant components, providing support to the claim that relying on domain-specific information is beneficial. I also like the additional ablation highlighting the utility of individual design choices in a controlled colored-MNIST setting. Although on a controlled and simplistic benchmark, this ablation is still useful for future work building on top of the proposed approach. \n\n[A] - Domain Generalization with Adversarial Feature Learning\n\nI don’t have major weaknesses to point out about the draft. The following comments / concerns are centered mostly around some clarity issues and lack of some necessary experimental comparisons / ablations.\n\n- A minor clarification question. In the meta-training for domain-specific information setup, the authors state in L207-208 that each source domain is split into two sub-domains. This is slightly confusing Does this mean instances from domain i in a minibatch are split into two groups? Or that k-source domains in a minibatch are split into k-1 meta-train and 1 meta-test domains? The algorithm seems to imply the latter but the text seems to imply something else. Could the authors clarify if it’s the latter?\n\n- The authors introduce the additional covariance minimization step between the domain invariant and specific representations to encourage disentanglement. While I appreciate the extensive ablations conducted by the authors to highlight the contribution of several design choices in Table 2 of the main paper (and in supplementary), the paper would benefit if the authors include additional ablations demonstrating the utility of the disentanglement loss (to what extent is performance impacted in the presence and absence of covariance minimization). This would help understand to what extent is incentivizing explicit disentanglement necessary on the DomainBed datasets and colored-MNIST. I would encourage the authors to include such an ablation.\n\n- If I understand correctly, learning domain specific and domain invariant representations in mDSDI and the obtained results are somewhat implicitly conditioned on the availability of domain labels -- instances belong to different domains because the domain labels say so. The experiments and ablations in the paper highlight the utility of domain specific representations and the underlying utility (even under the formalism) is conditioned on the availability of domain labels. While this is a perfectly fine assumption, an additional experiment supporting the claim of learning useful domain specific representations can further be strengthened by an experiment where domain labels are randomized -- as in, say if there are 4 source domains and 1 target domain, data at training time is grouped randomly into 4 bins instead of by domain. If the out-of-domain results obtained are worse, then there’s additional validation for the utility of domain-specific representations. If they’re not significantly different, then the implication is that domain-specific representations are likely not required. I think including such an experiment is beneficial for the underlying claims of proposed formalism and will likely improve the current paper. I would encourage the authors to conduct this experiment both for one of 5 datasets under DomainBed and the controlled colored-MNIST setting.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall, I generally like the paper and the formalism adopted by the authors to ground domain specific and domain invariant components. I don't have major limitations to point out beyond the ones highlighted under weaknesses. I think addressing the highlighted experimental concerns are critical to the proposed approach. Barring those points, the paper is generally well written and the authors do a good job of discussing scenarios where mDSDI tends to help for several datasets (in the main paper and in supplementary).\n\nI don't see any potential negative social impacts of the work.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "3516a149ekh", "reviewer": "Reviewer_4Vtm", "summary": "The paper proposes meta-Domain Specific-Domain Invariant method to jointly learn disentangled domain specific and domain invariant features for domain generalisation. The usefulness of domain specific features is analysed under information theory. Experiments on several benchmarks show the effectiveness of the method.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The analysis of the usefulness of domain specific features in domain generalization under information theory is interesting. However, the disentanglement of domain invariant and domain specific features have been proposed in domain generalization [1]. The techniques used in this paper, such as Meta-learning, adversarial learning for invariant features, are also well-studied in many other works [2, 3, 4], which are not cited in the paper.\n\nQuality: The paper proposes to exploit both domain specific and domain invariant features for domain generalization and provides analysis under information theory, which is interesting. However, the detailed implementation and experiments have little to do with the information theory, making the model and theoretical analysis a bit disjoint. There are also some further experiments needed to support the method. See limitations.\n\nClarity: The paper is clearly written. The implementations and experiments seem a bit disjoint with the theoretical analysis. See above.\n\nSignificance: The paper proposes an interesting theoretical analysis about the relationship between domain specific features and domain invariant features in domain generalization. \n\n[1] X Peng, Z Huang, X Sun, and K Saenko. Domain Agnostic Learning with Disentangled Representations. In International Conference on Machine Learning, 2019.\n[2] Q. Dou, D. C. Castro, K. Kamnitsas, and B. Glocker. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, 2019.\n[3] S. Zhao, M. Gong, T. Liu, H. Fu, and D. Tao.  Domain generalization via entropy regularization.  In Advances in Neural Information Processing Systems, volume 33, 2020.\n[4] Y. Balaji, S. Sankaranarayanan, and R. Chellappa. MetaReg:  Towards domain generalization using meta-regularization. In Advances in Neural Information Processing Systems, volume 31, pages 998–1008,2018.\n\nAdditional comments:\n\n   (1)\tThe method is proposed based on information theory but there is little modules or supports about the information theory in the implementation and experiments, making the method not so convincing.\n\n   (2)\tThe technical novelty seems not so strong because adversarial learning and meta-learning used in the methods are both widely studied in domain generalization.\n\n   (3)\tThe paper misses some important references, such as the references provided in Main Review.\n\n   (4)\tThe paper proposes to jointly learn domain invariant and domain specific features. A question is why the domain invariant and specific features need to be disentangled first and then aggregated for prediction. How about jointly learn the features without disentanglement and aggregation? More analysis and supportive experiments are needed.\n\n   (5)\tThe difference of the extracted domain invariant ZI and domain specific features ZS is not clear. What will happen if the LD term in Eq. (8) was removed? The other hyperparamters lambda in Eq. (8) also need ablations.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 day"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper. 

Though the theoretical analysis looks nice, it may not be useful for the domain generalization setting. Specifically, the key theoretical result (Theorem 1) suggests that the representation learned by incorporating both domain-invariant and domain-specific information has a stronger dependence on labels than that learned by only considering domain-invariant information. However, there is no guarantee that such representation can generalize well to the "unseen" target domain(s). The theoretical results are based on some assumptions on the domain-invariant and domain-specific feature extractors. However, in the setting of domain generalization, as no target domain data is accessible, the domain-invariant feature may be "invariant" to the source domains only, and the domain-specific features of the target domain are not able to be extracted (accurately). This means that the theoretical results are built based on a very ideal case, but not practical for the domain generalization setting. It would make more sense if this work is applied to the domain adaptation setting where unlabeled data and some labeled data from the target domain are available for training (semi-supervised domain adaptation). Alternatively, the authors need to analyze under what conditions or assumptions for the unseen target domain, the proposed method can generalize well.

Moreover, it seems that the theoretical results are borrowed from existing findings on multi-view learning [14,16]. Is the extension of the theoretical findings from the multi-view setting to the multi-domain setting straightforward? It would be more clear if the authors could highlight their theoretical contributions compared with the techniques used in multi-view learning.

By considering the above comments, I recommend rejection for this work. 

Decision: Accept (Poster)