Exploiting Domain-Specific Features to Enhance Domain Generalization
Authors: Ha Manh Bui
Toan Tran
Anh Tuan Tran
Dinh Phung
Venue: NeurIPS 2021 Poster (2021)
Keywords: Representation Learning
Domain Generalization

Abstract: Domain Generalization (DG) aims to train a model, from multiple observed source domains, in order to perform well on unseen target domains. To obtain the generalization capability, prior DG approaches have focused on extracting domain-invariant information across sources to generalize on target domains, while useful domain-specific information which strongly correlates with labels in individual domains and the generalization to target domains is usually ignored. In this paper, we propose meta-Domain Specific-Domain Invariant (mDSDI) - a novel theoretically sound framework that extends beyond the invariance view to further capture the usefulness of domain-specific information. Our key insight is to disentangle features in the latent space while jointly learning both domain-invariant and domain-specific features in a unified framework. The domain-specific representation is optimized through the meta-learning framework to adapt from source domains, targeting a robust generalization on unseen domains. We empirically show that mDSDI provides competitive results with state-of-the-art techniques in DG. A further ablation study with our generated dataset, Background-Colored-MNIST, confirms the hypothesis that domain-specific is essential, leading to better results when compared with only using domain-invariant.

Exploiting Domain-Specific Features to Enhance Domain Generalization
Manh-Ha Bui' Toan Tran'
Anh Tuan Tran' Dinh Phung
1 VinAI Research, Vietnam 2 Monash University, Australia (v. habm1, v. v.toantm3, v. anhtt152, v. v.dinhpq2)@vimaii.t
Abstract
Domain Generalization (DG) aims to train a model, from multiple observed source domains, in order perform well unseen target domains. To obtain the gen- eralization capability, prior DG approaches have focused on extracting domain- invariant information across sources generalize target domains, while useful domain-specific information which strongly correlates with labels in individual domains and the generalization to target domains is usually ignored. In this paper, we propose meta-Domain Specific-Domaa Invariant (DDSDI) a novel theoreti- cally sound framework that extends beyond the invariance view further capture the usefulness of domain-specific information. Our key insight is to disentangle features in the latent space while jointly learning both domain- invariint and domain- specific features in unified framework. The domain domain-speciftt specific representation is optimized through the meta-learning framework to adapt from source domains, targeting robust generalization on unseen domains. We empirically show that mDSDD provides competitive results with state-of-the-art techniques DG further ablation study with our generated dataset, Background-Coloree MNISS,,-MNIST confirms the hypothesis that -ppeafi is essential, leading to better results when compared with only using domain-invariant.
Introduction and Related work
Domain Generalization DG has recently become an important research topic in machine learning due its real-world applicability and its close connection the way humans generalize learn in new domain. In a DG framework, the learner is trained on multiple datasets collected under different environments without any access any data on the target domain (T). One of the most notable approaches to this problem is to learn the "domain-invarian"" features across these training datasets, with the assumption that these invariant representations are also held in unseen target domains (2) 3 5 (). While this has been shown to work well in practice, its key drawback is completely ignoring flomain-specific information that could aid the generalization performance, especially when the number of source domains increases (7).
For instance, consider the problem of classifying dog or fish images from two source domains: sketch and photo. While the sketch contains conceptual drawing of the animal, the photo includes their taken picture within a background. In this case, sketch domain-invariant is kept across domains, while domain-spenn speciiic, e.g., a dog in a house or fish in the ocean. will be discarded due only existing in the photo domain. However, this background information, when present, could lead to an improvement of the classification performance in target domains due common association between the objects of its background, and when negligent sketches are hard to distinguish. From theoretical standpoint, there has also been strong recent evidence to indicate the insufficiency of learning domain- invariant representation for successful adaptation domain adaptation problems (8.9). For example,
"Correspondence to Manh-Hi Ha Bui: <hb. bummbumianhhaa@aa....... com>.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Zhao et al. (8) has pointed out the degradation in target predictive performance if domain-invarian representations are forced while the marginal label distributions on the source and target domains are overly different.
Utilizing domain-speciff features in DG has been widely studied in recent works (e.g.. 1T0 Ding and Fu (10) introduce multiple domain-specitff networks for each domain, then use the structured low-rank constraints to align them with innaiinnnv While this encourages the better transfer of knowledge, its main problem is the requirement of too many domain-specific networks. More recently, Chattopadhyay et al. (7) proposed a masking strategy to disentangle nomain-invariant and domain-specific to further boost domain-specinn cettaaamdddmminsppeeiii representations might not be disentangled since the learning and inferring procedures are performed implicitly (i.e., without any theoretical guarantee) through mask general- ization process. That means it lacks a clear motivation as well as theoretical justifications.
learning, but its key drawback is that domain-
Regarding meta-learning related work, a typical approach involving meta-learning in DG is MLDG (1I that is based on gradient update which simulates train/test domain shift within each mini-batch, mainly to learn transferable weight representations from meta-source domains to quickly adapt the meta-target domain, and so improve generalization ability. However, their task objective adapts for all representation features which include domain-invariant, since low effectiveness because domain-invarian stable across domains, pushing to adapt those features might affect the stability of those domain-invariamt invariant, leading to lower generalization performance on the target domain.
To handle these domain-invariant shortcomings, in this paper, we propose a novel theoretically sound DGJ approach that aims to extract label- informiiforeeeive domain-specific and then explicitly disentangles the domain-invarianl and domain-specifie representations in an efficient way without training multiple networks for domain-specific. Following the meta-learning idea and mitigating previous work's s drawbacks, we apply meta-learning technique specifically to exploit domain-specific quality which should need to be adapted to unseen domains from source domains. Our contributions in this work summarized as follows:
We provide a theoretical analysis based on the information bottleneck principle to point out the limitation of only learning invariant and the importance of domain-specific representation by a certainly plausible assumption.
We then develop a rigorous framework to formulate elements of iammainnivariaan/dommiin specific representations, in which our key insight is to introduce an effective meta- optimization training framework (IT) to learn domain-specinf representation from multiple training domains. Without accessing any data from unseen target domains, the meta-training procedure provides suitable mechanism to self-learn domain-specific representation. We term our approach meta--omaan Specific-Domain Invariant (DDDDI) and provide necessary theoretical verifications for it.
To demonstrate the merit of the proposed mDSDI framework, we extensively evaluate MDSDI on several state-of-the-arn DG benchmark datasets, including -olore--MNIST, Rotated-MNIST, VLCS, PACS, Office-Home, Terra Incognita, DomainNet in addition to our newly created Background-Colle -oceroound-ooored---ee MoloST for the ablation study to examine the behavior of our DDDD]
2 Methodology
2.1 Problem setting and Definitions
Let X' C CR be the sample space and y IR the label space. Denote the set of joint probability distributions on X' x y by Pxxy, and the set of probability marginal distributions on X' by Px. A domain is defined by joint distribution P(x, y) E Pxxx, and let P be measure on Pxxx, i.e., whose realizations are distributions on X' x V
Denote source domains by Sti {(x,) 11 D i i= I...,, N, where n; is the number (6) (4)
of data points in S(A), i.e., (af0) )) std p(0 y) where (() ~ P; and z p(O) in which p(( Px. In typical DG framework, a learning model which is only trained on the set of source domains [ J without any access to the (unlabeled) data points in the target

---

domain, arrives at good generalization performance on the test dataset S = =1x2] 03 13 where
(ef.uf) std pI (ar, y) and pT (x,, y) P.
First, we present the definition of domain Nomain-invarian invariant representation a latent space Z under covariate shift assumption (i.e., the conditional distribution P(y(æ)
(z) is unchanged across the source domains): Definition A feature extraction mapping Q X + 2 is said to be domain-invariant if the distribution (x()(X)) is unchanged across the source domains, i.e, Vi,j -1...... N, i have
= Pg (Q(X)), where (Q(())) = Pa(Q(X)IX ~ p == 1...., N. In this case, the corresponding latent representation Z Q(X) is then called the representation (see (3) also).
As mentioned in the example in the introduction part, the definition reveals that the extracted nomain-invariian latent Z₁ could be the conceptual drawing of the animal which is shared in both sketch and photo domains. However, when existing background information is taken by a picture such as house ocean, is crucial take these backgrounds into account because the domain domain-invariant feature extraction Q might ignore them by only existing in the photo domain. Therefore, we next introduce the definition of domain domain-specifer specific in latent space as follows:
Definition 2. A feature extraction mapping R X Z is said to be domain-specific if Vi,j I...,,N, N, * j such that p(d (R(X)) + p() pl) (R(X)) where (R(()) (x([(X)]X P()), i= i=1...,, N. In this case, given X ~ P the corresponding latent representation 2.5 R(X) is then called the domain-specific representation w.r.. the domain S(3)
Definition
2 states that for any domain and j. the distributions of the domain-specifie latent pk (R(X) and p(5) (R(X)) must be completely different. For instance, following our mentioned example, given fish images from i and that are sketch and photo domain. the mapping R(X) should extract specific information that only belongs to the domain including the shadow of the fish drawing in the sketch and ocean background information in the photo domain.
To this end, this paper aims to show that only learning domain-invariant will limit the prediction per- formance and generalization ability. Hence, we provide formal explanation for the motivation of learning domain-specifec by showing the potential drawback of only learning domain-invariance in terms of predicting class labels.
2.2
A theoretical analysis under the Information bottleneck method
Discard redundant
Domain Specific
Domain Specific x(X=
Domain Specific
Domain Specific Redundant
min ((xxiX²)
[( [
X
22
Zx
((Zx)))
Domain Invariant x(7;
Domain Invariant x
y
Relevant Domain Specific
2x:
Extract relevant into
(c)
Figure Venn diagram showing relationships between source domains represented by X X² target domain represented by X" and label Y. (a) The learning procedure of minimal and sufficient lahel-related representation in definition 3 (b) Explaining the theorem T where the conain-invarian based method provides an inferior prediction performance our proposed method that incorporates both domain-invarian I[Zr-; Y) and label- related Nomain-speciffi values made by our assumption [ (c) A case when the unseen (target) domain different x and y", while -nnvaratn information is still stable across domains, some Lomain-speciffh in souree domains become redundant information.
Given three arbitrary random variables A, B, and C, let us use I(A; B) to represent mutual information between A and B: (A; B C)) to represent conditional mutual information of A and B given C: ((A) to represent entropy of A; and H(AB) random variables A given B. For simplicity, we consider the case with two source domains 5 S² (the results with multiple source domains can be naturally extended from there). We also define two
Notations.
represent conditional entropy for
3

---

corresponding random variables X ~ p() the domain S" i= 1,2.
x that are sampled from the marginal distribution Px in
Figure illustratts all the definitions and assumptions above used for our theoretical verification (in Theorem D. In particular, in that figure, each of the four colored rectangles represents an individual entropy: (x²) for domain S is in blue, H (X²) for domain S is in red, H (X") for the target domain is in black, and (Y) for the class label is in green border rectangle.
We first show the ineffectiveness of only learning domain-invariant information when compared with incorporating Jomain-specific in source domain S (and similarly with domain S²). Our justification partly relies the following assumption about the correlation between the domain- specific representation and the class label:
Assuming that there exists domainn-spcciff (1), correlates with label in domain 5 such that T(ZZ)) Y|X²) ((X1;1(x3 E1, where & 0 is a constant.
Assumption 1. (Label-correlated domain-specifee
representation z extracted by the deterministic mapping - = =R(X+) in definition which
Assumption indicates that, for the source domain St, we can learn -.²¹ = R(X!) such that 1 ;Y(x?) is strictly positive and equals to I (Xx)x where , is the specific information that correlates with the label in the domain 81, but not in the domain 5² (12). For instance, in the example mentioned in the introduction, if domain St is "photo" while S is "sketch", the value of e should be positive because the background information such as a house, the ocean also provides information to predict whether the object is dog fish without considering its conceptual drawing. This assumption is particularly valid and practically plausible and is demonstrated by several examples observed in our experiments. For instance, for the DomainNet benchmark dataset, in the real-world domain, many bed pictures contain a bed in the room or bike pictures that have bicycles parked on the street. Other examples are in PACS such as dogs in the yard or guitars lying on a table in photo and art domains. These examples are strongly related to assumption I in which specific information correlates with labels in particular domain.
We next present supervised learning frameworks under the umbrella of the information theory [3) and the information bottleneck method (5) that generalizes minimal sufficient statistics to the minimal (i.e., less complexity) and sufficient (i.e, better fidelity) representations. The learning process of such representations is equivalent to solving the following objectives:
Definition 3. (Minimal and sufficient representations with label (14)). Let Zx1 = G(X+) is the output of deterministic latent mapping G. A representation sup is said to be the sufficient label-related representation and Zzup' is said to be the minimal and sufficient representation if:
sup (2x:; Y) and Zsup- argminI X²) s.t. [(Zaup Y) is maximized. G
Zzup
The learning procedure for definition 3 is illustrated in Figure D (a). The method is equivalent to employ compressed representations reduce the complexity (redundant information) of I (Zxx; X²) by minimizing and providing sufficient representation to class label Y by maximizing / 1(2x1; x Y). Similarly and motivated by multi-vier view information bottleneck settings (16), present the objective of learning sufficient (and minimal) representations with domain- domain-invarian invariant information in the below definition:
Definition 4. (Minimal and sufficient representations with Comain-invariane
(16)). Let Zx
Q(X") is the output of deterministic domain invariant mapping in the definition ! Then Z₁
is said to be the sufficient domain-invarian representation and 7, is said to be the minimal and sufficient representation if:
Z1 I and Z₁- (ergmnn((2;;X)) s.I. T(Z;; X2) is maximized. 2,
Q
Definition introduces a learning strategy for
across domains (or views) that preserves shared information across two domains by maximizing I I(Zx:; X²); and also reduces
specificity (redundant information) of the domain 5 by minimizing I (Zx X1). We next present a lemma about the conditional independence between the latent representation Zxi and both the label Y and the random variable X when Q, R, and G are deterministic functions of the random variable -

---

x)) is a Dirac delta function. then the following con- ditional independence holds: Zxi(X and X² 1T Zx: [x' inducing a Markov chain X² 4 Y ++ X xxx.
Lemma 1. (Determinism (14)) If P(Zxı
The proof of Lemma i provided in Appendix A.1
Lemma simply states that Zx1 contains no more information than X Now, we show the ineffectiveness of only learning domain-invariant approach, based on the existence of the label-related domain-specific in the following theorem:
Theorem 1. (Label-related information with Nomuin-specificity Assuming that there exists domain- specific value Eι > O in domain St (see Assumption [). the label-related representation based learning approach (i.e., using 7spu and Zssup) ) provides better prediction performance thun the domain-inwe domain-invariant invariant representation based method (i.e., using 7, and 7 Formally,
I(x+:Y)= [uzupi Y) I [uup
I() + I(Zr,;Y).
The proof of Theorem T mainly based on the result of Lemma] T and is provided in Appendix A.2
The visualization of Theorem [ depicted by Figure [ (b), where the domain-specific value for domain S E2 is obtained in the same way as S1. It indicates that if an existing domain-specific representation has the positive corresponding information value e, the domain-invariance-boe learning method provides an inferior prediction performance to our proposed method that incorporates both domain-invariant and label-related domain- speciffc.
Now, Theorem D suggests that besides optimizing a comainn-nvariian mapping Q as usual, we should jointly optimize domain-specific mapping R to achieve better generalization performance. However, in domain generalization, we are not allowed to access the target domain for training and must use Q and R from source domains. As pointed out in (0) 7), although domain-invariant might be the same because it is unchanged across source domains, there is no guarantce whether this domain-specific information on the source domain is relevant to the target domain while making the prediction. Figure 1 (c) illustrates this case when the target domain has different X and Y then some extracted comain--pecifee from source domains become redundant information. Therefore, the next raising question is how to learn domain Nomain-invariant and domain-specinf effectively.
To handle these shortcomings, we next propose unified framework that jointly optimizes both Q and R by disentangling their feature representation. In particular, the deterministic mapping Q is optimized by adversarial learning to extract useful domain-invariane features across domains. Meanwhile, by leveraging the transfer weight representations from meta-source domains to adapt to a meta-target domain, we apply meta-learning deterministic mapping R to force it to extract relevant domain-specific features of the target domain to improve generalization ability.
2.3 Algorithm: meta-- Specific-Domain Invariant MDDSDI)
So far, we have discussed the main ideas of our proposed method. Here we discuss implementation details for our MDSDD apSooaaprocchh
Figure shows the graphical model and overview of our murDDDD] framework. In particular, our unified representation Z, Q9 (X); a domain-specific representation Zg = Re,, (X); ; a domain discriminator Den, Zi - I,N; a domain classifier Dons Zg + 1, N and classifier Fer Z, @ Zs y. We also denote domain random variable by D, sample space by DD and outcome by d.
network consists of the following components: a
Domain-Invarian and Domain-Specifie Extraction. The defined in Definition I is obtained by using an adversarial training framework (3). in which the domain discriminator DoD, tries to maximize the prediction probability of the domain label from the latent Zr, while the goal of the encoder Qit cannot discriminate the domain of X. This task can be performed by solving the following min max game:
representation Z
map the sample X the latent Z,, such that Døp,
min max [Tz, EE,,s-x. X (d log Dr(((=))))
89 8D,
To extract the domain- domain-specifie 7s defined Definition 2 we propose the use of the domain classifier Depg that is trained to predict the domain label from Zs. The corresponding parameters 0D and
5

---

Z1
D1
D
d
Adversaril
X
0Q
X
F
Meta
7
PR
n
R
Zg
Dg
D
(8)
[b]
including: domain- invariant Z is optimized via adversarial training with domain discriminator D,, domain-specific 2 is optimized via domain classifier Ds. these latent Z and Zy disentangled by using covariance matrix. To push them to contain label information, these latents integrated into classifier F which is optimized via cross-entropy with the label Y. To make the model able adapt specific information from source unseen domain while still remaining
Figure The graphical model (a) and overall architecture (b) for our proposed mDSDI I
information across domains, we additionally push Zş through meta-learning
procedure.
O are, therefore, optimized with the objective function below:
min (Lzg = -r,,d~x log Ds(R))))]
Bpg Ap.
Disentanglement between InvarnnInvariant and Domain-Specifie. The disentanglement condi- tion between two random vectors Z₁ and Zs can be solved by forcing their covariance matrix, denoted by Cov(Z,, Zs) close to 0. detailed discussion of disentangled two representations is provided in Appendix B.3 The related parameters (fq On) then updated in the following optimization
problem:
min (LD = Ex~x [l]Cov(Q(#)) R(x)))) 8q
where || 1l2 is the L2 norm.
Sufficiency of domain-specifie and comain-invariian w.r.t. the classification task. The goal of the classifier F parameterized by OF is to predict the label of the original sample X based the domain-invariant Z, and -ppaifi Zg, i.e.,
y= Fop (Zi @ 7s):
(4)
where denotes the concatenation operation. Then, the training process of F is then performed by solving
min 0gg 9p
(LT = -E log F(Q(z),
(5)
Meta- Training for Domain-Specific Information. To encourage the domain-specific representation Zs adapt information learned from the source domains to the unseen target domain, introduce the use of meta-learning framework (II) targeting a robust generalization. Note that the domain- invariant feature Z, remains during the meta-learning procedure. In particular, each source domain Sm. E 1, N is split into two sub-domains, namely meta-train Smr and meta-test Sme- The domain- specifcc parameters 0R and the classifier parameters 0F are then jointly optimized as follows:
min {LTm == f (w -f (w, Smr) Sme)}.
(6)
where w (0R, 0F) and
(w, (ww) - Ez.g log F(Z1, R(x))]
Training and Inference. The pseudo-code for training and inference processes of our proposed
mDSDI framework is presented in Algorithm Each iteration of the training process consists of two steps:
i) First, we integrate the objective functions function La defined as follows:
and to construct objective
min max (LA = L2, + Leg + +DDLD + Lr), 0g. 0g.0 0n On,
(8)
where λz, Azo and Ap are selected as the balanced parameters.

---

ii) The second step is to employ meta-training to adapt task-related domain- specific from source domains to unseen domains. In each split, then the gradient transformation step from meta-train domains to the meta-test domain is performed by solving the optimization problem (6).
the meta-train and meta-test are
Algorithm 1: Training and Inference processes of MDSDD
Training Input: Source domain S encoder Q8 Røg. domain classifier Dep, Deps for Zr, Zs, task classifier Før, batch size B, learning rate 1). Output: The optimal: Qea Rig Fōps
for ite + iterations do Sample Sy with a mini-batch B for each domain S(9; Compute LA using Eq. (8) and perform gradient update Veg.0n.0d, Ong Le with f).
jor = 1 + N (number of source domains) do
Split Meta-train Sz/j. Meta-test Sj:
Meta-train: Perform gradient update 0F by minimizing Eq. with SByj and yi
Meta-test: Compute Irr,, using Eq. (6) with Sj and updated gradient from Meta-train; Meta-optimization: Perform gradient
gradient update V Ven.0p Io.,. with vy:
end end
Inference Input: Target domain S" optimal: Qao Rig. Før Output: Y using Eq. (4)
3 Experiments
3.1 Experimental settings
Dataset. To evaluate the effectiveness of the proposed method, utilize 7 commonly used datasets
including: ColOred-MNIST (18): includes 70000 samples of dimension (2, 28, 28) in binary clas- sification problem with noisy label, from MNIST over 3 domains with noisy rate d E {0.1. 0.3, 0.9}. RotASed-MNISST [9); contains 70000 samples of dimension (1, 28, 28) and 10 classes, rotated from MNIST over 6 domains E {0, 15, 30, 45, 60, 75]. VLCS (20): includes 10729 samples of dimension (3, 224, 224) and 5 classes, over 4 photographic domains d (Caltech111, LabelMe, SUN09, VOC2007). PACS (2): contains 9991 images of dimension (3, 224, 224) and 7 classes, over 4 domains fartpaint, cartoon, sketches, photo). Office-Home 21: has 15500 daily images of dimension (3, 224, 224) and categories, over domains d E art, clipart, product, real]. Terra Incognita (22:: includes 24778 wild photographs of dimension (3, 224, 224) and 10 animals, over camera-trap domains d (L100, L38, L43, L46). and DomainNet (23): contains 586575 images of dimension (3, 224, 224) and 345 classes, over domains E (clipart, infograph, painting, quickdraw,
real, sketch}. The detail of each dataset provided in Appendix C.1
Baseline. Following DomainBed (24) settings, we compare our model with 14 related methods in DG which are divided by 5 common techniques, including: Standard Empirical Risk Minimization: Empirical Risk Minimization (ERM (25)); domaiin-peecfic-tearniig Group Distributionally Robust Optimization (GroupDRO (26), Marginal Transfer Learning (MTL (7))), Adaptive Risk Min- imization (ARM (28)); Meta-learning: Meta-Leernin for DG (MLDG (I);; domain-invariant- learning: Invariant Risk Minimization (IRM (18), Deep CORrelation ALigmment (CORAL (9h).
Maximum Mcan Discrepancy (MMD (30)), Domain Adversarial Neural Networks (DANN (II).
Class-conditional DANN (CDANN (32)) Risk Extrapolation (VREx (33); Augmenting data: Inter-
domain Mixup (Mixup 34
35 36)). Style-Agnostic Networks (SagNets (37)), Representation Self
Challenging (RSC 38) The detail of each method is provided in Appendix
We use the training domain validation set technique as proposed in DomainBed (24) for model selection. In particular, for datasets, we first merge the raw training and validation, then, we run the test three times with three different seeds. For each random seed, we randomly split training and validation and choose the model maximizing the accuracy on the validation set, then compute performance on the given test sets. The mean and standard deviation of classification accuracy
7

---

from these three runs are reported. We evaluate generalization performance based on backbones MNIST-Come Cet (24) for MNIST datasets and ResNet-55 (39) for nON-MNIST datasets to compare with the mentioned methods. Data-processing techniques, model architectures, and changes of objective functions during training are presented detail in Appendix C.3 C.5 All
source code to reproduce results are available at https:/gghhub.ccom/nnnnReeeeee
3.2 Results
Table 1: Classification accuracy (%) for all algorithms and datasets summarization. Our mDSDI method
achieves highest accuracy average when comparing 14 popular DG algorithms across 7 benchmark datascts.
Method
CMNIST
RMNIST
VLCS
PACS
OtticeHome
TerraIne
DomainNet
Average
ERM (5
51.5.0.1
98.0:1000
77.510.44
85.5+0.2
66.520.3
16
40.9=0.1
66.6
IRM [8
52,0#0.1
97.740.1
78.5:0.5
83.540.8
64.34222
47.6:0.8
33.912.8
65.4
GroupDRO 26
52.110.0
98.0+0.00
76.720..
84.4-0.8
66.0:0.7
43.2:11.1
33.3-0.2
Mixup R4 553 36
52.1+002
98.0+0.11
77.4+0.6
84.6+0.6
68.1+0.3
47.9+0.8
39.2+0.1
66.7
MLDG 11
51,5++0.1
97.9+0.0
77.2+0.4
84.9+110
66.8#0.6
47.7+0.9
41.2=0.1
66.7
CORAL 29
51.5+0.11
98.0++0.1
78.8++0.6
86.2+0.3
68.7+0.3
47.6=110
41.5++011
67.5
MMD
51.5+002
97,910.0
77.510.9
84.620.5
66.320.1
42.2-11.6
23.419.5
63.3
DANN (31)
51.5++0.3
97.8+0.1
78.6+0.4
83.6#0.4
65.9=0.6
46.7=0.55
38.3=0.1
66.1
CDANN (32
51.7.0.1
97.910.1
77.510.1
82.620.9
65.8.113
4..8-11.6
38.320.3
65.6
1
51.4+0.1
97.9+0.0
77.2+004
84.6+0.5
66.400.5
45.6+11.2
40.6+0.1
SugNets (37
51.7=0000
98.0:10.0
77.8+0.5
86.3:0.2
68.120.1
48.6=11.0
40.320.1
67.2
ARM 28
56,2+0.2
98.2+-0.1
77.6+0.3
85.1+0.4
64.8+0.3
45.5+0.3
35.5+0.2
66.1
VREx (33
51.8+0.11
97.9+0.11
78.3+00.
84.940.6
66.420.6
46.4-0.6
33.6=2.9
65.6
RSC (38
51.7+0.2
97.6+0.11
77.1=0.5
85.2+0.9
65.5=0.9
46.6=110
38.9=0.5
mDSDI Ours)
52.240022
98.010011
79.0
86,2#0.2
69.2=0.4
48.1
42.8-0.1
67.9
Table 1 summarizes the results of our experiments on 7 benchmark datasets when compared with mentioned methods. The full result per dataset and domain is provided Appendix C... From these results, we draw three conclusions about our ourImDSDI model:
Our MDSDI still preserves domain-invariane information. We observe in some target domains which have background-les images and assume only contain invariaan information such as -NISS, Notated-MNISS or Terra Incognita (similar observation in cartoon or sketch in PACS, clip-art or product in OfficeHome, and quickdraw in DomainNet. Full results in Appendix C.4). our DDSDI model still achieves competitive results with other baselines (e.g., 52.2% in Colored- MNIST, 98.0% in MNNIS, and 48.1 3..1% % in Terra Incognita) which are based on domain- invariant-learning techniques such as DANN, C-DANN, CORAL, MMD, IRM, and VREx. Those results demonstrate the effectiveness of our adversarial training technique for extracting domain- invariant features. Furthermore, due to considering disentangled domain-ine iiraainnniaarant and domain- specific latent, even in situations where the samples do not have domain-specifie, our model still performs well by retaining informative -ovaiininvariiace.
Our DSDI could capture the usefulness of domain-specific information. In contrast, we ob- serve that in some target domains that have relevant domain-specitf with source domains such as landscape background of the object class from photographic pictures in VLCS (similar observation in PACS such as dogs in the yard or guitars lying on a table photo and art domain, bed in the room or bike parked on the street in the Art and Real-world domain of OfficeHome. Full results in Appendix C.4). DDSDI VLCS, 86.2% in PACS). This means that our tion better but also our domain-specific ones cover helpful information in special scenarios such as backgrounds and colors related to objects in the classification task. Moreover, when comparing with other domain-specific based techniques such as GroupDRO, MTL, and ARM, the results showed that domain-specific features learned by meta-training from our model are more helpful than theirs and have captured useful domain domain-specific features from those object- background relations.
significantly higher results than other methods (e.g.. 79.0% in features not only support generaliza-
Extending beyond the invariance view to usefulness domain-specific information is important. As shown in Table [ our [DDDDI has the highest average number with 67.9% (highlighted with statistically significant according to a t-test at a significance level a 0.05). The reason why our model outperforms other baselines could be explained by the fact that their domain-invariant methods are not able to capture domain-specing information, and so have poor performance. Meanwhile, when comparing with other domain-specific based methods, their models only concentrate on domain- information to our techniques in some
specific techniques, and provide inferior
8

---

background-les images. In contrast, due to considering disentangled domain-invann invariant and domain- specific features, and having the right strategy to learn each latent, our model captures both this useful information, hence, outperforms their results. Not only has the highest average number, but our method also dominates other methods on known large-scale dataset such as 69.2% in Office-Homme or 42.8% in DomainNet. This implies that besides the essential combination between domain- invariant and domain-specific, when the number of datasets increases, our method can extract more relevant information for complex tasks, such as classifying 345 classes in DomainNet. These results also mean that our model has balance between informative domain-invariant and domain- specific features adapt better different environments than others, therefore showing the highest average in all settings.
3.3 How does m D work?
Domain-invariant (Clesses)
Domain-invariant (Domains)
Domain-speoltic (Classes)
Domain-specific (Domains)
Figure 3: Feature visualization for domain-invariant: (a): different colors represent different classes; (b): different colors indicate different domains. Feature visualization for domain-specific: (c): different colors represent different classes; (d): different colors indicate different domains. Source domain includes: art (red), cartoon (green). sketch (blue) while target domain photo (black) in domain plots. Best viewed color (Zoom in for details).
To better understand our framework, visualize the distribution of the learned features with t- SNE (40) to analyze the feature space of comain-invariaat and domain-specifie. As shown in Figure on PACS Dataset, our of the domains (see Figure 3 (b)). However, these domain-invariant features still make mistakes on
extractor can minimize the distance between the distribution
the classification task, indicated by mixture of points from different class labels in the middle (see
Figure 3 (a)), and many of these points are from the target domain (black color in the Figure 3 (b)).
Meanwhile, the domain-specific representation better distinguishes points by class label (see Figure 3
(c)). More importantly, the photo domain's 's specific features (black) are close to the art domain (red)
(see Figure 3 (d)). This is reasonable because only these two domains include backgrounds related
to the object class. It implies that meta-training in our model well learns specific features that can be adapted the new unseen domain.
3.4 Ablation study: Important of mDDDI on the Background- Colored-MNoEE dataset
This section examines our system design by checking its performance under different settings in the real scenario with our generated dataset (a similar experiment with PACS benchmark dataset is in Appendix C.6).
dataset. Figure 4 illustrates our Baakground-Colored--aa erated from the original MNIST. We assume the domain-invarian is the digit's 's's sketch and design the dataset so that the background color is domain-specific. As result, on the unseen domain, domain-specifé will be useful for the classification task. Specifically, the dataset includes three source domains di,, different by digit's color {red, green, blue }. generated from a subset with 1000 training images for MNIST per each domain. In each source domain, the background color is the same for intra-class images but different across classes. the target domain, 10000 testing images of MNIST are colored for one target domain die with digit color (orange), ). In this domain, each class's background color is similar to the same class's background color in one of three source domains.
Background-Colordd-MNNN
gen-
Importance of DDDD We aim to prove the combination of learning disentangled representation Nomain-specific, domain-invariant, and meta-training on domain-specific are important in this sce- nario. To do compare model under nine settings: learning domain domain-invariin invariant only (DI),

---

learning comainnsspecinn only (DS), meta- mheta-training on Comminn-ivvrriin (DI-Meta), meta-training on domain-specinf (DS-Meta), combination of tanglement loss Lp (DSDI-Withon) I.D). combination of domain-invarian and domain-specitfe without meta-training
and domain-specifia without disen-
Meta), meta-training on both representation Z, and Zs (DSDI- Meta), meta-training on domain-invarian without domain-specinn (DSDI-Meta DI) and our proposed framework (mDSDi-Meta DS), which is meta- vecta-training on domain-specific without domain-invariant Table2showws that our model is the best setting with 89.7% It proves that combining domain-invarinn and domain-specific is crucial by dominating the settings with only domain-invariant or domain- specific (DI, DI-Meta, DS, DS-Metal Regarding disentangling worth noting that without disentanglement loss Lp. the model only achieves 81.4%. which is lower than mDSDI It reveals adding disentanglement loss Lp is essential to boost model performance. Compared with meta-training on both, which only reaches 82. 1% and on domain-invarisn it implies that meta-training is necessary but only for the domain-specifie, and our arguments reasonable. It also shows that our combination with domain-invariant and domain-specifie is not easy: deep ensemble between two neural networks, but existing deep-down insight in framework in DG when compared with MDSDI HIDSDi-Without Without Meta which can be seen as a type of ensemble, is only around 80.4%.
representations Z₁ and Zs, it is
are 79.0%,
Table 2: Classification accuracy (%) on Background- Colored-MNIST. Ablation study shows impact of domain-invariant when combined with meta-training on domain-specific our method.
Source domains:
Blue
S
Green
Method
Accuracy
Red
DI
65.714.6
DI-Meta
63.611..1
DS
70.7=4.88
Target domain:
DS-Meta
75.3
Orange
DSDI-Without LD
81,4+2.6
DSDI-Without Meta
80.4
Figure 4:
Beckground-Coooced-NNii Dataset,
DSDI-Meta
82,1+114
where source domains include (rdd, green, blue) digit colors and target domain has (orange) color.
DSDI-Meta DI
79.0+2.3
mDSDD Meta DS (Ours)
89.7++0.8
Conclusion and Discussion
Despite being aware of the importance of domain-specifie information, little investigation into the theory and a rigorous algorithm to explore its representation. To the best of our knowledge, our work provides the first theoretical analysis to understand and realize the efficiency of domain-specific information in domain generalization. The domain-specing contains unique characteristics and when combined with domain-invariant information can significantly aid performance on unseen domains. Following our theoretical insights based on the information bottleneck principle, we propose mDSDI algorithm which disentangles these features. We next introduce the use of the meta-training scheme to support domain-specitn to adapt information from source domains to unseen domains. Our experimental results demonstrate [DDDII brings out competitive results with related approaches in domain generalization. In addition, the ablation study with our further illustrated and demonstrated the efficiency of combining domain-invariant and domain-specific via our proposed MDDDI Our theoretical analysis and proposed MDSDD framework can facilitate fundamental progress in understanding the behavior of both domain fomain-invariant and domain-specnt representation in domain generalization.
Toward robustness algorithm that can effectively learn both features, there are certainly many challenges that remain in our paper, for example, theorem to explain when domain- sppeciii may hurt performance in the unseen domain, a stronger connection between theory in implementation, a method to make two representations to be non-linearly in- dependent as well as lower computational cost of the covariance matrix. In future, we plan to continue tackling these challenges to provide better understanding and learning framework in domain generalization then extending to broader settings of transfer learning. A detailed clarification, discussion, and plausible methods in the future work are provided in Appendix B
and domain-specitfe
10

---

References
[1] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifi- cation tasks to new unlabeled sample. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24, pages 2178 2186. Curran Associates, Inc., 2011.
[2] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 1342-5550 2017.
[3] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018.
[4] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multido- main discriminant analysis, 2019.
[5] Maximilian Ilse, Jakub M. Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders, 2019.
[6] Ching- Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via ionainninvariant representations, 2020.
[7] Prithvijit Chattopadhyay, Yogesh Balaji. and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization, 2020.
[8] Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On learning invariant representation for domain adaptation. CoRR, htss/1901.09443 2019.
(9] Fredrik D. Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant representations, 2019.
[10] 7. Ding and Y. Fu. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, 27(1):304-313, 2018.
[11] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learnig for domain generalization, 2017.
[12] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368-377, 1999.
[13] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc, of the 37-th Annual Allerton Conference Communication, Control and Computing, pages 368-377, 1999.
[14] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from multi-view perspective. In International Conference on Learning Representations, 2021.
[15] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations, 2018.
[16] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. CoRR, abs/2002.07017, 2020.
[17] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common-specific low-rank decomposition, 2020.
[18] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization, 2020.
[19] Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. CoRR, 508.07680, 2015.
11

---

[20] Chen Fang. Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web imgges for softening bias. In 2013 IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[21]
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation, 2017.
[22] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. CoRR, htss/180704975, 2018.
[23] Xingchao Peng, Qinxun Bai, Xide Xia, Zijum Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation, 2019.
[24] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. CoRR, abs/2007.0144 01444,, 2020.
[25] Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
[26] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020.
[27] Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning, 2021.
[28] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group distribution shift, 2021.
[29] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation, 2016.
[30] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018
[31] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversariy training of neural networks, 2016.
[32] Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representation, 2018
[33] David Krueger, Ethan Caballero, Heem--enrikk Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. -ut-of-distributtin generalization via risk extrapo- lation (rex), 2021.
[34] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup, 2019.
[35] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. 2020.
[36] Yufei Wang, Haoliang Li, and Alex C. Kot. Heterogeneous domain generalization via domain mixup. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2020.
[37] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias, 2021.
[38] Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross- domain generalization, 2020.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
12

---

[40] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):557--266 2008.
[41] Thomas M. Cover Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006.
[42] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):223-77, 2012.
[43] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
[44] Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. CoRR, abs/ las//9977002755 2019.
13


Reviews:
[{"review_id": "nK1BcPWFAeO", "reviewer": "Reviewer_xiMp", "summary": "The article describes an approach for domain generalization (DG) that exploits domain-specific features. In particular, the model uses two feature extractors, one extracting domain-aligned features (adversarially learned), and one extracting domain-specific ones (by minimizing a domain classification loss). To enforce that the extracted domain-specific features are useful for unseen target domains, a meta-learning procedure is adopted. Experiments on the recent DomainBed benchmark show that the approach outperforms the state of the art on average, achieving consistently top or comparable results across all datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the method described in the paper interesting, exploiting both the information shared across domains (through the domain-aligned features) and generalizable domain-specific information (through the domain-specific features extractor). The results clearly show the effectiveness of the approach in the challenging DomainBed benchmark. The article has some weaknesses related to referencing relation to previous works and comparisons with closer baselines. However, I do not deem them to be major, thus I lean toward a positive score. Below I detail my comments.\n\nStrengths:\n+ While the use of domain-aligned features has been widely explored in the DG literature (e.g. [2,3]) it is still unclear how to make effective use of domain-specific ones. Previous works integrated them through ensembles (e.g. [43]) or meta-learning techniques (e.g. [11,a]). However, few papers consider using both domain-aligned and domain-specific features (e.g. [7]). This paper is an effective step in this direction, learning to integrate domain-specific features into domain-aligned ones through a meta-learning procedure. \n\n+ The results on DomainBed are impressive in some challenging scenarios. For instance, for DomainNet the proposed approach achieves 42.8% accuracy vs 41.5 of the best competitor. Also in Office-Home, the advantage over the best competitor is +0.5% and in VLCS (here ERM already is a strong baseline, due to the limited domain-shift), the method improves the best competitor by more than 1%.\n \n+ Analyses (e.g. Fig. 3, Table 2) clearly show the impact of each of the proposed components.\n\nWeaknesses:\n\n1. The paper is closely related to [7], which also tries to exploit both domain-specific and domain-aligned features. However, [7] is not present in the experimental comparisons, similarly to [43], another effective approach that uses an ensemble of domain-specific parameters. While they are not present in DomainBed, performing comparisons in their same settings and splits (e.g. PACS [7,43] following [2], Office-Home [43], DomainNet [7]) would strengthen the results, showing that the proposed approach is actually the state of the art for exploiting domain-specific features.\n\n2. Following on the previous, the description and relations with the most related works in the DG literature is limited to the final part of the introduction. It would have been helpful to add a specific section to highlight both the existing domain alignment methods (e.g. [2,3]) and domain-specific architectures [43] as well as methods exploiting both specific and agnostic features [7] and meta-learning techniques [11]. Clearly describing the related works and the differences with the proposed approach is fundamental to highlight the contributions and should not be limited to few paragraphs of the intro (given that a reader may even not be familiar with the DG problem).\n\n3. In Table 3, the ablation study is conducted in a dataset created by the authors. This choice may seem suspicious to readers since the same dataset is not present in the experimental analysis and may appear specifically created for the ablation study. I would suggest either incorporate the dataset into the wider experimental comparisons of Table 1 or replacing Table 2 with the same ablation study but conducted on PACS (Table 9 of the supplementary).\n\n4. Definitions 1 and 2, as well as the alignment objective of Eq. (1), consider only the marginal input distribution, without considering the joint distribution of input and labels. This assumes the presence of a covariate-shift among the domains which may actually be limiting in practice due to the strong assumptions needed for achieving a good cross-domain alignment [b]. Have the authors considered applying conditional alignment techniques to improve the results?\n\n5. Footnote 1, describing the differences of the approach w.r.t. MLDG [11] should be integrated into the main text, to better highlight the contribution of the approach w.r.t. existing meta-learning techniques. \n\nReferences:\n\n[a] Li, Da, et al. \"Episodic training for domain generalization.\" ICCV 2019.\n\n[b] David, Shai Ben, et al. \"Impossibility theorems for domain adaptation.\" International Conference on Artificial Intelligence and Statistics 2010.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The article does not discuss either limitation of the approach or possible societal impact. For the latter, there are not clear impacts (other than performance guarantees in non-tested scenarios), but limitations of the model (e.g. need of domain labels) should be better discussed in the conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "iktYnt8oTI", "reviewer": "Reviewer_SAJX", "summary": "This work claims that the domain-specific knowledge also can make contributions to the classification of unseen target images in domain generalization setting, and attempts to prove the point from information theory. Moreover, this work develops the corresponding algorithm to solve domain generalization and carries out experiments on several benchmarks by comparing with many baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work claims that the domain-specific knowledge also can make contributions to the classification of unseen target images in domain generalization setting, and attempts to prove the point from information theory. Moreover, this work develops the corresponding algorithm to solve domain generalization and carries out experiments on several benchmarks by comparing with many baselines.\n\nHowever, the major concern is the limitation of application based on the proposed idea.  Concretely, can this idea also work for the situation where the domain-specific information (background, color) of unseen target images is totally different from that of multiple source domains? In addition, although the theoretical analyses illustrate the domain-specific knowledge involves semantic information related to category, these analyses do not explain why the application of domain-specific knowledge can improve the generalization ability of model. Moreover, it is not clear why the developed algorithm can explicitly decompose the features into domain-specific representation and domain-invariant ones. Finally, it would be better to compare this work with the recent work [1] on domain generalization to verify the  advantage of the proposed method.\n\n[1] Xu, Qinwei, et al. \"A Fourier-based Framework for Domain Generalization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The major concern is the limitation of application based on the proposed idea.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "Obju5CvUQcm", "reviewer": "Reviewer_sfZU", "summary": "This paper proposed a meta-learning based domain generalization method that aims to extract both domain-invariant and domain-specific information of the inputs. The key idea is to apply two domain-adversarial training losses and then use cross-covariance to minimize the correlation of these two features. Experiments on several datasets have shown its effectiveness.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Inconsistency between main idea and algorithm: The main theorem is based on mutual information, while the algorithm is based on domain-adversarial learning. However, it remains unclear why adversarial training can replace mutual information, or it is not feasible to match such algorithm with the idea. This makes it inconsistent.\n\n2. Algorithm is not novel: it is common to disentangle the domain-invariant and domain-specific features for multi-domain learning. This algorithm is an incremental version of MLDG where they further apply two adversarial losses.\n\n3. Missing several related work: As I stated above, there are tons of works that uses disentanglement to learning two kinds of features. However, authors did not cite or discuss with them in the paper. See the reference section.\n\n4. Improvements are not significantly: The results of CORAL (second-best) in table 1 are only 0.6% worse than this method, but it remains much easier than this one. You should also compare their actual running time.\n\nReferences:\n\n[1] Learning Disentangled Semantic Representation for Domain Adaptation, IJCAI-19\n\n[2] Domain Agnostic Learning with Disentangled Representations， ICML-19\n\n[3] A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation, NeurIPS-18\n\n[4] DISENTANGLED MULTIDIMENSIONAL METRIC LEARNING FOR MUSIC SIMILARITY, ICASSP-20\n\n[5] Learning Causal Semantic Representation for Out-of-Distribution Prediction. arXiv 2011.01681\n\n[6] DIVA: Domain Invariant Variational Autoencoders. PMLR 2021.\n\n[7] Towards Better Understanding of Disentangled Representations via Mutual Information. arXiv 1911.10922\n\n[8] A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms. ICLR-20", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ESQnEx-fTuZ", "reviewer": "Reviewer_FWcu", "summary": "This paper studies the importance of relying on both domain-specificity (DS) and domain-invariance (DI) in domain generalization (DG) problems. It formalizes the problem through the lens of information theory - more specifically, through the lens of the information bottleneck theory. It advocates for learning both DS and DI representations, and proposes a well motivated method that accommodates this need. Results on the standardized DomainBed benchmark are strong, and proper ablation studies are provided.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper: it proposes a well-formulated analysis on the importance of relying on both DS and DI representations in domain generalization. The proposed ideas are well exposed both from a theoretical perspective and from a more practical view. While some parts result rather mathematically heavy for \"more empirical\" readers, the Authors made the effort of making their insights accessible to the broader audience - for example, I believe Figure 1 is a very effective one.\n\nThe proposed method is reasonable, and supported by strong results on a recently released testbed - DomainBed. The importance of each contribution is supported by a detailed ablation study (Sec. 3.4). While the exploitation of both DS and DI representations is not novel (previous works have been properly cited), I believe that this work brings enough contributions to consider the proposed approach original.\n\nI think the work could be improved in a couple of directions:\n\n- After having finished Sec. 2.2, I was expecting to find a method more connected with information theory. It is true that domain classifiers within an adversarial training framework can be used as a proxy for the Mutual Information, but some more sound approaches would be applicable, too. For example, one could use MINE [Belghazi et al., ICML 2018] to both estimate the MI between two sources and directly optimizing w.r.t it - since the module is fully differentiable. Have the Authors thought about narrowing the gap between theory and implementation?\n\n- I have found the disentanglement part very difficult to follow. What does it mean \"the disentanglement condition between two random vectors $Z_I$ and $Z_S$ can be solved by forcing their covariance matrix [...]\"? Is it possible to explain this intuition in other terms, since we still need to converge towards a proper definition of \"disentanglement\"?\n\nSummarizing, I believe this is a good work, that can definitely be interesting for the NeurIPS community. I look forward reading the Author response and discussing with other Reviewers.\n\nSome minor comments below:\n\n- Should Definition 2 not also take into account the labeling? Domain specificity is an interesting property if the distributions are different within the same classes, otherwise it can be seen as a trivial property.\n\n- How realistic is Assumption 1 in practice? I think it is worth a couple of lines.\n\n- Line 129: \"learning procedure\"?\n\n- Meta-learning could be effective for both DS and DI, why just DS?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. It would be valuable a discussion about possible situations in which related works perform better. For example, can the Authors imagine some situations where instead learning just DI representations is a better choice? For what concerns societal impact, DG's goal is to make machine learning systems safer when deployed in the wild, hence something along these lines may be proper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "/"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received primarily positive reviews. The authors provided sufficient additional details during the rebuttal to reduce most concerns about the work. There was originally some concerns about the gap between the presented theory and the implemented algorithm reiterated by a few reviewers. The authors should be sure to include their clarifications presented during the discussion phase in the final version of their paper. The authors should note that the AC has read the concerns brought up by SAJX as well as the author rebuttal and is convinced by the responses provided. Once again, please do provide the discussions and clarifications in the final version. There was substantial new clarifying information during the discussion phase and this is needed to be included in the final draft. 
Decision: Accept (Poster)