Gated Domain Units for Multi-source Domain Generalization
Authors: Simon Föll
Alina Dubatovka
Eugen Ernst
Siu Lun Chau
Martin Maritsch
Patrik Okanovic
Gudrun Thaeter
Joachim M. Buhmann
Felix Wortmann
Krikamol Muandet
Venue: Accepted by TMLR ()

Abstract: The phenomenon of distribution shift (DS) occurs when a dataset at test time differs from the dataset at training time, which can significantly impair the performance of a machine learning model in practical settings due to a lack of knowledge about the data's distribution at test time. To address this problem, we postulate that real-world distributions are composed of latent Invariant Elementary Distributions (I.E.D) across different domains. This assumption implies an invariant structure in the solution space that enables knowledge transfer to unseen domains. To exploit this property for domain generalization, we introduce a modular neural network layer consisting of Gated Domain Units (GDUs) that learn a representation for each latent elementary distribution. During inference, a weighted ensemble of learning machines can be created by comparing new observations with the representations of each elementary distribution. Our flexible framework also accommodates scenarios where explicit domain information is not present. Extensive experiments on image, text, and graph data show consistent performance improvement on out-of-training target domains. These findings support the practicality of the I.E.D assumption and the effectiveness of GDUs for domain generalisation.

Published in Transactions on Machine Learning Research (09/2023)
Gated Domain Units for Multi-source Domain Generalization
Simon Föll!*
sfoell@ethz.d ch
Alina D
alina.dubbttonaaaiinff
ethn etha.
Eugen Ernst $1
euernst@kit. edu
Siu Lun Chau
siu-lun. de
Martin Maritsch'
mmaritcch attth,
Patrik Okanovic
pokanovic@etha. ch
Gudrun Thäter
gudrun. thenterraal edu
Joachim M. Buhmann
jouhmann@inffetlle etthe
Felix Wortmann
felir. worbmann@uuiiggggg ch
Krikamol 5uandet5
muandet&@cippa.de de
DDpartment of Management, Technology, and Economics, ETH Zurich, Switzerland Department of Compuler Science, ETH Zurich, Switzerland Department of Mathematics, Karlsruhe Institute for Technology, Germany "Istitute for Technology Management, University of St. Gallen, Switzerland
CISPA Helmhollz Center for Information Securily, Germany
Reviewed on OpenReview: Cttp:s//ppaaeeoooeeree//////////
Abstract
The phenomenon of distribution shift (DS) occurs when a dataset at test time differs from the datasct at training time, which can significantly impair the performance of machino learning model in practical settings due a lack of knowledge about the data's 's distribution at test time. To address this problem, we postulate that real-world distributions are composed of latent Invariant Elementary Distributions (L.E.D across different domains. This assumption implics invariant structure in the solution space that enables knowledge transfer to unseen domains. To exploit this property for domain generalization, we introduce modular neural network layer consisting of Gated Domain Units (GDUs) that learn a representation for each latent clementary distribution. During inference, a weighted ensemble of learning machines can be created by comparing new observations with the represent atins of each elementary distribution. Our flexible framework also accommodates scenarios where explicit domain information not present. Extensive experiments on image, text, and graph data show consistent performance improvement out-of-trainiie target domains. These findings support the practicality of the I.F.D assumption and the effectiveness of GDUs for domain generalisation.
"SF' and AD contributed equally this paper. TEE and ST.C contributed equally this paper.

---

Published in Transactions on Machine Learning Research (09/2023)
1 Introduction
Machine learning relies on the fundamental assumption that training and test data are independently and identically distributed (L..D.), which ensures the learning machine attains the lowest achievable risk as the sample size grows under the empirical risk minimization (ERM) framnwork (Vapnik) 1998 Schhlkopl
2019
However, numerous research and real-world applications (Zhao et al., 2018 2020; Ren et al. 2019 Taori ct al. 2020) have provided staggering evidence against this assumption; shown in D'Amour et al.
2020
In practice, the I.D., assumption is often violated due to distribution shift. (DS), which occurs where a model is applied to dataset that differs from its training data (Sugiyama & Kawanabe 2012). resulting in significantly impaired performance.
To lackle DS, recent work advocates for domain generalization (DG): Given data from multiple domains, e.g., Continental Europe hospitals, how to train a model that can generalize well to unseen domains, e.g., US hospital? (Blanchard al. 2011 Muandet et al. 2013 Zhou al. 2021a). The ability to generalize entirely new domains is crucial for the robust and safe deployment of machine learning models, particularly when unforesccable domains arise after deployment. Nevertheless, the question of how to identify the appropriate invariance that enables such generalization remains an open and unresolved issue, as noted by Guirajani & Lopez- Paze 20200
In the following, we introduce two examples from Koh et al. (2021a) to challenge common assumptions of domain definitions from the well-cstablished benchmark for DG. In camelyon17 (Bándi et al., 2019), the task is to train a model to classify a tumor (i.e., benign or malign) based on images of lissue slices from few hospitals and test the model on an unseen hospital. Here, each hospital is considered domain. However, when studying tissue slices under a microscope, the source of variation arises from diffcrences in patient population, slide staining, and image acquisition, and thus not necessarily from the hospital itself Veta et al. (2016); Komura & Ishikawa (2018); Tellez et al.] (2019). Further, in OGB-MoiPCBB (Hu et al. 2021), the task is to predict the biochemical properties of small molecules from a set of possible molecules and generalize to a set of molecules structurally different from those seen in the training set. Each scaffold (i.c., subset of structurally similar molecules) is considered a domain, thus yielding over 120,000 domains in OGB-MolPCBA. Though the definition of domains is bio-chemically motivated, this assumption causes computation overhead, making it challenging to find an invariance across these scaffolds.
Prior work has made a similar motivation to advocate for learning so-called latent domains in the combined source dataset through an unsupervised manner Deecke et al. (2022); : Matsuura & Harada 2020 Chen et al. (022a)). The motivation is to become independent of domain labels, when applying DG that learn an invariant structure across domains during training (see Section 2 for a discussion). However, this line of work does not draw connection between the latent source domains and test domains. In contrast, our paper follows the current trend in DG rescarch that establishes connections between test-time and source domains. Unlike previous work that assumes test-time domains are convex combinations of source domains, we propose that both domains are linear combinations of some latent invariant units, which can be learned induces an invariant structure in the
during training. This assumption, formally introduced in Definition [ hesot space, which are practically appealing for domain generalization (cf. Proposition 3.1. To turn theory into practice, we develop a neural network layer consisting of gated domain units (GDUs) that learn a geometrie representation of each elementary domains in the form of kernel mean embedding (KME) (Berlinet) & Thomas-Agnan, 2004 Smola et al. 2007 Muandet et al. 2017 During inference, weighted ensemble of learning machines can be created by comparing new observations with the elementary domains embeddings. We depicted our idca in Figure
We summarise our contributions as follows:
1. We propose the Invariant Elementary Distribution (I.E.D) assumption, postulating both test-time and source domains consist of latent elementary distributions that can be learned during training, and show the assumption's practical appeal for domain generalization.
We develop a modular neural network layer consisting of Gated Domain Units, each captures and learns an elementary distribution via kernel mean embeddings. These representations can thus be used to adapt the ensemble weights to an unseen domain test. time.
2

---

Published in Transactions on Machine Learning Research (09/202))
During training
During inference
extract the set of unobserved elementary distributions that remain invariant from the observed data.
weighted ensemble using similarities between unseen observation and elementary distributions.
Source
Source
Source Hospital
Unseen Hospital
Hospital
Gated Domain Units
Blemenary Domain
Elementary
Blementayy Donain
Demain
Versin-Fersic
E
Figure 1: An illustration of invariant elementary distributions in camelyton17 dataset and the concept of the Gated Domain Units (GDUs). While data sources are often considered as domains, each hospital contains heterogene subsets that are shared across sources. We can depict examples of these subsets (latent domains) in three different shapes (rectangle, octahedron, and circle) for each domain following the method of Matsuura & Harada (2020). In contrast to existing work, the invariant clementary distribution (I.E.D.) assumption advocates that these latent domains remain invariant across the hospitals. Therefore, we propose to learn representations of these elementary domains akin to mixture models, and utilise our Gated Domain Units to build weighted ensemble during inference time.
3. We verify the L.E.D assumption by extensive experiments using publicly available benchmarking WILDS. Specifically, we validate our method on image, text, and graph datascts, showing consistent improvement on aut-of-training target domains.
4. We provide an effective TensorFlow and PyTorch implementation applicable to different feature extractors such as ResNet50, DistillBERT, and GIN virtual and thus a broad audience of researchers and practitioners to enable a fast adaption of our method.
The remainder of this paper is organized follows: Section 2 outlines related work, and lays out research gaps that we aim to address. Our theoretical framework presented in Section 3 followed by our modular Gated Domain layer implementation shown in Section 4 Our experimental evaluations are then presented in Section 5 Finally, we discuss the potential limitations and future work in Section b
2 Previous Work and Motivation
DG among the hardest problems machine learning (Blanchard al. 2011
Muandet al. 2013 Arjovsky
et al. 2019). In contrast, domain adaptation (DA), which predates DG, deals with a slightly simpler scenario in which some data from the test distribution available (Ganin et al. 2015 Hence, based on the available
data, the task is to develop learning machines that transfer knowledge learned in source domain specifically to the target domain. Approaches pursued in DA can be grouped primarily into (1) discrepancy-based
DA (Sun et. al. 2016 Peng & Saenko 2018 Ben-David et al.] 2010 Fang el al. 2020 Tzeng el al.
2014 Long
et al.
2015) (2) adversary-ban
DA (Tzeng et al. 2017 Liu Tuzel 2016 Ganin et al. 2015 Long et al,
2018
and
reconstruction-base DA (Bousmalis
2016 Hoffman et al. 2018b Kim al. 2017
et al. 2017 Zhu 2017 Ghifary et 2014). In DA, learning the domain-invariann
components requires
access to unla unlabeled data from the target domain. Unlike problems in DA, where the unlabeled data from
3

---

Published in Transactions on Machine Learning Research (09/2023)
the test domains is accessible to find the right invariant structures (Ben-David et al. 2010), the lack thereof in DG calls for a postulation of invariant structure that will enable generalization Guulaajani & Lopez- Paz 2020).
To enable generalization to unseen domains without any access to data from them, researchers have made
significant progress in the past decade and developed a broad spectrum of methodologies (Zhou al.] 2021a by
Li et al.
2019
Blanchard et al. 2011). For thorough review see; e.g., Zhou et al. (2021a); Wang et al. [2021]
Existing works can be categorized into methods based on domain-invariant representation learning (Muandet
et al.
2013
Li et al. 2018b c meta-learning (Li et al., 2018a
Balaji et al. 2018). data augmentation (Zhou (2023); [aaawa et al. (2020), to name a few.
ct al.
2020 distributionally robust optimization Dai et al.
Another recent stream of research from a causal perspective includes invariant risk minimization (Arjovsky
it al.
2019
invariant causal prediction (Peters et al.) 2016), and causal representation learning (Schökkopf
ct al. domain-speeifte spurious correlations. In other words, it is postulated that "causal" features are the right kind of invariance that will enable generalization. On the other hand, latent domain discovery is another popular approach in DG. Finding pseudo-domains through unsupervised manner allows one to learn invariant feature extractors across such latent domains (Hoffman et al. 2012 Li ct al. 2020 Gong et al.
2021 Chen et al., 2022b The overall motivation here to learn the representation that is robust to
2013 Matsuura & Harada 2020). This line of work is practically appealing when collecting explicit domain labels are difficult in practice, as discussed in Appendix B.2.2
Finally, ensemble learning has been shown to be effective in DG (Wang et al.,
2021 A particular branch
of ensemble learning is domain-speeific models, which aim learn a source domain-specific model, ie., models specializing in each source domain (Piratla et al. 2020 Monteiro et al. 2021). When determining the weighting for an ensemble, it has been shown that making the weights adaptive during inference is advantageous instead of just averaging the prediction. This adaptive weighting adjusts the ensemble weights during inference based on how similar each source domain is to the test domain Zhang et al. (2023); Gao
ct al. (2008); Yao & Dorctto (2010). However, these domain-specific models require domain labels to train the domain classifiers and additional models, making it challenging to inlegrate the approach into a holistic
and efficient framework. Further, and most importantly, these approaches have not been combined with the idea of latent clementary domains.
Motivation. Our work focuses on addressing the following two challenges to generalize the idea of latent domains. First, previous work has not considered the invariance of latent representations between source and lest domains, which limits their effectiveness for test-time adjustments and adaptation. To address this, we introduce the L.E.D assumption. Second, creating effective architectures for these approaches can be difficult, especially ones that can seamlessly integrate with existing deep learning frameworks. To overcome this challenge, we introduce our Gated Domain Units, which leverage the L.EDD assumption. Practitioners require methods that can handle feature extractors for various data types, such asResNet-50 (He et al., 2016), 1enseNet-121 Huang et al. (2017), Graph Isomorphism Networks with virtual nodes (GIN-virtual; Xu et al. (2019) and Gilmer et al. (2017)), DistilIBERT [aanh et al. (2020), with minimal integration effort. In our work, we propose the 'invariant elementary distributions assumption to tackle the first challenge, and we propose an effective layer consisting of Gated Domain Units to address the second challenge. More details on this can be found in the next two sections.
3 Domain Generalization with Invariant Elementary Distributions
We leverage the above observation for domain generalization by considering the mizture component shift the most common form of DS which states that the data made up of different sources, each with its own characteristics, and their proportions vary between domains (Quinonero- Cannonner--Cnlllaaaaaa et al. 2022 pp. 19).
3.1 Invariant Elementary Distributions
Let X' and y be the input and output space, with a joint distribution P. Let X and Y be random variables taking values in X and D, respectively. We are given a set of D labeled source datasets {D;H2, with Dj C X x y. Each of the source datasets is assumed to be I.I.D. generated by a joint distribution TP; with
4

---

Published in Transactions on Machine Learning Research (09/2023)
31
((I;)
Av
0.8
0.5
0.0
0.4
IV22
0.2
1/1
1/2
During training
During inference
Figure 2: A visualization of "invariant elementary distribution (I.F.D.)" assumption for domain generaliza- tion (DG): both observed and test-time data distributions (orange and violet) are composed of the same set of wnobserved elementary distributions (blue and red) that remain invariant across different domains. Hence, the first challenge, during training (left panel), is to extract these elementary distributions from the observed data (orange). The unobserved clementary distributions are represented by the elementary bases V/1 and V/2 (cyan and magenta). The second challenge, during inference (right panel), is to create a weighted ensemble of learning machines that utilize the similarities between the embedding of the unseen observation d(x;; and the embeddings of these distributions μV, and Nv, in the RKHS H (green rectangle) weights Bar and Bi2.
support on X' x y, heneeforth denoted domain. Note that the conditional distributions of different domains may change. Hence, our work differs from related work in DG that rely on the covariate shift assumption, i.., the conditional distribution of the training and test data stays the same (David et al. 2010 The set of probability mcasures with support on X' x y is denoted by P. The multi-source dataset D comprises the merged individual source datasets {D))== We aim to minimize the empirical risk, see Section 4.2 for details.
Similar to Mansour et al. 2009 2012 and Hoffman et al. 2018a we assume that the distribution of the source dataset can be described as convex combination F Ej= a, where a = (a) is an
element of the probability simplex, i.e., a* E A := {a E PR ay 2 E 1}. In other words, af quantifies the contribution of each individual source domain to the joint source dataset. For the test domain, prior work Mansour et al.] (2009: 2012) and Hoffman et al.) (2018a) further makes the important assumption that it lives in the convex hull of the source domains. We aim tie on these assumptions and connect them to the idea of latent domains, as will discuss in the following paragraph.
For this connection, express the distribution of each domain as a convex combination of K elementary distributions {K C P, meaning that P Σ o5P; where a" E A Our main assumption is that these elementary distributions remain invariand across the domains. The advantage is that we can find an invariant subspace at more clementary level as opposed to when we consider the source domains some sort of basis when generalizing to unseen domains. For a previously unseen distribution, we aim to determine the coefficients af and quantify the similarity to each elementary domain. Figure p illustrates this idea.
Table 1: Notation
number elementary distributions
number elementary domain bases
mumber basis vectors
P
combined multi-source distribution
P
j-th single-source distribution
P5
j-th elementary distribution
1,
j-th domaii basis
k-th vector V2
Pareto invariance The L.E.D assumption implics the invariant structure in the hypothesis space that can be exploited during train- ing, as shown in the following proposition. The proof is given Appendix A.1
coefficient for F;
coefficient for F,
coefficient sample at and py;
Definition 1. Let F be a hypothesis space of functions and (Ri...., Rx) : F IRK vector of risk functionals for K 2 2. Then, the hypothesis f F is said
5

---

Published in Transactions on Machine Learning Research (09/2023)
to be Pareto-optimal w.r.t. F if there evists no g GEZ F such that R2(9) - R,(f) for all j E f1...., KK with R3(g) < R5( for some j; see, also, Sener & Koltun 2018 Defimition 1).
In other words, f E F is Pareto optimal there exists no hypothesis g E F that optimizes every risk functional R2 better than ʃ.
Let L: y + be non-negative loss function, F a hypothesis space of functions x and IP (x,))) data distribution. Suppose the I.E.D assumption holds, i.e., there exist K elementory distributions Pi....,,PK such that P = =' a for some a E AA K Then, the corresponding Bayes predictor f E arg merr E(x.r) [C(Y, [xX) Pureto-optimal w.r.t. F and elementary risk functionals (R1....,Ry, Rk) where Rf(f) B (CC.)((.)))
Proposition 3.1.
Proposition 3.1 implics that, under the L.E.D assumption, Bayes predictors must belong to a subspace of F called the Parelo set F C which consists of Pareto-optimal models. In other words, the L.E.D assumption allows to translate the invariance property of data distributions to the hypothesis space. Since Bayes predictors of all future test domains must lic within the Parcto set, which is a strict subset of the original hypothesis space, it is sufficient for the purpose of generalization to maintain only solutions within this Pareto set during the training time. This makes the solutions within the Pareto set independent. of actual coefficients a of the source distribution and potential target distributions. Unfortunately, neither the elementary distributions nor the weights a are known in practice. Motivated by this thcoretical insight, our method presented in Section 4 is designed uncover them from a multi-source training dataset D Reducing the search space from the entire hypothesis space F to the Pareto set w.r.t. elementary risk functionals (Ri,..., Rx) suggests a modular structure of our proposed Gated Domain layer that consists of Gated Domain Units and each of these units optimizes one risk functional Ry. The proposed decomposition also suggests that the Pareto set is independent on the actual coefficients a which suggests an ensemble-like design of our approach. The final solution is a linear combination the functions optimizing the elementary , distribution and the elementary distribution.
risk functionals .R,...., Rx) ) with coefficients a that are derived from the similarity between the target.
3.2 Kernel Mean Embedding of Elementary Distributions
We employ the kernel mean embedding (KME) (Berlinet Thomas- Agnan 2004 Smola et al. 2007 Muandet et al. 2017) to represent the elementary distributions. Let H be reproducing kernel Hilbert space (RKHS) of real vvalev functions on X' with a reproducing kernel k X' x X + R (Schölkopl et al. 2001). The KME of probability mcasure IP in the H is defined by ((P) μp == fx k(x, .) (x). Given I.I.D. samples { from P, uP can be approximated by the empirical KME for n Σ k(r;, KME is popular in practice because no explicit distributional assumption, e.g. normality, is required. As such, it has been applied to
various machine learning applications, including hypothesis testing (Gretton et al. (012)), training deep generative models (Li et al. 2017), causal inference (Chau al. 2021). explainability (Chau et al., 2022 and many more.
In this work, we aim to utilise KME to represent invariant elementary distributions. Moreover, given new test samples, we aim to build a weighted ensemble of predictors from each elementary distributions, where the weights denote similarities between test-time domain and elementary distributions. However, this leads to two challenges, see Figure 2 for a visualisation.
First, we have no access to samples from the unknown elementary distributions. Thus, the elementary KME cannot be estimated directly from the samples at hand. To this end, we instead seek a proxy KME MV; := - 'N' k(rk,-) for each elementary KME May using a domain basis V, where Vj [of...., uN S X for each elementary domain j E (1,..., M). Hence, the KME μV, can be interpreted as the KME of the empirical probability measure PPv; N-1 -- 8zg. Here, we assume that M = K meaning that we approximate all clementary distributions. The sets V, are referred to as elementary domain basis. Intuitively, the elementary domain basis Vi...., VM represents each elementary distribution by a set of vectors that mimic samples generated from the corresponding distribution.
6

---

Published in Transactions on Machine Learning Research (09/2023)
The second challenge lies in measuring the similarity between samples at test-time and the elementary distributions. We consider two similarity measures utilising their RKHS presentations in the coming section.
With this two challenges tackled, we can proceed to build convex combination of elementary domain-specific learning machine.
To this end, we instead soek a proxy KME MV, == N (ogg.) for each elementary KME Mry using a domain basis Vy where Vj = fo%.................. w S for each elementary domain [I,...,M]}
4 Gated Domain Units
This section aims to answer the aforementioned question by instantiating the theoretical ideas presented in Section 3 as a neural network layer consists of a novel gated domain unit (GDU). For the purpose of implementation, let x E R denote the input data point and hg Rhx : + R the feature extractor that transform the input into a representation E R", e.g. ResNet-50 (He et al. 2016). DenseNet-12 121 Huang et al. (217), Graph Isomorphism Networks with virtual nodes (GIN-virtual; Xu al. (2019) and Gilmer al. (2017)) ) DistilIBERT Sanh al. (2020). The final prediction layer denoted as go : R y.
GDU consists of three main components:
A similarity function y H. x H + E defined on the RKHS that is shared across GDUs.
2. Elementary basis Vj, and
3. learning machine f(ž, 0z) for each elementary domain j
hg x2))
24
Big
CDD
GDUS
SS Par)
(ob(())) Mv,)
Au
V/3
Σ
Figure 3: Visualization of the layer (upper panel) and its main component, the GDU (lower panel). The layer consists of multiple GDUs that represent the elementary distributions. During training, the GDUs learn the elementary domain basis Vi,..., Va that approximate these distributions.
In Figure 3 depict the architecture and how data is processed and information is stored. Given representation é, each GDU, computes the similarity Big between this sample with their elementary basis Vj using the similarity function 7 acting on their RKHS mappings, i.e. Bij = > (Φ(ā.), ,v;). Each GDU is then connected to a learning machine f(i,, f2) that performs an elementary domain-specifts inference. We build the final predictor by taking ensemble of these learning machines by setting go(ā,) EM f 0,).
In summary, the GDUs leverage the invariant elementary distribution (I.E.D.) assumption and represent our algorithmic contribution. The elementary domain bases (Vj}, are stored as layer weight matrices, thus allowing us to learn them efficiently through backpropagation and avoid the dependency on problem-adaptive methods (e.g., domain-adversaria training) and domain information.
4.1 Choice Similarity Functions
In this work, we consider two categories of similarity functions 7: one based on distances between inputs and the elementary basis, and the other projection, akin to kernel sparse coding (Gao et al. 2010 2013)
7

---

Published in Transactions on Machine Learning Research (09/2023)
4.1.1 Geometry-basee based Generalization
We consider two geometry-based similarity measure for 7, namely the cosine similarity (CS) Kim et al.
2019) and maximum mean discrepancy (MMD) (Borgwardt et al. 2006 Gretton et al. 2012 The former considers angles, while the latter considers distances measure similarity. To ensure B4 (6(5)) mv,) sum up to apply the kernel softmax function Gao et al. (2019)
(v,))
Bij (()
Σ (xpl (o(E)) (va))
where &. > 0 is a positive softness paramcter for the kernel softmax. For CS, we pick
(d(I:), NV,)
H(o(B:), [0(0;)] 7| lv, 11
while for MMD, we choose
H (((())), = -1)(a;;) pv, 11
instcad. When given a batch of samples {0: [ we can compute the similarity between the batch with each domain basis by replacing ((x:) with (i n Σ ((ā))) in the above equations.
4.1.2 Projection-based Generalization
Besides geometry-based similarity, we can also seek for an optimal projection of ((z) onto the span of {uv, 3 such that Σド [(((;) Bijuy, l small. Note that as the subspace H.M := span l L...., M, are learnt during training, could incorporate pairwise orthogonality constraints, i.c. (μv,, = 0 for any + j, to maximises the "representation" power of this subspace. As result, this leads to the best possible approximation of the projection of ((ē)) onto H. if d(t;) actually lies in the span. See Proposition 4.1
In practice, given the hth batch of test samples { we could instead find a projection that minimises M Σe |Aa BijHyy ll where üú is the KME of the 7th batch.
Proposition 4.1. Given a distribution P where us spontuv, |j L,... M) such that (uv, pv.) 0 for M
all i + j. The value of Ee= 1p BjMy3 li is minimised at 13 (JIT, /v,, (V,,) H// h///////ll 11t-
See proof in Appendix A.2 In practice, when given single sample ãs we could set
(0(T:); mv3))
Bi | 11
a a batch of samples are given instead, we have,
(ūt HvV))
3,j
1| 11
4.2 Model Training
Following Zhuang et al. (2021), our learning objective function is formalized as C(g) + ADLDD(1ollæx) The goal of the training can be described in terms of the two components this function. Consider a batch of training data {21,..., 2.), where b is the batch size. During training, we minimize the loss function
1 & C(g)
Σ
C(D. vi)
Σ c
for an underlying task and the respective batch size. addition, our objective that the model learns to distinguish between different domains. Thus, the regularization Ωp is introduced to control the domain
8

---

Published in Transactions on Machine Learning Research (09/2023)
basis. In our case, require the regularization Gi to ensure that the KMEs of the elementary domain basis are able to represent the KMFs of the elementary domains, Therefore, minimize the MMD between the feature mappings ( and the associated representation Σك=1 Biguy. Note that Bi = ( pv,). Hence, the regularization 22p DD is defined
8 1
M Σ ==1
2B (1191H)) = Σ (o(ā.)
Σ
By 1
The intuiiion is the objective to represent each feature mapping o(t;) by the domain KMEs nv, Thus, we try to minimize the MMD between the feature map and combination of ,v, The minimum of the stated regularization can be interpreted as the ordinary least square-solution of a regression-probble of o(ā;) by the components of HM. In other words, we want to ensure that the basis V, is contained in feature mappings d(..).
In the particular case of projection, want the KME of the elementary domain to be orthogonal to ensure high expressive power. For this purpose, the additional term 2 will be introduced to ensure the desired orthogonality. Considering a kernel function with (xx,x) 1, orthogonality would require the Grain matrix Kyy = (nv, ) to be close to the identity matrix L. There are a variety of methods for regularizing matrices available (Xie et al. 2017 Bansal et al.] 2018). well-known method to ensure orthogonality is the soft orthogonality regularization 27 = A||K T (Bansal et al., 2018). As pointed out by Bansal et al. (2018), since spectral restricted isometry property and mutual coherence regularization can be a promising alternative, we provide an additional implementation for both. Hence, in the case of projection, the regularization is given by Ptp (11a1l)) = Aor.s (I1rl)) (I1allx)) + A (119ll)), ((oll)) Aor.s, AORTH 2 0.
Lastly, sparse coding is an efficient technique to find the least possible basis to recover the data subject to a reconstruction error (Olshausen & Field 1997). Several such applications yield strong performances, for example the field of computer vision (Lee et al. 2007 Yang et al. 2009). Kernel sparse coding transfers the reconstruction problem of sparse coding into H. by using the mapping o, and, by applying a kernel function, the reconstruction error is quantified as the inner product (Gao et al. 2010 2013). To ensure sparsity, we apply the Iy-norm the coefficients 3 and add 2 (Ilrl)) == ( with the corresponding coefficient AL,-
1V Il to the regularization term Up
5 Experiments
First, we conduct a proof-of-concept study using a smaller Digits dataset in Section 5.1 performance of the Gated Domain layer against simple yet effective Empirical Risk Minimization (ERM) baseline. In addition, we compared our method against an ensemble of such ERM models, where the ensemble components are jointly trained on the combined source data. The number of ensembles in this experiment is equal to the number of source domains. We also use this digits dataset to discuss heuristic for choosing the main hyper-parameters and performing ablation study. Second, in Section 5.2 compare approach to stato-of-the-art DG methods, c.g., CORAL, IRM, and Group DRO, using standardized WILDS benchmark Koh et al. (2021b). In our experiments, we distinguish two modes of training our layer: (i) fine tuning, where we extract features using a pre-trained model, and (ii) end-to-end training, where the feature extractor (FE) and the Gated Domain layer are jointly trained. Our code is publicly available for TensorFlow (https://github. htt.ggittubccommee.ee /im//im--aa waann-tthhhppbbeee gdu4dg) and PyTorch wtt.cooooooaaa http:/iiiitthgggdddddeee pytorch).
We compare
5.1 Proof-of-concept based on Digits Classification
Following Feng et al. (220 among others, we create a multi-source dataset by combining five publicly available digits image datasots, namely MNIST Lecun al. (1998), MNIST-M Ganin & Lempitsky (2015), SVHN Netzer et al. (2011). USPS, and Synthetic Digits (SYN) Ganin & Lempitsky (2015). Each dataset, except USPS, is split into training and test sets of 25,000 and 9,000 images, respectively. For USPS, take

---

Published in Transactions on Machine Learning Research (09/2023)
the whole dataset for the experiment since it contains only 9,298 images. The task is to classify digits between zero and nine. We perform five leave-one-out experiments. For each experiment, one of these five datasets is considered an out-of offorfriiiinggggggooio target domain which is inaccessible during training, and the remaining four are constitute the source domains. Details regarding the model architecture, implementation as well as hyper-paremeters are given Appendix B.1.
5.1.1 Parameter selection
From a practical perspective, our layer requires choosing two main parraeeerrmtters: the number of clementary domains M and since we use the charactoristics Gaussian kernel the corresponding parameter a. The parameter M defines the number of elementary domains and determines the size the ensemble (i.e., the network size).
Here, we discuss a heuristic set M. Similar Matsuura & Harada (0200), pass the training data through FE and apply k-means clustering algorithm on the output. Then, we use the Davies-Bouldin score to select the optimal number of clusters as the basis to choose M. For example, using MNIST-M as the test domain, cluster scores suggest 10 to 12 clusters (see Figure 4).
Rumber of cuiters
As for the parameter 0, we resort to the median heuristic proposed in Muandet et al. (2016) that is o² median || ã. 1
Figure 4: Visualization of the mean and standard deviation of the Davics-Bouldis Score across five runs for MNIST-M as the held-out test domain.
.n). While both heuristics require a pre-trained FE, cross-validation can act as a reasonable alternative given that we have access to adequate validation data (Koh et al. 2021a Gulrajani & Lopez-Pas 2020
=1,
5.1.2 DG performance
Although the cluster score has its optimum around 12, we set M to 10 for this experiment considering the additional computational complexity. The syper-parameter relevant for our layer are summarized Table 7 in Appendix B.1 In Table we present the final results for our proof-of-concept experiment. We compare the performance of the Gated Domain layer with different similarity functions (CS, MMD, Projected) trained in fine tuning (FT) and end-to-end (E2F) modes. Additionally, Risk Minimization (ERM) algorithm, which trains the entire classification model using all available source data without enforcing any invariance. We also consider an ensemble of M = 10 ERM classifiers to make a fair comparison with the Gated Domain layers, which use multiple domains. Our method noticeably improves across all tasks the mean accuracy and decreases the standard deviation compared to the ERM and ERM ensemble baselines, making the results more stable across ten iterations.
compare approach against. the Empirical
Table 2: Results of the Digits experiment. All experiments were repeated ten times and the mean (standard deviation) accuracy is reported. Best mean results are bold.
MNIST
MNIST-M
SVHN
USPS
SYN
97.98 (0.34)
63.00 (3.20)
70.18 (2 74)
93.70 (1.74)
83.62 (1.47)
ERM
(0.39)
62.87 (3.50)
72:01 (3.59)
(0.89)
83.80 (1.22)
CS
98.75 (0.09)
69.44 (0.57)
79.65 (0.89)
96.43 (0.35)
87.94 (0.61)
FT
MMD
98.74 (0.12)
69.32 (0.46)
79.50 (1.
96.54 (0.27)
87.78 (0.6)
PRO
98.71 (0.11)
69.44 (0.33)
79.4 (1.15)
96.37 (0.36)
.8 (0.622
(0.09)
(2.25)
78.76 (1.63)
(0.45)
87.41 (1.08)
E2B
MMD
98.52 (0.11)
68.80 (0.68)
78.21 (1.09)
95.22 (0.91)
87.78 (0.91)
PRO
98.13 (0.41)
63.15 (4.32)
75.04 (1.71)
96.49 (0.79)
86.04 (1.27)
10

---

Published in Transactions on Machine Learning Research (09/2023)
5.1.3 Ablation study
First, we discuss the effect of deviating from the cluster heuristic in choosing the number of elementary domains M. For this, we vary the corresponding parameter M and analyze the classification performance to validate our heuristic and discover sensitivity of our Gated Domain layer to the choice of M. The results for the most challenging DG task (MNIST-M) are depicted in Figure 3 (left). In general, the performance difference is slight across the choice of M. However, for FT, we generally observe higher classification performance than E2E, which also increases with higher M. For FT, we have the highest performance around the heuristically estimated M. In E2E, the performance is slightly lower, and we reach the highest performance when we set M smaller than suggested by the heuristic. This behavior can be explained by the fact that in E2F, the feature extractor and GDUs are jointly trained, and the lower dimensional embedding (i.e., Ø) is stochastic, which makes learning to approximate the elementary distributions more challenging, especially when their number high. These results support our heuristic, which we later apply to real-world datasets with much larger number of source domains.
Lastly, analyze the scnsitivity of the GDUs to the number of basis vectors in each elementary domain (N). For this, we vary N and keep the number of elementary domains fixed to the value suggested by the cluster heuristic. We observe that smaller N yields higher results while having the beneficial side-effect of roducing the computational overhead. We explain such an effect on the performance with noise reduction and lowering variance as well as avoiding the curse of dimensionality, which makes the embedding sparse and dissimilar, thus degrading the performance of the similarity function. In other words, reducing the number of basis vectors can be viewed as a dimensionality reduction technique, similar to Principle Component Analysis for high-dimensional data.
Number of Elementary Domains (M)
Number Doma n Bosis (1)
0.290
Melhed C5
Meihos - MVD
Method Projects
0.700
Melhod CC
Method UMC
Nehho Pigected
2.530
Tea sing
0,500
Trariig
2.50
0.000
3570
2
10
5
10
15
0.0T0
12
20
10
26
20
Figure 5: Mean and standard deviation of classification accuracy over 10 runs for varying number of elementary domains (M, left panel) and varying number of vector for each domain basis (N, right panel) for MNIST-M dataset.
The next paragraph focuses on ablating the regularization terms introduced in Section 4h, which dependents on the form of generalization (i.e., geometry-based or projection-basee For the geometry- based generalization, the regularization is
(19lll) AorsoRLs (191))) + >r, 2 (lhYl),
(1)
where \OLS, AL, 2 0. In the case of projection, the regularization is given by
(Ialll) = SOLS (IIll)) + (
(2)
with Nons AORTH 0.
We vary in Equation I and Equation 2 the corresponding weights AOLS and AL, (Eq. D or AORTH (Eq. 2 in the interval of [0; 0.1] and display the mcan classification accuracy for the most challenging classification task of MNSIT-M in the form of a heatmap. In Figures [ a-f, we see that the classification accuracy remains on an overall similar level which indicates that our method is not very sensitive to the hyper- parameter change for MNIST-M as the test domain. Neverthcless, we observe that ablating the regularization terms by sotting the corresponding weights to zero decreases the classification results and the peaks in performance occur when the regularization is included during training of our method.
11

---

Published in Transactions on Machine Learning Research (09/202))
05.07
01.22
25.
05.0
05.14
085
60.5
090
Aey
0003
-80
675
70.14
64.14
47.
Aperv
(b)
(c)
100
55.40
02.72
02.29
68-95
05.5 DV
00.15
65.03
18 27
095
60.0
Asy EBtt
85.5
68.03
68:75
08.
o0t3
07.0
06:00
Axt
Apenv
(d)
(I)
Figure 6: Classification results for varying λL, and AOLS in the interval of [0; 0.1] for FT-CS (a), E2E-CS (d), FT-MMM (b) and E2E-MMD (c) and for varying AORTH and Aors in the interval of 0.1] for FT-Projectod (c) and PrE-Projected (f) Projection on MNIST-M.
Applying GDUs comes with additional overhead, especially the regularization term that ensures the thogonality of the elementary domain bases. This additional effort raises a question whether ensuring the theoretical assumptions outweigh the much higher computational cost. Thus, in second step, analyze how the orthogonal regularization affects the orthogonality of the elementary domain bases (i.e., spectral restricted isometry property (SRIP) value) and the loss function (i.c., categorical cross-cntropy).
1.0
Label
Label E2E
E2E-Aommm0.0
A2E-Aom--0
E2
0.09
TTAomumm
0.8
OT D1
0.6
0.07
0.4
15
20
25
35
40
5
10
15
40
Epochs
Epochs
Figure 7: Effect of omitting the orthogonal regularization term 2 Spectral restricted isometry property (SRIP) (left) and categorical cross-entropy (right) with and without orthogonal regularization and their evolution during training for MNSIT-M dataset. The mean and standard deviation presented for End-to-end (E2E) and Fine-tuning (FT) training scenarios are calculated over 10 runs.
In Figure 7 we depict the mean and standard deviation of the spectral restricted isometry property (SRIP) value and loss over five runs for 40 epochs. The SRIP value can be tracked during training with our layer's callback functionalities. First, we observe that the elementary domains are almost orthogonal when initialized. Training the layer leads in the first epochs to a docrease orthogonality. This initial decrcase happens because cross-entropy has a stronger influence on the optimization than regularization the first epochs. After five epochs, the cross-entropy decrease to threshold when the regularization becomes more effective and the
12

---

Published in Transactions on Machine Learning Research (09/2023)
orthogonality of the elementary domain bases increases again. In Figure 7 also observe that ablating the orthogonal regularization, while leading to better orthogonality of the domains, does not significantly affect the overall cross-entropy during training.
This analysis revealed stable results across different sets of typer-parametenn While the layer not noticeably sensitive the choice of regularization strength, we recommend not omit the regularization completely, although the computational expenses decrease without the orthogonal regularization.
Third, in the E2E scenario, the GDUs are jointly trained with the feature extractor. In this particular scenario, we seek to understand how the GDU affect the latent representation learned by the feature extractor. For this, we project the output of the FE trained with a dense layer (ERM) and our layer by t-SNE in Figure [ The GDU-trained FE yields more concentrated and bounded clusters in comparison to the one trained by ERM. Hence, we observe a positive effect on the FEs representation.
ERM
MMD
Projection
Figure 8: Visualization of t-SNE Embedding on unseen Synthetic Digits Dataset. Colors encode true label.
Lastly, one of the essential questions we seek to answer in this ablation study is on what distributional structure given by the dataset the Icarned elementary domains represent. Figure g depicts the t-SNE embedding of the joint source domains (left), the lest, and the elementary domains learned by the GDUs (right). Nine of ten elementary domains form dense clusters around the joint source dataset, indicating that the GDUs learns to represent distributional structures in this multi-source dataset.
During training
During inference
01912
Domara
Figure 9: t-SNE visualization of source (left), the elementary and test domains (right). MNIST-M as held-out test domain, Gated Domain layer with MMD similarity trained in the fine tuning mode (FT-MMD). Colors encode source datasets (left) and elementary domains (right). Note: Marginals have different scales.
5.2 WILDS Benchmark
In addition to the ERM and ERM ensemble methods discussed in the previous section, we also compare our approach with state-of-tin the-rrrttttrttttoo algorithms. Currently, two open-source benchmarks are available for a reproducible and rigorous comparison of DG mothods, namely DomainBed Guirajani & Lopez-Paz (2020) and WILDS Kohh al. 2021a). WILDS is semi-synthetic benchmark that operates under similar assumptions the source component shift and represents DS occurring under real-workd conditions. The advantage in
13

---

Published in Transactions on Machine Learning Research (09/2023)
comparison to DomainBed is that WILDS reveals methods that work well on synthetic shifts may drastically fail on real-world shifts such as, for example, invariant risk minimization Gulrajani & Lopez-Pas (2020). To challenge the GDUs under potential model deployment, we chose WILDS.
The following methods reffect the potential and limitations of the research streams followed in DG (see Section 2.1), and thus, serve as our benchmark: CORAT, LISA, IRM, FISH, Group DRO, CGD, and ARM-BN. Appendix [32.3] provides details on the baseline algorithms. Note that these algorithms require access to domain information, unlike the approaches discussed in the previous section. Depending on the nature of the data, these labels may be difficult to access, discussed in Appendix [B2.2] We closely follow Koh et al. (2021a) for the experiments using eight datasots: Camelyon177, F,oW, Amazon, iWildCam, and RaRæl, OGB-MolPCBA, Givil-Commenis and PoverlyMap. Details on datasets are given in Appendix B.2.1
5.2.1 Benchmark results.
To provide a reasonable default parameter for our layer, we apply our heuristic to real-world conditions with a higher number of domains (c.g., 323 camera traps in iWildCam). Figure 10 depicts the cluster score for each of the eight dataset. The scores reach an agreement M 5, which we set the default value considering the model complexity associated with higher number of components. Table g from Appendix B.2 presents the default parameters of the Gated Domain layer used for the benchmarking. Please note that used the same set of parameters for all WILDS experiments.
fatoo
12-
Reary Ni
077
Figure 10: Visualization of the mean and standard deviation of -ouili- Score for each WILDS dataset with different number of clusters. The left panel depicts the scores when the output of the FE was clustered with k-means, while the right panel depicts the scores when the FE output was first embedded into a 2D space via t-SNE. Reducing dimensionality of the embeddings yielded better clustering results (i.e., lower Davics-Bouldes scores).
Table 3 reports the results for M Our approach achieves state-of-the-art OOD performance, however does not outperform the strongest benchmark LISA. Still, the consistent improvement to ERM is significant advantage compared to the benchmarks since they fall at least once short against ERM (e.g., IRM in RxRx1, Fish in iWildCam, or Coral in OGB-MoIPCBA). Thus, our method is especially appealing when domain information is challenging to obtain.
5.2.2 Worst-case vs. average-case case performance.
Methods trained to perform well on the worst-case (e.g., underrepresented populations) are prone a well-known tradc-off, thus performing less well on the average cases Eastwood al. (2022); Rice et al. (2021) h. Therefore, it is crucial to understand how DG methods are alfected by this trade-off. We use the pre-defined average- and wors-case subsets to compute the performance scores and analyse how each method is affected based on the Poverty, FMoW, and Ciniloomentss dataset.
As an example we provide the results for this analysis in Table 4 In general, we observe that this trade-off is apparent also for the benchmarking methods. Of particular note, our method balances this trade-off well, achieving improvements in worst- and average-case performance except for Civiloomennss For this dataset, obtaining domain labels might be the most challenging, yet, using this information seems essential for generalization.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Table 3:
Benchmarking results. A grey background highlights methods using no domain information for DG. We compute the metrics following Koh al. (2021a) and report the mean (standard deviation). Best benchmark and GDU results are bold.
Camelyon17
PMoW
Amazon
IWildCam
RxRx1
OGB- MolPCBA
Civil Com- ments
Poverty Map
No. domains
16
3,920
323
51
120,084
16
23 x
Scores
Avu Ace
Wounn-Ru Acc
10% Ace
MaCRO
AvG
Avt WOBNT-
REGIOO Acc:
GROUP Ace
Woust-
WUISt-U/R it
ERM
70.3 (6.4)
31.3 (0:17)
13.8 (0.8)
31.0 [1:3)
29.9 (0.4)
27.2 (0.3)
56.0 (3.6)
0:45 (0.06)
ERM Ensemble
(0.4)
26.9 (0.3)
55.6
0.52 (0.08)
CORAL
30.3 (7.7)
32.8 (0.66)
52.9 (0.8)
32.8 (0.1)
28.4 (0.3)
17.9 (0.5)
65.6 (1.3)
0.44 (0:07)
Fish
74.7 (7.1)
34.6 (0.18)
53.3 (0.0)
22.0 (1.8)
75.3 (0.6)
IRM
64.2 (8.1)
32.8 (2.09)
52.4 (0.8)
16.1 (4.9)
8.2 (1.1)
15.6 (0:3)
64.2
0.43 (0.07)
Group
(6)
58.3 (0.0)
23.9 (2.1)
22.5 (0.3)
22.4 (0.6)
08.4 (7.3)
(0.06)
LISA
77.1 (6.9)
35.5 (0.81)
54.7 (0.0)
31.9 (1.0)
72.9 (1.0)
CGD
09.4 (7.91
32.0 (2.26)
09.1 (1.9)
0.43 (0.04)
ARM-BN
24.4 (0.54)
23.3 (2.8)
31.2
Ours
68.5 (8:3)
31.8 (1:2)
(0.8)
31.2 (0.8)
29.9 (0.3)
27.5 (0.2)
50.0 (3.7)
0:62 (0.13)
FT
MMD
67.9 (8.0)
31.9 (1.2)
54.2 (0.8)
31.2 (0.8)
29.9 (0.3)
27.5 (0.2)
56.9 (3.7)
0.62 (0.13)
PRO
66.7 (8.9)
31.8 (1.0)
64.2 (0.8)
30.4 (1.0)
29.8 (0.3)
27.4 (0.2)
56.5 (2.9)
0.62 (0.12)
66.7 (8.9)
34.0 (1.9)
53.9 (0.7)
27.8 (2.1)
29.7 (0.4)
26.9 (0.1)
55.9 (0.8)
0:46 (0.07)
E2E
MMD
65.7 (6.7)
34.4 (0.7)
64.2 (0.8)
27.4 (1.6)
29.6 (0.2)
27.0 (0.5)
50.8 (0.7)
0.50 (0.06)
PRO
72.3 (9.5)
32.9 (0.8)
53.8 (0.8)
30.1 (1.2)
29.0
26.6 (0.3)
56.4 (2.1)
0.49 (0.07)
Table 4: Dotailed results on the trade-off between worst- and average case performance. grey background highlights methods using no domain information for DG. We compute the metrics following Koh al. (2021a) and report the mean (standard deviation). Best benchmark and GDU results are bold.
Pav
CivilComments
FMoW
Average-case
Worst-case
Average- case
Worst-case
Average- case Acr
Worst-case
PReMISN
WORST-U1/R
Acr
WOnST-
WONST-
PRANSON
RIGIIX Ace
RISSION Ace
ERM
0.78 (0.04)
0:45 (0.06)
92.1 (0.1)
56.0 (3.6)
52.7 (0.23)
31.3 (0:17)
ERM Ensemble
0.8 (0.04)
0.52 (0.06)
92.1 (0.0)
55.6 (0.8)
53.6 (0.06)
31.3 [1.85)
CORAL
0.78 (0.05)
0.44 (0.07)
88.7 (0.5)
65.6 (1.3)
50.1 (0.07)
32.8 [0
Fish
(0.3)
TRM
(0.05)
(0.07)
(0.7)
12.
Group DRO
0.75 (0.07)
0.30 (0:06
89.9 (0.5)
70.0 (2.0)
(1.15)
31.1 11
LISA
90.1 (0.3)
72.3 (1.0)
52.8 (1.15)
35.5 (0.81)
CGD
0.77 (0.04)
0.48 (0.04)
80.6 (0.4)
69.1 (1.9)
50.6 (1.39)
32.0 (2.26)
ARM-BN
23.8 (2.0)
72.7 (2.0)
(0.04)
(0.13)
(0:2)
53.1 (0.22)
FT
MMD
0.85 (0.04)
0.62 (3))
82.3 (0.2)
56.0 (3:7)
53.1 (0.22)
31.9
PRO
0.84 (0.04)
0.62 (0.
92.3 (0.3)
56.5 (2.9)
52.7 (0.14)
31.8 (1.05)
08
(0.07)
(0:25)
(1.66)
MMD
(0.05)
0.50 (0.06)
92.3 (0.2)
56.9 (0.7)
(0.45)
(0.71)
PRO
0.78 (0.04)
0.49 (0.07)
92.2 (0.3)
66.4 (2.1)
62.7 (0.68)
32.9 (0.78)
5.2.3 IID vs. 00D performance.
The relationship between IID and OOD performance can vary, from a positive correlation to a trado-off Wenzel et al.) (202)); Teney et al. (002)). Therefore, as last step, we seek understand how our method performs IID data compared to the OOD data, and compare the performance to the benchmarking methods. For this analysis, we leverage the IID) and OOD subsets of the iWildCam and RæRæI experiments.
The IID and OOD performances are presented in Table 5 The trade-off between OOD and IID performance is apparent for some methods (e.g., ARM-BN in RxRx1, Coral in iWildCam). For RaRzl, the GDUs' OOD performance equals that of ERM; however, we improve IID performance. For :WidCCam we observe a positive relation between IID and OOD performance. In summary, our modular layer consisting of the GDUs balance the performance on the IID and OOD subsets.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Table 5: Detailed results on the trade-off between IID and OOD performance. A grey background highlights methods using no domain information for DG. We compute the metrics following Koh et al. (2021a) and report the mean (standard deviation). Best benchmark and GDU results are bold.
IID
OOD
IID
OOD
Ace
MACRO F1 SCORIC
Ave
BOOBIC
ERM
35.9 (0.4)
29.9 (0.4)
47.1 (1.5)
75.7 (0.4)
31.8 (1:3)
71.3 (2.6)
ETM Ensemble
15.6 (0.4)
29.9 (0.4)
45.3 (2.1)
74.6 (0.2)
20.5 (0
69.6 (3.2)
CORAL
34.0 (0.3)
25.4 (0.3)
43.6 (3.3)
75.8 (0.3)
32.7 (0.2)
13.3 (4.3)
Flsh
(1.4)
(1.1)
(7.7)
59.8 (8.2)
15.1
(3.8)
Group DRO
28.1 (0.3)
22.5 (0.33
37.5 (1.9)
71.6 (1.7)
23.8 (2.0)
72.7 (2.0)
LISA
41.1 (1.3)
31.9 (1.0)
CGD
ARM-BN
(0.2)
(0.1)
(5:4)
(4.0)
23.3 (2.8)
36.0 (0.4)
(0.4)
(0:1)
76.6 (0.3)
(0.8)
72.7 (1.8)
FT
MMD
36.0 (0.2)
29.6 (1.2)
47.7 (0.1)
76.6 (0.3)
31.2 (0.8)
72:7 (1.8)
PRO
35.1 (0.3)
29.0 (0.2)
48.2 (0.5)
76.3 (0.44
30.5 (1.0)
72.3 (1.8)
(0.4)
29.9 (0.3)
(2.0)
74.9 (0:5)
27.8 (2.2)
(0.3)
E2E
MMD
36.2 (0.4)
29.9 (0.3)
43.4 (1.6)
75.3 (0.2)
27.4 (1.6)
68.3 (2.6)
PRO
36.0 (0.5)
29.8 (0.3)
45.4 (3.2)
75.0 (1.4)
30.1 (1.5)
71.6 (4.7)
6
Discussion and Conclusion
This work investigated the domain generalization (DG) problem under the L.E.D. assumption: real-world distributions are composed of elementary distributions that remain invariant across different domains. We showed its thcoretical implications and empirical effectiveness when instantiated in the form of a neural architecture called the GDUs.
Limitations. In contrast to prior work, mostly focusing on aligning training source domains on an observational level given fixed domain labels, our assumption enables modeling the lower-level shifts between them. However, the number of elementary distributions M must be specified in advance by the practitioners. While the application-agnostic hcuristic used in our experiments has proven effective, the optimal choice of M remains an open problem. Furthermore, our GDU layer induces additional computational overhead due to the regularization and model size that increases Noteworthy, our improvement is achieved with relatively small number of elementary domains indicating that the increased complexity is not cocrcive consequence of applying the GDUs. Also, the results achievod are not a consequence of increased complexity, as the ensemble baseline shows.
function of the number of elementary domains.
Outlook. Finally, for the L.E.D. assumption and GDUs to be fully adopted, yielding novel applications that tackle DG in general, is essential develop flexible architectures that can efficiently adapt the number of GDUs during training. For example, could increase M adaptively during training given proper clustering criteria. Following the deterministic annealing formulation Rose [1998), one could split an elementary domain into two once variance within the domain becomes too large. A generalization of the L.E.D. assumption to situations when some elementary distributions can differ across domains is also an interesting direction to pursue.
Acknowledgments
This project was partially funded by grant provided by the Hasler Stiftung (Project No. 22079) and the Swiss Heart Failure Network, SHIRT122/001DRRII (J. M. Buhmann, PI). Simon Föll acknowledges the Helmholtz Information & Data Science Academy (HIDA) for providing financial support enabling a short-term research stay at CISPA (Application No. 14877). We also thank Andrés Sass for contributing the PyTorch implementation.
16

---

Published in Transactions on Machine Learning Research (09/2023)
References
Martin Arjovsky, Lóon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. 2019. URL wttp://wwwrrraaa///....ooo arxiv:1907.022: 02289..
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. MetaReg: Towards domain generalization using meta-regullarizaionn. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (ods.), Adv. Neural Inf. Process. Syst., volume 31. Curran Associates, Inc., 2018.
P. Bándi, O. Geessink, Q. Manson, M. Van Dijk, M. Balkenhol, M. Hermsen, B. Ehteshami Bejnordi, B. Lee, K. Paeng, A. Zhong, Q, Li, F. G. Zanjani, S. Zinger, K. Fukuta, D. Komura, V. Ovtcharov, S. Cheng, S. Zeng, J. Thagaard, A. B. Dahl, H. Lin, H. Chen, L. Jacobsson, M. Hedlund, M. Çetin, E. Halıcı, H. Jackson, R. Chen, F. Both, J. Franke, H. Küsters- Vandevelde, W. Vreuls, P. Bult, B. van Ginneken, J. van der Laak, and G. Litjens. From Detection of Individual Metastases Classification of Lymph Node Status the Patient Level: The CAMELYONI7 Challenge. IEEE Transactions on Medical Imaging, 38 (2):550-560, February 2019. ISSN 1558-254X. doi:
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regularizations in training deep networks? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. R. Garnett (ods.), Adv. Neural Inf. Process. Syst., volume 31. Curran Associates, Inc., 2018.
and
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Mach. Learn., 79(1-2):151-175, 2010. ISSN 15730565.
doi: 10/c5bhdk.
Alain Berlinet and Christine Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probobility and Statistics. Kluwer Academic Publishers, 2004.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from Several Related Classification Tasks to a New Unlabeled Sample. In Advances in Newrol Information Processing Systems, volume 24, pp. 9, 2011.
Karsten M Borgwardt, Arthur Gretton, Malte J Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by Kernel Maximum Mean Discrepancy. Bioinformatics, (14):e49 e57, July 2006. ISSN 1367-4803. doi: 10/czwvqs.
Konstantinos Bousmalis, George Trigcorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain Separation Networks. In D Lee, M Sugiyama, Luxburg, I Guyon, and R Garnett (eds.), Adv. Neural Inf. Process. Syst., volume 29. Curran Associates, Inc., 2016.
Siu Lun Chan, Jean-Francois Ton, Javier González, Yee Tch, and Dino Sejdinovic. Bayesimp: Uncertainty quantification for causal data fusion. Advances in Neural Information Processing Systems, 34:3466 3477, 2021.
Siu Lum Chau, Robert Hu, Javier Gonzalez, and Dino Sejdinovic. Rkhs-shap: Shapley values for kernel methods. Advances in Neural Information Processing Systems, 35:13050 13063, 2022.
C. Chen, J. Li, X. Han, X. Liu, and Y. Yu. Compound domain generalization via meta-knowledge encoding. In 2022 IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pp- 7109-7119, Los Alamitos, CA, USA, jun 2022a. IEEE Computer Society. doi: 10.1109/ 10.1109/CVPR5266S 0PPPP55688.20222.0000 URL https://doi. http://bo..ieeecoppeeessrceeeeee
0rg/10..1109/CVP552688
2022.00698
Yongqiang Chen, Yonggang Zhang, Yatao Bian, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Learning causally invariant representations for out-of-distribution generalization on graphs, 2022b.
Rui Dai, Yonggang Zhang, Zhen Fang, Bo Han, and Xinmei Tian. Moderately distributional exploration for domain generalization, 2023.
17

---

Published in Transactions on Machine Learning Research (09/2023)
Alexander D' D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Fisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Niclson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Soqucira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellic Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification Presents Challenges for Credibility in Modern Machine Learning. November 2020. doi: 2011 2011.03395v2.
Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility thcorems for domain adaptation. In Proc. Thirleen. Inl. Conf. Artif. Intell. Stal., pp. 129 136. JMLR Workshop and Conference Proceedings, 2010.
Lucas Deecke, Timothy M Hospedales, and Hakan Bilen. Visual representation learning over latent domains. In International Conference on Learning Representations (ICLR 2022), April 2022. URL https://icll: cc/ Tenth International Conference on Learning Representations 2022, ICLR 2022; Conference date: 25-04-2022 Through 29-04-2022.
Cian Eastwood, Alexander Robey, Shashank Singh, Julius von Kügelgen, Hamed Hassani, George J. Pappas, and Bornhard Schölkopf. Probable Domain Genoralization via Quantile Risk Minimization. In Adv. Neural Inf. Process. Syst., volume 35. Curran Associates, Inc., October 2022.
Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Adv. Neural Inf. Process. Syst., volume 33, pp. (1996-12000) Curran Associates, Inc., 2020.
Hao-Zhe Zhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, and Chen Wei. KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation. ArXie Prepr. ArXiv2011088 109757, 2020. doi: 2011.09757.
Yaroslav Ganin and Victor Lempitsky. Unsupervised Domain Adaptation by Backpropagation. In Francis Bach and David Blei (eds.), Proc. 32nd Int. Conf. Mach. Learn., volume 37, pp. 1180 1189. PMLR, June 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. AdverriAdversariill Training of Neural Networks. Adv. Comput. Vis. Pattern Recognit., 17(0788331958344)111888 -09, May 2015. ISSN 21916594. doi: (0/gige2j.
Jing Gao, Wei Fan, Jing Jiang, and Jiawei Han. Knowledge Transfer via Multiple Model Local Structure Mapping Categories and Subject Descriptors. Kdd'08, pp. 283-291, 2008 doi: 10/d93d7f.
S. Gao, I. W. Tsang, and L. Chia. Sparse Representation With Kernels. IEEE Transactions on Image Processing, 22(2):423 434, February 2013. ISSN 1941-0042. doi: 10/fAnhfq.
Shenghua Gao, Ivor Wai-Hung Tsang, and Liang-Tien Chia. Kernel Sparse Representation for Image Classification and Face Recognition. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios (eds.), Compul. Vis. ECCV 2010, pp. 1 1-14, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN
978-3-642-15561-11
Yingbo Gao, Christian Herold, Weiyue Wang, and Hermann Ney. Exploring Kernel functions in the softmax layer for contextual word classification. arXiv, 2019. ISSN 23318422.
Muhammad Ghifary, W. Bastiaan Kleijn, and Mengjie Zhang, Domain adaptive neural networks for object recognition. Lect. Notes Comput. Sci. Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinforma., 8862: 898 904, 2014. ISSN 16113349. doi: 10/gigo2m.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Tch (ods.), Proceedings of the 94th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1263 1272. PMIRR 06-11 Aug 2017. URL https:
mlr. wttp://........eee/////////iiiaa http://ggmerrrr// html
18

---

Published in Transactions on Machine Learning Research (09/2023)
Boqing Gong, Kristen Grauman, and Fei Sha. Reshaping visual datasets for domain adaptation. In C.J. Burges, L.. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Adv. Newral Inf. Process. Syst. volume 26. Curran Associates, Inc., 2013.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13:723 773, 2012. ISSN 15324435.
Ishaan Gulrajani and David Lopez-Paz. In Search of Lost Domain Generalization. ArXin200701434 Cs Stal, July 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. ISBN 1063-6919. doi: 10/gdefkn.
C. Hillermeier. Generalized homotopy approach to multiobjective optimization. Journal of Optimization Theory and Applications, 110(3):557 583, 2001a.
C. Hillermeier. Nonlinear Multioojectiiee Optimization: A Generalized Homotopy Approach. International Series of Numerical Mathematics. Springer Basel AG, 2001b.
Judy Hoffman, Brian Kulis, Trevor Darrell, and Kate Saenko. Discovering Latent Domains for Multisource Domain Adaptation. In Andrew Fitzgibbon, Svetlana Lazebnik, Pictro Perona, Yoichi Sato, and Cordelia Schmid (eds.), Compul. Vis. - ECCV 2012, pp- 702-701 215, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-642-33709-3.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorit and theory for multiple-source adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Coso-Bianchi, and R. Garnett (ods.), Adv. Neural Inf. Process. Syst., volume 31. Curran Associates, Inc., 2018a.
Judy Hoffman, Eric Tzeng, Tacsung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexii Efros, and Trevor Darrell. CyCADA: Sycle-Consistent Adversarial Domain Adaptation. In Jennifer Dy and Andreas Krause (eds.), Proc. 35th Int. Conf. Mach. Learn., volume 80, pp. 1989-1998, Stockholmsmässan, Stockholm Sweden, 2018b. PMLR.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In Jennifer Dy and Andreas Krause (eds.), Proc. 35th Int. Conf. Mach. Learn., volume 80 of Proceedings of Machine Learning Reseorch, pp. 2029-2037. PMLR, July 2018.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovee. Open graph benchmark: Datascts for machine learning on graphs, 2021.
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conf. Comput. Vis. Pattern Recognit. CVPR, pp. 2261-2269, 2017. doi: 10 10/gfhw3n.
Geewook Kim, Akifumi Okuno, Kazuki Fukui, and Hidetoshi Shimodaira. Representation learning with weighted inner product for universal approximation of general similarities. In Proc. Quenty-Eightt Int. Jt. Conf. Artif. Intell. IJCAI-19, pp. 5031-5038. International Joint Conferences on Artificial Intelligence Organization, July 2019. doi: 10/ gigddb..
Tacksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to Discover Cross- Domain Relations with Generative Adversarial Networks. Tn Doina Precup and Yee Whye Teh (eds.), Proc. 34th Int. Conf. Mach. Learn., volume 70, pp. 1857-1865, International Convention Centre, Sydney, Australia, 2017. PMLR.
Pang Wei Koh, Shiori Sagawa, Hcnrik Marklund, Sang Michael Xic, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovee, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: Benchmark of in-the-Wild Distribution Shifts. In Marina Meila and Tong Zhang (eds.), Proc. 38th Int. Conf. Mach. Learn., volume 139, pp. 5637-5664, Proceedings of Machine Learning Research, 2021a. PMIR.
19

---

Published in Transactions on Machine Learning Research (09/2023)
Pang Wei Koh, Shiori Sagawa, Henrik Markhund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Trena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts, 2021b.
Daisuke Komura and Shumpei Ishikawa. Machine Learning Methods for Histopathological Image Analysis. Computational and Structural Biotechnology Journal, 16:34-42, January 2018. TSSN 2001-0370. doi: cj. 2018.01.001.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basee based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) November 1998. ISSN 1558-2256. doi: 10/d89c25.
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng, Efficient sparse coding algorithms. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Adv. Neural Inf. Process. Syst., volume 19. MIT Press, 2007.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain Generalization with Adversarial Feature Learning. In 2018 IEEECVF Conf. Comput. Vis. Pattern Recognit., pp. 5400-540000 140005509, Salt Lake City, UT, June 2018b. TEEE. ISBN 978-1-5386-6440-9 doi: 10/ghv245.
Haoliang Li, Wen Li, and Shiqi Wang. Discovering and incorporating latent target-domains for domain adaptation. Pattern Recognition, 108:107536, December 2020. ISSN 0031-3203. doi: 107536.
2020.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep Domain Generalization via Conditional Invariant Adversarial Networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Comput. Vis. ECCV 2018, pp. 647-663, Cham, 2018c. Springer International Publishing. ISBN 018-3-0300-1266-00
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Featuro-critic networks for heterogencous domain generalization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proc. 36th Inl. Conf. PMLR, June 2019.
Mach. Learn., volume 97 of Proceedings of Machine Learning Research, pp.
Ming- Yu Liu and Oncel Tuzel. Coupled Generative Adversarial Networks. In D Lee, M Sugiyama, U Luxburg, Guyon, and R Garnett (eds.), Adc. Neural Inf. Process. Syst., volume 29. Curran Associates, Inc., 2016.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael 1. Jordan. Learning transferable features with deep adaptation networks. 32nd Int. Conf. Mach. Learn. ICML 2015, 1:97-105, 2015.
Mingshong Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Adu. Neural Inf. Process. Syst., volume 31. Curran Associates, Inc., 2018.
Pingchuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration in multi-task learning. In Proceedings of the 37th International Conference on Machine Learning. JMLR.org, 2020.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (cds.), Adv. Neural Inf. Process. Syst., volume 21. Curran Associates, Inc., 2009.
Yishay Mansour, Mchryar Mohri, and Afshin Rostamizadch. Multiple Source Adaptation and the Renyi Divergence. Proc. 25th Conf. Uncertain. Artif. Intell. UAT 2009, pp. 367-374, 374, May 2012.
20

---

Published in Transactions on Machine Learning Research (09/2023)
Toshihiko Matsuura and Tatsuya Harada. Domain Generalization Using Mixture of Multiple Latent Domains. AAAT, 34(07):11749 11756, April 2020. doi: 10. 12.1609/aaai..34107..6888
Joao Monteiro, Xavier Gibert, Jianqiao Feng, Vincent Dumoulin, and Dar-Shyang Lee. Domain Conditional Predictors for Domain Adaptation. In Luca Bertinetto, João F. Henriques, Samuel Albanie, Michela Paganini, and Gül Varol (eds.), NearIPS 2020 Workshop Pre- Pegist. Mach. Learn., volume 148, pp. 193-220, Proceedings of Machine Learning Research, 2021. PMLR.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain Generalization via Invariant Feature Representation: Sanjoy Dasgupta and David McAllester (eds.), Proc. 30th Inl. Conf. Mach. Learn., pp- 10 18. PMLR, February 2013.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel Meon Embedding of Distributions: A Review and Beyond, volume 10. May 2016.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperunbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 10(1-2): 1-141, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits natural images with unsupervised feature learning. In NIPS Workshop Deep Learn. Unsupervised Feature Learn. 2011,2 2011.
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Res., 37(23) 3325, 1997. ISSN 0042-6989. doi: 10/dn4knd.
X. Peng and K. Saenko. Synthetic to Real Adaptation with Generative Correlation Alignment Networks. In 2018 IEEE Winter Conference on Applicutions of Computer Vision (WACV), pp. 1982-1991, March 2018. doi: 00/gie2k.
X. Peng, Q Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment Matching for Multi-Source Domain Adaptation. In 2019 IEEE/CVT International Conference on Computer Vision (ICCV), pp. 1406 1415, 2019. ISBN 2380-7504. doi: 10/ghfhmtt
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Cansal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7854:4977-1112, 2016.
Vihari Piratla, Pranceth Netrapalli, and Sunita Sarawagi. Efficient Domain Gencralization via Common- Specific Low-Rank Decomposition. In Hal Daumé III and Aarti Singh (eds.), Proc. 37th Int. Conf. Mach. Learn., volume 119, pp. 7728-7787 1728,, Proceedings of Machine Learning Research, 2020. PMLR.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Focus the Common Good: Group Distributional Robustness Follows, April 2022.
J. Quinonero-Candela, M. Sugiyama, A. Schwaighofer, and N.D. Lawrence, Dataset Shift in Machine Learning. Neural Information Processing Series. MIT Press, 2022. ISBN 978-0-262-54587-7.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snock, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distributio detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, F. Fox, and R. Garnett (eds.), Adv. Neural Inf. Process. Syst., volume 32. Curran Associates, Inc., 2019.
Leslie Rice, Anna Bair, Huan Zhang, and J. Zico Kolter. Robustness between the worst and average case. In M. Ranzato, A. Beygekimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Adv. Neural Inf. Process. Syst., volume 34, pp. 27840-27851. Curran Associates, Inc., 2021.
K. Rose. Deterministic annealing for clustering, compression, classification, regression, and related optimization problems. Proceedings of the IEFE, 86(11):2210 221002299, 1998. doi: 10. 2.111195577760
21

---

Published in Transactions on Machine Learning Research (09/2023)
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In International Conference on Learning Representations 2020, pp. 19, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, laster, cheaper and lighter, 2020.
Bernhard Schölkopf. Causality for Machine Learning. arXiv, pp. 1-20, 2019. ISSN 23318422.
Bernhard Schölkopf, Ralf Herbrich, and Alex J. Smola. A Generalized Representer Theorem. In G. Goos, J. Hartmanis, J. van Leeuwen, David Helmbold, and Bob Williamson (eds.), Computational Learning Theory, volume 2111, pp. 416-426. Springer Berlin Heidelberg, Berlin, Heidelberg, 2001. ISBN 978-3-540-42348-0 978-3-540-44488144 doi: 1400073-540-4588---1 27.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612 634, 2021.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 525 536, Red Hook, NY, USA, 2018. Curran Associates Inc.
Yuge Shi, Jeffrey Seely, Philip H. S. Torr, N. Siddharth, Awni Hammun, Nicolas Usunier, and Gabriel Synnaeve. Gradient Matching for Domain Generalization. arXie,, 2021. doi: 2104.09937.
Alexander J. Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. Hilbert space embedding for distributtions. In Proceedings of the 18th International Conference on Algorithmic Learning Theory (ALT), pp. 13 31. Springer-Verlag, 2007.
Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Ennironments: Introduction to Covariate Shift Adaplation. The MIT Press, March 2012. ISBN 978-0-262-01709-L
Baochen Sun, Jiashi Feng, and Kate Sacnko. Return of frustratingly casy domain adaptation. 30th AAAI Conf. Arlif. Intell. AAAI 2016, (Figure 1):2058 2065, 2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring Robustness to Natural Distribution Shifts in Image Classification. arXiv:2007.0064/ September 2020.
David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John- Bokhorst, Francesco Ciompi, and Jeroen van der Laak. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical Image Analysis, 58:101544, December 2019. ISSN 1361-8415. doi: 10. 10111/1..mdii.a220 2019.ee01...111115.4...
Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets, September 2022.
E. Tzeng, Hoffman, K. Saenko, and T. Darrell. Adversarial Discriminative Domain Adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2962-2977 July 2017. ISBN 1063-6919. doi: 10/gf5br3.
Eric Tzeng, Judy Holfman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep Domain Confusion: Maximizing for Domain Invariance. January 2014.
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
Mitko Veta, Paul J. van Diest, Mehdi Jiwa, Shaimaa Al-Janabi, and Josien P. W. Pluim. Mitosis Counting in Breast Cancer: Object-Level Interobserver Agreement and Comparison an Automatic Method. PLOS ONE, 11 1188)e006112) 286, August 2016. doi: 10.1371 10.1371//jorrall. /oo.:ralll..oo..01006 016118.....
22

---

Published in Transactions on Machine Learning Research (09/2023)
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. CoRR, hhs/2103.03097, 2021. URL. httpp:://arxiv.org/// wwtp:///gg/abb/////////
03097
Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann SimonnGabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schicle, Bernhard Schölkopf, and Francesco Locatello. Assaying Out-Of-Distribution Generalization in Transfer Learning. In Adv. Neural Inf. Process. Syst. 22. Curran Associates, Inc., October 2022.
Pengtao Xie, B. Póczos, and E. Xing. Near-orthogonality regularization in kernel methods. In UAI, 2017.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka, How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https://openreviost http://pprrevvvooo.eee/trrraadda net/formm?id ryGa61A5Km
Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching using sparse coding for image classification. In 2009 IEEE Conf. Comput. Vis. Pottern Recognit,, pp. 1794- 1804,1 2009. doi: 10/d5229q.
Huaxiu Yao, Yu Wang. Sai Li, Linjum Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving Out-of- Distribution Robustness via Selective Augmentation, June 2022.
Yi Yao and Gianfranco Doretto. Boosting for transfer learning with multiple sources. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1855-1862, 2010. doi: (A110//CVPR.2000553388
Z. Yi, H. Zhang, P. Tan, and M. Gong, DualGAN: Unsupervised Dual Learning for Image- Loage-to-Inee Image Translation. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2868 2876, 2017, ISBN 2380-7504. doi: 10/ /gv876.
Kun Zhang, Mingming Gong, Petar Stojanov, Biwei Huang, QINGSONG LIU, and Clark Glymour. Domain adaptation as a problem of inference on graphical models. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Adv. Neural Inf. Process. Syst., volume 33, pp. 4963 4976. Curran Associates, Inc., 2020.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Adv. Newral Inf. Process. Syst., volume 34, pp. 11664-23678.. Curran Associates, Inc., 2021.
Yi-Fan Zhang, Jindong Wang, Jian Liang, Zhang Zhang, Baosheng Yu, Liang Wang, Dacheng Tao, and Xing Xic. Domain-speeific risk minimization for out-of-distribution generalization, 2023.
Han Zhao, Shanghang Zhang, Guanhang Wu, José M F Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial Multiple Source Domain Adaptation. In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa- Bianchi and R Garnett (eds.), Adv. Neural Inf. Process. Syst., volume 31. Curran Associates, Inc., 2018.
S. Zhao, X. Yuc, S. Zhang, B. Li, H. Zhao, B. Wu, R. Krishna, J. E. Gonzalez, A. L. Sangiovanni-Vincon S. A. Seshia, and K. Keutzer. A Review of Single-Source Deep Unsupervised Visual Domain Adaptation. IE... Transactions on Neural Networks and Learning Systems, pp. 1-21, 2020. ISSN 2162-2388. doi: 10/ghtz3h.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep Domain-Adversarti Adversarial Image Generation for Domain Generalisation. AAAI, 33(00) April 2020. doi: 10/gj4mr2.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain Generalization: A Survey. Ar Arx2200000000 Cs, March 2021a.
23

---

Published in Transactions on Machine Learning Research (09/2023)
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain Generalization with MixStyle. In International Conference on Learning Represenlations, pp. 15, 2021b.
J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired Image-to- to-Imae Translation Using Cycle- Consistent Adversarial Networks. In 2017 TEEE International Conference Computer Vision (ICCV), pp. 2242-222222 2251, 2017. ISBN 2380-7504. doi: 10/gfhw33.
F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. A Comprehensive Survey on Transfer Learning. Proceedings of the IEEE,
January 2021. ISSN 1558- 2256. doi: 10/ghesqr.
24

---

Published in Transactions on Machine Learning Research (09/2023)
A Proofs
A.1 Proof of Proposition 3.1
Proof. The result holds trivially for K = 1. For K 2 2 and by the L.E. assumption, P (x
Σ (X,,,(X, Y) for some E A Then, we can write the risk functional for each E F R(J)
[ f(x)) (z, = I Cly f(z)) dE)= (z.)(z.v)) = Σラ= og f())) (z, = プØ= (f) where F R+ is the elementary risk functional associated with the elementary distribution P, PX, Hence, the Bayes predictors satisfy
K Σ Rg)) 1=1
f" E min R()) = arg min JEF
(A.3)
JEF
Since the rhs of equation A.3 corresponds to the linear scalarization of a multi-objective function (R.,.... Rx) its solution (i.e., a stationary point.) is Pareto-optimal with respect to these objective functions (Ma al. 2020 Definition 3.1); see, also, (Hillermeier 2001a b). That is, the Bayes predictors for the data distribution that satisfies the I.E.D assumption must belong to the {f f arg minjer aj= a e A } C FPareto C F, where Farcto comprises predictors that are Pareto-optimal with respect to hypothesis space F and risk functionals (Rk)...
A.2 Proof of Proposition
Proof. Suppose we have a representation,
M Σ
ByKV,
= Ovi -w
(A.1)
i.e. (uv..... myy are pairwise orthogonal. We want minimize the MMD by minimizing
M
M
M
M By
||
Bykv, 11 = (e,, Nee. - 2(2
By mu +
(A.2)
1,==
jml
M
=11mz11 -2 Σg j=1
B3 Hy, +
8,8g
(uv (uve:Mr)
(A.3)
6,g (Mag Muy, )===
M
M Σ 1=1
= =11uz112 - 2 >>
Bi (up, ((p,,v,)> mV3) +
33
l 11:e
(A.4)
1=1
By defining
M Σ
((3) = |iee
Sy l
(A.5)
we can simply find the optimal 3, by using the partial derivative
OΦ
22/1r, MV,) + 2/3, (aa, 11 1 + 3,lllv, 11: = (ne, MV;
(A.4)
03,
(A.5)
* - (up,Hv;) 2x
(A.6)
Iwv, 11x
Please note that the function & is convex.
25

---

Published in Transactions on Machine Learning Research (09/2023)
B
Experiments
In this section, we provide a detailed description of the DG experiment presented in Scction 5 Our Digits experiments are implemented using TensorFlow 2.4.1 and TensorFlow Probability 0.12.1. For the
WILDS benchmarking we use our PyTTrch (version 1.11.0). All source code is available on GitHub https: //gthub. com.commmmmmm. coom/im-etee wwweeethhhpuuuudgee (TensorFlow) and (PyTorch).
For the Gated Domain layer, we considered two modes of model training: fine tuning (FT) and end-to-end training (E2E). In FTT scenario, first pre-train the FE in the ERM single fashion. Then, extract features using the pre-trained model and pass them to the Gated Domain layer for training the latter. For the E2E training, however, the whole model including the FE and Gated Domain layer is trained jointly from the very beginning.
B.1
Digits Experiment
Our experiment setup is closely related to Peng et al.
(2019) ; Feng et al. (2 Zhang et. al. 2020 Zhao et al. (018 J. We used the digits data from https: : / /githuub
Table 6: Feature Extractor used for the Digits Experiment
com Nomm/enggHZ/KDaL [last accessed on 2022-05-17 17, available
under MIT License.] published in Peng al. (2020). Our experimental setup regarding datasets, data loader, and FE are based on existing work (Feng et al. 2020 Peng ct al. 2019). The structure of the FE is summarized in Table and the subsequent learning machine is a dense layer.
FEATURE EXTRACTOR
LAYER TYPE
OUTPUT SHAPE
2D-CONVOLITINNA LAYER
[32, 32, 64)
BATCH NORMALZZTTIO
(32. 32,
MAXPOOLING 2D
[16. 16, 61)
CONVOLLUTOOOALLL
[16.
BATCH NORMALIZATION
[16. 16, 64)
MAXPUOLING 2D
[8. 8, 64)
AD-CONVOLUTOONL LAYER
(8. 8. 128)
In the Empirical Risk Minimization (ERM) single ex- periment, add dense layer with 10 outputs (activa- tion=tenh) classifier to the FE. In the Empirical Risk Minimization (ERM) ensemble experiment, we add M classification heads (a dense layers with 10 outputs and tanh activation each) to the FE and average their output for the final prediction. This sets a baseline for our Gated Domain layer to show performance gain against the ERM model with the same number of learning machines.
BATCH NORMALIZATIO
(8, 8, 128)
MAXPOOLING 2D
(4, 128)
FLATTEN
(2048)
DENSE
(3072)
BATCH NORMALIZATION
(3072)
DROPOUT
(3072)
BATCH NORMALIZTTION
(2018)
DENSE LAYER
(2048)
For training, resorted to the Adam optimizer with learning rate of 0.001. We used carly stopping and selected the best model weights according to the validation accuracy. For the validation data, we used the combined test. splits only of the respective source datasets. The batch size was set to 512. Although the Gated Domain layer requires more computation resources than the ERM models, all digits experiments were conducted on a single GPU (NVIDIA GeForce RTX 3090).
Table 7: Parameters for Gated Domain layer in Digits and Digit-DG Experiments for the Fine Tuning (FT) and End-to-end training (E2E) Settings. In case of Projection, we chose the spectral restricted isometry property (SRIP) as the orthogonal regularization 2t In the FT setting, we applied the median heuristics presented above to estimate a of the Gaussian kernel function, where the estimator is denoted as ō. Since median heuristic not applicable for the F2E scenario, o was fixed 7.5 for E2F.
An,
Vor.
AORTH
CS
10
le
le
2
MMD
10
le
le-3
2
PROJECTION
le 3
1e 8
CS
10
le
le
7.5
2
E2E
MMD
10
le
7.5
2
PROJECTION
10
le
16
26

---

Published in Transactions on Machine Learning Research (09/2023)
B.2 WILDS Benchmarking Experiments
For comparison of our approach and benchmarking, we followed the standard procedure of WILDS experiments, described in Koh et al. (2021a). As a technical note, all WILDS experiments have been implemented in Pytorch (version >= 1.7.0) based on the specifications made in Koh et al. (2021a) and their code published on http:/p-aammd//vidddd [last accessed on 2022-05-17 17, available under MIT License]. Table S presents the parameters of the Gated Domain layer used for all WILDS experiments. The results for the benchmarks were retrieved from the official leaderboard https:///iiii stanford. edu/leaderboard/ llatt accessed on 2022-09-26).
attps://githun:
Table Parameters for Gated Domain layer in WILDS experiments for the Fine Tuning (FT) and End-to-end training (F2F) Settings.
EXPERIMENT
dp
AoLS
AORTH
WILDS E2E
CS MMD
10
1e
1e
4
2
BENCHMARK
10
le
le
1
2
PROJECTION
1e-3
16-3
16
B.2.1 Benchmark datasets
Camelyon17 In medical applications, the goal to apply models trained on a comparatively small set. of hospitals to a larger number of hospitals. For this application, we study images of tissue slides under a microscope to determine whether a patient has cancer or not. Shifts in patient populations, slide staining, and image acquisition
impede model accuracy in previously unseen hospitals. Camelyon17 comprises images of tissue patches from five different hospitals. While the first three hospitals are the source domains (302,436 examples), the forth and fifth are the validation (34,904 examples) and test domain (85,054 examples), respectively.
We strictly follow the specifications made (Koh et al. 2021a) using DensoNet-121 (Huang et al. 2017 the feature extractor, a learning rate of 10e-3, an L2 regularization of 10e-2, batch size of 32, and stochastic gradient with momentum of 0.9. We trained for 5 epochs using early stopping.
We deviate from the specifications made in (Koh et al. 2021a) in terms of the FE. We use the FE from Feng ct al. (220)); Peng et al. (2019) since we observed higher mean accuracy and faster training than with the by Koh et al. (2021a) originally proposed DenseNet-121 FE (Huang et al.) 2017]. We trained the FF from scratch. Both, ERM and the DG were trained over 250 epochs with early stopping, a learning rate of 0.001, which is reduced by a factor of 0.2 if the cross-entropy loss has not improved after 10 epochs. All results were aggregated over ten runs.
FMoW Analyzing satellite images with machine learning (ML) models may enable novel possibilities in tackling global sustainability and economic challenges such as population density mapping and deforestation tracking. However, satellite imagery changes over time due to human behavior (e.g., infrastructure devel- opment), and the extent of change is different in each region. The Functional Map of the World (FMoW) dataset consists of satellite images from different continents and years: training (76,863 images; between 2002-2013), validation (19,915 images; between 2013 and 2016), and test (22,108 images, between 2016-2017). The objective is to determine one of 62 building lypes (e.g., shopping malls) and land-use.
As instructed in Koh et al. (202aa), without I.2-regularization. For the optimization, we use the Adam optimizer with learning rate of 1c-4, which is decayed by a factor of 0.96 per epoch. The models were trained for 50 epochs with early stopping and a batch size of 64. Additionally, we report the worst-region accuracy, which is a specific metric used for FMoW. This worst-region accuracy reports the worst accuracy across the following regions: Asia, Europe, Africa, America, and Occania (sce Koh et al. (2021a) for the details). Again, we report the results over three runs.
used the 1enseNet-121 (Huang et al. 2017 pre-trained on ImageNet
27

---

Published in Transactions on Machine Learning Research (09/2023)
Amazon Recent research shows that consumer-facing machine learning application large performance disparities across different, set of users. To study this performance disparities, WILDS (Koh et al. 2021a] leverages a variant of the Amazon Review dataset. The Aamazon- WILDS dataset is composed of data from 3,920 domains (number of reviewers) and the task is a multi-class sentiment classification, where the model receives a review lext and has to predict the rating from one to five. To split this dataset, a ween training, validation, and test disjoint set of reviewers is used: training (245,502 reviews from 1,252 reviewers), validation (100,050 reviews from 1,334 reviewers), test (100,050 reviews from 1,334 reviewers).
For the experiments and baseline models, use the specifications made in Koh et al. (021a).. As for the FE, we used DistiiBERT-base-mmaan
models (Sanh et al.) (0020). For ERM, we use batch size of 8, learning rate le-5, L2 regularization of 0.01, 3 epochs with early stopping and 512 as the maximum length of tokens. For training the Gated Domain layer, we used the same specifications as made for ERM. The performance is measured 10th percentile accuracy.
iWildsCam Wildlife camera traps offer an excellent possibility to understand and monitor biodiversity loss. However, images from different camera traps differ in illumination, color, camera angle, background, vegetation, and relative animal frequencics. We use the iWildsCam datasct consisting of 323 different camera traps positioned in different locations worldwide. In the dataset, we refer to different locations of camera traps as different domains, in particular 243 training traps (129,809 images), 32 validation traps (14,961 images), and 48 test traps (42,791 images). The objective is to classify one of 182 animal species.
Following the instructions by Koh et al. 2021a), we used again the ResNet50 pre-trained on ImagNet He et al. 2016). For ERM, we used a learning rate of 3e-5 and no I.2rregularization The models were trained for 12 epochs with a batch size of 16 with the Adam optimizer. In addition to the accuracy, we report the macro F1-score to evaluate the performance on rare species (see Koh el al. (2021a) for details). All results were aggregated over three runs.
RxRx1 In biomedical research areas such as genomics or drug discovery, high-throughput screening techniques generate a vast amount of data in several batches. Because experimental designs cannot fully mitigate the effects of confounding variables like temperature, humidity, and measurements across batches, this creates heterogeneity in the observed datasets (commonly known as batch effect). The RxRx1 dataset comprises images obtained by fluorescent. microscopy from 51 domains (disjoint experiments): training (40,612 images, 33 domains), validation (9,854 images, 4 domains), and test (34,432 images, 14 domains). The aim is to classify one of 1,139 genetic treatments. All results were aggregated over three runs.
We conducted the RxRx1 experiments in accordance with the specifications made in (Koh et al. 2021a As for the FE, we, thus, used the ResNet50 pre-trained on ImagNet (He et al.) 2016). We trained the models using AdamW with default parameters 31 = 0.9 and 33 = 0.999 using a learning rate of 1e-4 and a L2-regularization with strength 1e-3 for 90 epochs with a batch size of 75. We scheduled the learning rate to lincarly increase in the first ten epochs and then decreased it following a cosine rate. For training the Gatod Domain layer, we chose the same parameters for the ERM. All results were aggregated over three runs.
OBG-MoIPCBA In biomedical research, machine learning has the potential to accelerate drug discovery while reducing the experimental overhead due to lowering the number of experiments required. However, to leverage the potential of machine learning, the models need to generalize to molecules structurally different. from those scen during training. To study this OOD generalization across molccule scaffolds, we use the OGB-MoIPCBA dataset. This dataset is split into the following subsets according to the scaffold structure: training (44,930 domains), validation (31,361 domains), and test (43,739 domains). The task is to classify the presence/all aabennce of 128 biological activities based on a graph representation of molecule.
In line with Koh et al. (2021a), we use a Graph Isomorphism Network (GIN) combined with virtual nodes (GIN-virtual; Xu et al. (2019) and Gilmer et al. (2017) as the FE. For training ERM and our DG, we use the default parameters: five GNN layers with a dimensionality of 300 and a learning rate of 0.001. We train for 100 epochs using carly stopping. As for the performance, we report the mean and standard deviation of the average precision across all scaffolds (domains) over three runs.
28

---

Published in Transactions on Machine Learning Research (09/2023)
CivilComments In the last decades, users have generated vast amount of text on the Internet, some of which contain toxic comments. Machine learning has been leveraged for automatic text review to flag toxic comments. However, the models are prone to learn spurious correlations between toxicity and information on demographics in the comment, which causes the model performance to drop in specific subpopulations. To study this OOD lask; we leverage the modified CivilComment, dataset from Koh el. al. (2021a). Based on text input, the task is to predict a binary label, toxic or non-toxic. The domains are defined according to eight demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black, and White. All comments were randomly split into disjoint training (269,038 comments), validation (45,180 comments), and test (133,782 comments) set.
Again, we follow Koh et al. (2021a) and use a following parameters: batch size = 16, learning rate = 1e-5, AdamW optimizer, number of epochs = 5, L2 regularization 0.01, and the maximum number of tokens of 300. We use these default parameters for training our Gated Domain layer. The performance is measured in the worst-group accuracy and we report mcan and standard deviation across five runs.
model (Sanh et al. 2020 with the
PovertyMap As the FMoW example shows, satellite images in combination with machine learning models can been used to monitor sustainability and economic challenges on a global scale. Another application of these satellite images is poverty estimation across different spatial regions. However, there exists a lack of labels for developing countrics since obtaining the ground truth is expensive, which makes this application attractive for machine learning models. To study the OOD generalization to unseen countries, we use a modified version of the poverty mapping dataset of WILDS (Koh et al. 2021ah. The task is to predict a real-valued aset wealth index between 1 and 5 based a multi-spectral satellite image. The domain refers to the country and whether the the the image is from a rural or urban are. In contrast to the other datasets, this dataset is split. in five different. folds, whereby in each fold the the training, valdiation and test set contains a disjoint set of countries, however, data from both rural and urban regions. The avergae size of each set across the 5 folds is for the training -10,000 images (13-14 countries), ~4,000 images (4-5 different countries), and for the test set -4,000 images (13-14 14 countries).
We follow Koh et al.. (2021a) and use a pre-trained ResNet-18 model minimizing the sqarred crror loss. For the optimization, we rely on the Adam optimizer with the following parameters: learning rate of le-3 with a decay of 0.96 per epoch, batch size of 64 and early stopping based on the OOD evaluation score. For evalnation, we report the Pearson correlation (r) between the predicted and actual asset wealth indices across the five different folds.
B.2.2 On the challenge of obtaining domain labels.
In the example of hospitals (e.g. Camelyon17 dataset), domain labels come, in fact, for free. However, other examples, such as the CivilComments dataset, show the opposite. This dataset requires additional annotations (i.e., demographic identities), which can be tedious to obtain in practice. Some algorithms need these domain annotations to achieve superior performance on each subgroup. Furchermore, the task of subgroup detection in itself is a difficult and relevant problem. Coming back to our hospital example, even people from the same hospital might belong to different subpopulation (c.g. gender, race, age) and these demographic subgroups are often more relevant for diagnosis than which hospital a patient comes from. This information, however, is not always available (due to anonymization standards, for instance) and, therefore, the relevant domain annotation might be hard to obtain.
B.2.3 General benchmark methods
Following the WILDS benchmarking procedure Koh et al. 2021a), we compare our proposed Gated Domain layer to the following baselines. First, empirical risk minimization (ERM), which minimizes the average training loss over the pooled dataset. Second, a group of DG algorithms provided by the WILDS benchmark, namely, Coral, Fish, IRM, and DRO.
The Coral algorithm introduces a penalty for differences in means and covariances of the domains feature distributions (i.e., the activation of the last layer of neural networks) between the source and test domain. Thus, Coral is representative of the class of methods that promote the learning of feature representations that.
29

---

Published in Transactions on Machine Learning Research (09/2023)
have similar distributions across domains. The Fish algorithm achieves DG by approximating inter-domain gradient matching objective, i.e., maximizing the inner product between gradienis from different domains (Shi et al. 2021). Conceptually, Fish learns feature representations that are invariant across domains. Iuvariant risk minimization (IRM) introduces a penalty for feature distributions with different optimal classifiers for each domain (Arjovsky et al. 2019). The idea is to enable generalization by learning domain-invariant causal predictors. Lastly, group distributionally robust optimization (DRO) explicitly minimizes the training loss the worst-case domain (Sagawa al.) 2020 Hu et al.) 2018). This training method is an example of distribationally robust optimization (DRO). It aims to optimize for the potential test distributions' worst-case
loss.
In addition to the baselines originally presented in Koh et al. (2021a), we consider the following more recent DG baselines. First, we describe LISA, which instead regularizing the internal representations for generalization, seeks to learn domain-invariant predictors with selective data augmentation Yao al. (2022). Conceptually, Common Gradient Descent (CGD), introduced by Piratla et al.) (2022), emphasizes groups that enable better performance, thus differs from Group DRO that focuses groups with the highest regularization. CGD, technically speaking, focuses on shared gradients that result in improvement for all groups throughout the training process. Last, Adaptive Risk Minimization using batch normalization (ARM-BN) by Zhang et al. (2021) is different from the methods presented since it adapts to previously unseen domains during test time using unlabeled observations from this test domain.
30


Reviews:
[{"review_id": "bJZeURSna7", "reviewer": "Reviewer_o5Po", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This is a novel idea.\n    I like the idea. In fact, I want to use the same idea (IED assumption) to do my work last year.\n\n2. Clear presentation and organization.\n\nWeakness:\n1. I am little confused on Definition 1. There are no such f st R(g)>R(f). That is means R(f) >= R(g)? Is there some issues here?\n\n2. The exmperiments are week. Maybe more experiements are required.\n\n3. How to select the basis Vj?\n\n4.  Missing references:\n    Please cite \n    1) Moderately Distributional Exploration for Domain Generalization, ICML 2023\n    2) Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs\n\nIn summary: I think the idea of this paper should be carried forward. I like it from the theoretic views.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the domain generalization issue under the Invariant Elementary Distribution (I.E.D) assumption.\n\nFrom theory, the paper argues why the IED assumption is important and some novel theoretical results based on IED assumption.\n\nFrom algorithm, the paper designs GDUs to present the effectiveness of their idea.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Given more details on Definition 1.\n\nMore experiments to support the proposed methods.\n\nMore references to support the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XZ1MqJQtYb", "reviewer": "Reviewer_2Zu3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very clearly written and provides ample relevant links to previous literature in the area. I found the paper very easy to read.\n\nThe proposed modelling assumption is novel, as far as I am aware. Given that it is a strict generalisation of previous work that has assumed novel domains are a convex combination of seen domains, this new assumption makes sense. I.e., treating each domain as a mixture distribution and having novel domains be a novel combination of the underlying mixture components seems like a sensible approximation of the underlying problem.\n\nThe proposed neural network component is simple and well-motivated, having clear connections to previous work in kernel machines. The ability to train it with standard learning algorithms is also a bonus.\n\nThe experimental evaluation is quite thorough, and the analysis is level-headed and informative. The ablations, such as the analysis of the heuristic for selecting the number of latent domains, are useful.\n\nSomething that would be nice to include, but I am not sure how feasible it is, would be an investigation into the extent to which the proposed modelling assumption holds in practice. An easy way to partially address this might be to demonstrate that there are plausible scenarios where it is accurate, via constructing synthetic data for a simple but realistic problem. Ideally it would be good to demonstrate that the assumption holds on much more realistic data, but I am not sure this is possible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission studies the problem of domain generalisation, a particular instantiation of the out-of-distribution generalisation class of problems. A new modelling assumption about the structure of novel distributions, and how they relate to distributions seen during training, is proposed. It is a strict generalisation of some previous work in this area. A new neural network layer is developed that leverages this modelling assumption. Experiments are conducted on a number of DG benchmark datasets, measuring both performance on unseen domains and as investigations into the robustness of the method to choices of hyperparameters and inclusion of different components in the method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Additional analysis of whether the modelling assumption holds in practice is welcome, but not required for acceptance.\n\nI would recommend tweaking the first sentence of Section 2. Out-of-distribution generalisation is a much broader term than encompasses specific problem settings such as domain adaptation and domain generalisation, among many others.", "broader_impact_concerns": "It is hard to predict direct broader impact from this work, as it is quite foundational. Some aspects of potential broader impact are discussed in the introduction section while explaining why DG is a useful problem to study.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BKRHNDtlVE", "reviewer": "Reviewer_4JDc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. Interesting idea that seems to be novel; \n2. Simple, yet efficient implementation that can be used with different backbones;\n3. Theoretical justification \n4. Reasonable effectiveness \n\n**Weaknesses**\n\n1. Link between the theory and implementation is somewhat unclear\n2. Not clear whether the method is compared to a simpler baseline decomposing each test domain as a convex combination of source domains \n3. The results are good although the proposed method requires some ad-hoc routines to set its parameters ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new domain generalization (DG) method build on the idea of invariant elementary distributions. In short, the latter postulates that source and test domains are all composed of elementary distributions that remain invariant across them. This is contrary to other methods that seeks an invariant representation space that is invariant over the whole basis and not only over some of its elements. The authors further propose a gated unit to decompose test time domain into a convex combination of such IEDs using a similarity between them in the kernel mean embedding space. The results presented in the paper show that the method works and can provide a good OOD generalization on challenging datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major changes**\n\n1. I would really encourage the authors to revise their presentation of the theoretical results. After rereading Proposition 3.1 multiple times, I still do not understand how it connects to the algorithm that is proposed. In case of Proposition 4.1 it is much clearer but for the part on Pareto optimality I struggled to understand its importance. \n\n2. I feel that the experimental part of the paper can be made more clear with a more clear presentation of the baselines. I had to go through the text multiple times to see how the baselines differ one from another. \n\n**Minor changes**\n\nThe manuscript requires a thorough proofreading as it is currently contains a fair amount of typos and errors. \n\n1. The motivation is to become independence of domain labels -> independent \n2. The second challenge is the lack of effective architectures for these approaches, particularly ones that can be easily integrated into existing deep learning frameworks -> I'm not sure  that this is true, seems like the authors may want to downgrade this claim a bit\n3. In Tale 2,  -> Table 2\n4. reveals: M -> reveals methods\n5.  Davies-bouldinn Score -> Davies-Bouldin ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of domain generalization. Based on the Invariant Elementary Distribution (IED) assumption, which represents new domains as a convex combination of seen domains, the authors introduce Gated Domain Units (GDUs) to learn the representation for each latent elementary distribution. Experiments are conducted on Digits datasets and on standard datasets for domain generalization, i.e., DomainBed and WILDS. The paper initially received positive feedback. The reviewers appreciated the relevance of the proposal, the theoretical analysis, and the experiments. However, the reviewers also raised concerns about the validity domain of the IED assumption, insights into the theoretical results, and clarification on experiments for the baselines or hyper-parameter tuning. The authors' feedback, discussions, and the revised manuscript convinced the reviewers, who unanimously recommended the paper for acceptance.

The AE has carefully read the submission and the discussions. The AC considers that the approach is sound and relevant, and that the IED model and the proposed method to learn it are validated by evidence. The AE also appreciates the quality of the paper's presentation and the inclusion of the changes that have been requested by the reviewers. Therefore, the AE recommends acceptance. As a minor suggestion, the AE recommends that the authors clarify whether they will include new results to assess the validation of the IED assumption, as requested by R2Zu3 (though it is not required for acceptance).
Decision: Accept (poster)