Gated Domain Units for Multi-source Domain Generalization
Authors: Simon Föll
Alina Dubatovka
Eugen Ernst
Martin Maritsch
Patrik Okanovic
Gudrun Thaeter
Joachim M. Buhmann
Felix Wortmann
Krikamol Muandet
Venue: Submitted to ICLR 2023 (2023)
Keywords: Robust machine learning
domain generalization
out-of-distribition generalization
kernel theory
distribution shift
deep learning

Abstract: Distribution shift (DS) is a common problem that deteriorates the performance of learning machines. To tackle this problem, we postulate that real-world distributions are composed of elementary distributions that remain invariant across different environments. We call this an invariant elementary distribution (I.E.D.) assumption. The I.E.D. assumption implies an invariant structure in the solution space that enables knowledge transfer to unseen domains. To exploit this property in domain generalization (DG), we developed a modular neural network layer that consists of Gated Domain Units (GDUs). Each GDU learns an embedding of an individual elementary distribution that allows us to encode the domain similarities during the training. During inference, the GDUs compute similarities between an observation and each of the corresponding elementary distributions which are then used to form a weighted ensemble of learning machines. Because our layer is trained with backpropagation, it can naturally be integrated into existing deep learning frameworks. Our evaluation on image, text, graph, and time-series data shows a significant improvement in the performance on out-of-training target domains without domain information and any access to data from the target domains. This finding supports the practicality of the I.E.D. assumption and demonstrates that our GDUs can learn to represent these elementary distributions.

Under review as a conference paper at ICLR 2023
GATED DOMAIN UNITS FOR MULTI-SOURCE DOMAIN GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Distribution shift (DS) is a common problem that deteriorates the performance of learning machines. To tackle this problem, we postulate that real-world dis- tributions are composed of elementary distributions that remain invariant across different environments. We call this an invariant elementary distribution (LE.D.) assumption. The I.E.D. assumption implies an invariant structure in the solution space that enables knowledge transfer to unseen domains. To exploit this property in domain generalization (DG), we developed a modular neural network layer that consists of Gated Domain Units (GDUs). Each GDU learns embedding of an individual elementary distribution that allows us to encode the domain similarities during the training. During inference, the GDUs compute similarities between an observation and each of the corresponding elementary distributions which are then used form weighted ensemble of learning machines. Because our layer is trained with backpropagation. it can naturally be integrated into existing deep learning frameworks. Our evaluation on image, text, graph, and time-series data shows significant improvement in the performance on out-of tittoff-triinnggg target do- mains without domain information and any access to data from the target domains. This finding supports the practicality of the I.E.D. assumption and demonstrates that our GDUs can learn to represent these elementary distributions.
INTRODUCTION
A fundamental assumption in machine learning is that training and test data are independently and identically distributed (L..D.). This assumption ensures consistency-resule from statistical learning theory, meaning that the learning machine obtained from an empirical risk minimization (ERM) attains the lowest achievable risk as sample size grows (Vapnik, 1998; Schölkopf. 2019). Unfortunately. considerable amount of research and real-world applications in the past decades has provided staggering evidence against this assumption (Zhao et al., 2018; 2020; Ren et al., 2019; Taori et al., 2020) (see D' Amoor et al. (2020) for case studies). The violation of the I.I.D. assumption is usually caused by a distribution shift (DS) and can result in inconsistent learning machines (Sugiyama & Kawanabe, 2012), implying the loss of performance guarantee of machine learning models in the real world. Therefore, to tackle DS, recent work advocates for domain generalization (DG) (Blanchard al., 2011: Muandet et al., 2013; Li al., 2017: 2018b; Zhou et al., 2021a). This generalization to utterly unseen domains is crucial for robust deployment of the models in practice, especially when new, unforeseeable domains emerge after model deployment. However, the most important question that DG seeks to answer is how to identify the right invariance that allows for generalization.
The contribution of this work is twofold. First, we advocate that roal-world distributions are composed of smaller "units" called invariant elementary distributions that remain invariant across different domains; see Section 2.1. Second, we propose to implement this hypothesis through so-called gated domain units (GDUs). Specifically, we developed modular neural network layer that consists of GDUs. Each GDU learns an embedding of an individual elementary domain that allows us to express the domain similarities during training. For this purpose, we adopt the theoretical framework of reproducing kernel Hilbert space (RKHS) to retrieve a geometrical representation of each distribution in the form of kernel mean embedding (KME) without information loss (Berlinet & Thomas-Agno 2004; Smola et al., 2007; Sriperumbudur et al., 2010; Muandet et al., 2017). This representation accommodates methods based on analytical geometry to measure similarities between distributions.

---

Under review as a conference paper at ICLR 2023
We show that these similarity measures be learned and utilized to improve the generalization capability of deep learning models to previously unseen domains.
The remainder of this paper is organized as follows: Our thcoretical framework is laid out in Section 2 with our modular DG layer implementation shown in Section 3. In Section 4, we outline related work. Our experimental evaluations are presented in Section 5. Finally, discuss potential limitations of our approach and future work in Section 6.
2 DOMAIN GENERALIZATION WITH INVARIANT ELEMENTARY DISTRIBUTIONS
We assume a mixture component shift for the multi-source DG setting. This shift refers to the most common DS stating that the data is made up of different sources, each with its own characteristics, and their proportions vary between the training and test scenario Our work thus differs in the assumption from related work in DG, in which the central assumption is the covariate shift (i.e., the conditional distribution of the source and test data stays the same) (David et al., 2010). In the following, let X' and y be the input and output space, with a joint distribution P. We are given set of D labeled source datasets {D;) with D; C X V. Each of the source datasets is assumed to be I.I.D. generated by a joint distribution P; with support on N' x y. henceforth denoted domain. The set of probability measures with support on X x y is denoted by P. The multi-source dataset 'D* comprises the merged individual source datasets {D) 1- We aim to minimize the empirical risk, see Section 3.3 for details. Important notation is summarized in Table 1.
Mainonero-Candele et al., 2022).
2.1 INVARIANT ELEMENTARY DISTRIBUTIONS
Similar to Mansour et al. (2009; 2012); Hoffman et al. (2018a), we assume that the distribution of the source dataset can be de- scribed as a convex combination |P = ΣE= a3P5 where a* =
Table 1: Important notation
K
number of elementary distributions
(af, ,......,.0 (a) .b) is an element of the probability simplex, i.e., a² E = {a E RD aa 2 0 Ei= aj 1}. In other words, aj quantifies the contribution of each individual source domain to the combined source domain.
M
number of elementary domain bases
N
number of basis vectors
p
comhined multi-soure distribution
PM
j th single source distribution
In contrast, we generalize their problem descriptions: We ex- press the distribution of each domain as a convex combination of K elementary distributions {IP, 1y= C P. meaning that
P
j-th elementary distribution
V2
j-th domain basis
A-th vector in V2
ps ード a,P, where E AK. Our main assumption is that these elementary distributions remain invariant across the domains. The advantage is that we can find an invariant subspace more elementary level, as opposed to when we consider the source domains as some sort of basis for all unseen domain. Figure 1 illustrates this idea.
coefficient for P7
a
cuefficient for F'y
8.g
coefficient for sample x, and μVv,
Theoretically speaking, the I.E.D assumption is appealing because it implies the invariant structure in the solution space, shown in the following lemma. The proof is given in Appendix 1.
Lemma 1. Let L : y x y + R4 be a non-negative loss function, F a hypothesis space of functions f: X + Y, and Pp (Px Y) data distribution. Suppose that the I.F.D) assumption holds, i.e., there exist K elementary distributions PI,...,
......,rr PK such that any data distribution can be expressed as (X, = ΣØ ,X,, Y) for some a E AK, Then, the corresponding Bayes predictor I E arg minser E B ((X)) is Pareto-optimal with respect to a vector of elementary risk functionals (Rı...., Rk): : F RK where R2(f) == E(x,))=,,) [C(Y, (CY,, fX))]-
Lemma implies that, under the L.E.D assumption, Bayes predictors must belong to a subspace of I called the Pareto set F C F which consists of Pareto-optimal models. The model f is said to be Pareto-optimal if there exists no g E F such that Rg(g) 2 (f) for all j E (1.... K} with (g) > R3 (f) for some j; see, e.g.. Sener & Koltun (2018, Definition 1). In other words, the I.E.D assumption allows us to translate the invariance property of data distributions to the solution space. Since Bayes predictors of all future test domains must lie within the Pareto set, which is
2

---

Under review as a conference paper at ICLR 2023
31
d(z))
0.8
812
0.
0.0
0.4
HV
0.2
1/1
1/2
During training
During inference
Figure 1: A visualization of an "invariant elementary distribution (I...D.) assumption for domain generalization (DG): the observed data distributions (orange and violet) are composed of the same set of unobserved elementary distributions (blue and red) that remain invariant across different domains. Hence, the first challenge during the training phase (left panel) is to extract these elementary distri- butions from the observed data (orange). The unobserved elementary distributions by the elementary bases V1 and V2 (cyan and pink). The second challenge during the inference phase (right panel) is create weighted ensemble of learning machines that utilize the similarities between the embedding of the unseen observation ((x;;) and the embeddings of these distributions j4v, and nv, in the RKHS H (green rectangle) weights 31 and 8:2-
represented
strict subset of the original hypothesis space, it is still possible identify the optimal predictors of future test domains, even without additional data from the test domains, except the I.E.D. assumption itself. Hence, given data from the training domains, it is sufficient for the purpose of generalization to maintain only solutions within this Pareto set during the training time.
Unfortunately, neither the elementary distributions nor the weights are known in practice. Motivated by this theoretical insight, our DG layer presented in Section is designed to uncover them from multi-source training dataset D While Lemma shows the theoretical appeal of the I.E.D. assumption, we discuss below a situation in which it might hold in practice. The limitations will be discussed later in Section 6.
Real-world example. In this work, we postulate that the elementary domain bases are the invariant subspaces that allow us to generalize to unseen domains. In practice, the question arises if and when elementary domains evolve. Consider that we aim learn to predict the risk of developing Diabetes from laboratory data from Europe and then infer the risk from data from the United States of America. Naturally, factors influencing the data-generating process may change, such as the level physical activity and nutritional habits. While, to certain degree, these common factors remain invariant across continents, each of these factors' contributions may differ. In terms of our assumptions, we model each of these factors with corresponding elementary distribution Py. For previously unseen individual, we can then determine the coefficients ag and quantify each factor's contribution without any information about the individual's origin.
2.2 KERNEL MEAN EMBEDDING OF DISTRIBUTIONS
We leverage the KME of distributions (Berlinet & Thomas-Agnan 2004; Smola et al., 2007; Muandet et al.. 2017) to discover the elementary distributions and evaluate similarities between them. Let H be a reproducing kernel Hilbert space (RKHS) of real-valued functions on X' with a reproducing kernel k: X x X' + R (Schölkopf et al., 2001). The KME of a probability measure P E ' in the RKHS is defined by a mapping ((P)) μr := fxx (xx, -) dx)(x). We assume that the kernel is characteristic, i.e., the mapping μup is injective (Fukumizu et al., 2004; Sriperumbudur et al., 2008). Theoretically, this essential assumption ensures that there no information loss when mapping the distribution into H. Given the samples (.1,.... In) generated I.L.D from P. μp be approximated by the empirical KME Aup = (1/n) Σ k(x,,)) :) = (1/n) Σ d(x)). We refer non-expert readers to Muandet et al. (2017) for thorough review this topic.
3

---

Under review as a conference paper at ICLR 2023
Challenges. Figure 1 depicts two challenges that come with our assumption of elementary distribu- tions. First, since we do not have access to the samples from the hidden elementary distributions, the elementary KME cannot be estimated directly from the samples at hand. To overcome this challenge, we instead seek proxy KME μV; (L/N) Ek=1 ((v))
(1/N) EN k(r for each elementary KME μe, from domain basis Vj. where V3 = {ui,.... .Nr S X for all j E [L...., 3. Hence, the KME μV, can be interpreted as the KME of the empirical probability measure Pv; = (1/N) Ex= Rzg. Here, we assume that M = K. The sets Vi are referred to as elementary domain basis. Intuitively, the elementary domain basis Vi,..., Vw represents each elementary distribution by a set of vectors that mimic samples generated from the corresponding distribution. In Figure 1, V/1 and 1/2 as well as their mapping in H. visualize this first challenge.
The second challenge is the objective of learning the unknown similarity between a single sample æ; and an elementary domain Vj. which we denote by Bij. Considering the advantage of KMEs, that to tackle this challenge from a geometrical viewpoint, we quantify similarities between KMEs. For example, in Figure I, the similarity between Φ(x,) and μıv; (B:1) and μvv, (8x2) could be quantified as their distance angle. These similarity coefficients enable our Domain Generalization Layer to represent a convex combination of elementary domain-specific learning machines, commonly known as ensembles. We introduce our layer in the following Section 3.
3 DOMAIN GENERALIZATION LAYER
This section aims to transfer the theoretical ideas presented in Section 2 into a deep learning framework. For the purpose of implementation, let E R denote the input data point and he R + R the feature extractor (FE) that maps the input into a low-dimensional representation R Then the prediction layer go R + y infers the label y. To tackle the DG problem, we introduce a layer module called the gated domain unit (GDU). A GDU consists of three main compo- nents: (1) similarity function ? H x H -> IR that is the same for all elementary domains, (2) an elementary basis V3 and (3) learning machine f(ī,0) for each elementary domain j E {L..... M).
The architecture the layer proposed herein is depicted in Figure 2.
Ø
Bg
CDU
CDUSs
(((()) mVg)
V2
Py;
Visualization of the DG layer (left panel) and its main component, the GDU (right panel). The DG layer consists of several GDUs that represent the elementary distributions. During training, these GDUs learn the elementary domain bases Vi,..., Var that approximate these distributions.
Figure 2:
Essentially, the process as follows: First, the j-th GDU takes on as an input and yields B, as an output. The KME of each domain basis Vi is required in order to apply y to compute simi- larity between &4 and Vj. These KMES are obtained by <(V)) == ,v, = (1/N) E d(v²) (1/N) ΣN k= k( .). The GDU, therefore, has the task to allocate coefficients Big for each ele- mentary domain based on a similarity function y. The function 7 outputs the 3,j = ((((;)) ,v;) coefficients that in turn represent similarities between the KME of both, the corresponding domain basis V, and the input ë.. Theoretically speaking, WV, and the feature mapping ((x)) are elements of the associated RKHS H. which allow to evaluate similarities of non-lincat features in higher dimensional feature space. Each GDU is then connected to a learning machine fă,, 6,) that yields an elementary domain-specific inference. The final prediction of the layer is then an ensemble of
4

---

Under review as a conference paper at ICLR 2023
these learning machines (²,) Bey where 0 = (01,...,,0a, In Figure 2, give an overview of how data is processed and information is stored in the GDU.
In summary, GDUs leverage the invariant elementary distribution (I.E.D.) assumption and represent our algorithmic contribution: The elementary domain bases are stored weights in the layer. Storing information as a weight matrix (i.e., domain memory) allows to learn the elementary domain bases efficiently using backpropagation. Hence, avoid the dependency on moblem-addapiiv methods (e.g., domain adversaal training) and domain information (c.g., domain labels).
3.1 DOMAIN SIMILARITY MEASURES
For the similarity function 7, we consider two similarity measures H()(ū), jv, namely the cosine similarity (CS) (Kim et al., 2019) and maximum mean discrepancy (MMD) (Borgwardt et al., 2006; Gretton et al., 2012). To ensure that the resulting coefficients B, lie the probability simplex, apply the kernel softmax function (Gao et al., 2019) and interpret its output as the similarity between an observation & and elementary domain basis V,. We get
Big = ,v, = (xp(x:H((5.))) Hvy))
(o(A)) (x))))))((()) (v.))
where K >> 0 is positive softness parameter for the kernel softmax. Geometrically speaking, these similarities correspond to the angle and distance two KMEs the RKHS H. The function & maps the observation I and domain basis Vi into H meaning that ((T) μóg = k(ū,-) is the KME of Dirac measure 63 and (VV)) = μv, = (1/N) E k(rgs)).
(.v.)). CS. The CS function H(}(;:), To( IPyllx
is used angle-based similarity.
MMD. We consider the MMD for calculating a distance-based similarity measure. The distance is then given (Ø(ž;) fv, llve. Subsequently, the similarity function H is the negative MMD: H (o(#)), (v,) = - [)(;;) - fV, ||v. The intuition behind the negative MMD is to put higher weights on samples that closer to the KME of elementary domain basis.
3.2 PROJECTION-BASED GENERALIZATION
For classification tasks, we introduce an alternative approach to infer the 33 coefficients that based on the idea of kernel sparse coding (Gao al., 2010; 2013). Herein the goal is to find an approximated representation of each feature mapping ((x+) using the elements of a dictionary {uv, LM M J==1- This
approach allows us to approximate the feature mapping with these elements by ((ž,) Σド Bippy, In contrast to the aforementioned approaches, an elementary domain KME μU, does not necessarily represent the KME of an elementary distribution μup, Therefore, we present another approach that aims to find set (nv; 1 that permits μ4p= to be represented as a linear combination.
Since P is assumed to be convex combination of elementary distributions, we can find a linear combination to represent utz by the domain KMES ,V,, as long as /4p= E Hrs == span(Mv, |j 1...., M.. The RKHS HM a subspace of the actual RKHS H. which allows us to represent elements of H at least approximately in the subspace Har By keeping the HM large, we gain more representative power. To make H. as large as possible, we have to ensure its spanning elements are linearly independent
even better, orthogonal. Orthogonal KMEs ensure two desirable properties. First, pairwise orthogonal elements in HM guarantee no redundancy. Second, having orthogonal elements allows to make use of the orthogonal projection. This projection geometrically yields the best approximation of ((t) in H other words, we can achieve the best possible approximation of the feature mapping by using its orthogonal components (see Proposition 3. The orthogonal projection is given by Nx H-- HM, ((ū) +
M (Φ(T), HV, HV.
(2)
1w 1
5

---

Under review as a conference paper at ICLR 2023
Proposition 3.1. For KME un of given mixture distribution IP the following holds pi span(ev, Vj.Vj = I.....M, where (Jv,, My,,), = 0, Vi + 3 (i.e., the KME of the elemen- tary domains basis are pairwise orthogonal). The value of the function Σ Eimiluu - Byuv, li is minimal if the coefficients are set as 83 = (pp, py,)n// /////////,,, l1
The Proposition 3.1 can be used to give an approximation of ue by projecting it into Has i.c. Hu Rs Σ}= 3y where 3, = (u,, ///////l /llu,,, 11 This best approximation property is the main advantage of our assumption in Proposition 3.1 (i.e., having orthogonal KME) and thus potential advantage of ingjeciionbbseed DG. Appendix A.2 provides the proof of Proposition 3.1.
3.3 MODEL. TRAINING
For model training, we adapt the domain adaptation (DA) framework from Zhuang et al. (2021). Thus, our learning objective function is formalized as L(g) + *DDDp(1lllxx) The goal of the training can be described in terms of the two components of this function. Consider batch of training data {x1,..., }, where b the batch size. During training, we minimize the loss function ((g)) = = Ei= L(ū.- v1) = + E ا(Eم= ((((,)) MV3) (f(())))))) for an underlying task and the respective batch size. In addition, our objective is that the model learns to distinguish between different domains. Thus, the regularization 12p is introduced
control the domain basis. In our case, we require the regularization 12p to ensure that the KMES of the elementary domain basis are able represent the KMES of the elementary domains. Therefore, we minimize the MMD between the feature mappings ((+) and the associated representation ΣØ= Bijtyn. Note that Big = (b(T;)) (V,)). Hence, the regularization Ωn 2DLL is defined as S2gLS (I1al1)) =
E [l()))) E) Besluv, 1/24 (see Appendix B.2 for details). The intuition the objective to represent each feature mapping ((ā:)) by the domain KMFS ,uv,- Thus, we try to minimize the MMD) between the feature map and combination of 'v, The minimum of the stated regularization can be interpreted as the ordinary least square-solution of a regression pagression-rroblemm of o(z;) by the components of H.M In other words, we want ensure that the basis V₁ contained in feature mappings d(ā;).
In the particular case of projection, we want the KME of the elementary domain to be orthogonal to ensure high expressive power. For this purpose, the additional term L will be introduced to ensure the desired orthogonality. Considering a kernel function with k(x, x) = 1, orthogonality would require the Gram matrix Ki = (ju,, : (Vv;) 7 to be close to the identity matrix J. There are a variety of methods for regularizing matrices available (Xie et al., 2017: Bansal et al., 201 8). A well-known method to ensure orthogonality is the soft orthogonality (SO) regularization 2 = A||K- - 111 (Bansal et al., 2018). As pointed out by Bansal et al. (2018), the spectral restricted isometry property (SRIP) and mutual coherence (MC) regularization can be a promising alternative for SO and thus additionally implemented in the DG layer. Hence, in the case of projection, the regularization is given by 2p (199] = Aor.s 20t. (I19l)) + AORTHIL (I19lHH), Aor.s: AORTH 2 0.
Lastly, sparse coding is efficient technique to find the least possible basis to recover the data subject to a reconstruction error (Olshausen & Field, 1997). Several such applications yield strong performances, for example in the field of computer vision (Lee al., 2007; Yang al., 2009). Kernel sparse coding transfers the reconstruction problem of sparse coding into H by using the mapping é, and, by applying kernel function, the reconstruction error is quantified as the inner product (Gao et al., 2010; 2013). To ensure sparsity, we apply the Loom on the coefficients 3 and add 2% (I/rl))= == (,;)), (id)),yy) HV2))l1 to the regularization term Up with the corresponding coefficient AL. Appendix B.3 gives visual overview of the model training.
RELATED WORK
DG, also known as out-of-distribution (OOD) generalization, is among the hardest problems in machine learning (Blanchard et al., 2011; Muandet et al., 2013; Arjovsky et al., 2019). In contrast, DA, which predates DG and OOD problems, deals with a slightly simpler scenario which some data from the test distribution available (Ganin et al., 2015). Hence, based on the available data, the task is to develop learning machines that transfer knowledge learned in a source domain specifically to the target domain. Approaches pursued in DA can be grouped primarily into (1) discrepancy- based

---

Under review as a conference paper at ICLR 2023
DA (Sun et al., 2016; Peng & Saenko, 2018: Ben-David et al., 2010; Fang et al., 2020; Tzeng et al., 2014; Long et al., 2015; Baktashmotlagh et al., 2016) (2) adversary-basee
DA (Tzeng et al., 2017: Liu DA (Bousmalis et al., 6016; Hoffman et al., 201 Kim et al., 2017; Yi et al., 2017; Zhu et al., 2017; Ghifary et al., 2014). In DA, learning the domain-invariant components requires access to unlabeled data from the larget domain. Unlike problems in DA, where the observed data from the test domains can be used to find the most appropriate invariant structures (Ben-David al., 2010), the lack thereof in DG calls for postulation of invariant structure that will enable the OOD generalization.
& Tuzel, 2016; Ganin et al., 2015; Long al., 2018), and (3) reconstruction-basee
To enable generalization to unseen domains without any access to data from them, researchers have made significant progress in the past decade and developed a broad spectrum of methodologies (Zhou et al., 2021a;e; et al., 2019; Blanchard et al., 2011). ). For thorough review see, e.g., Zhou al. (2021a); Wang et al. (2021). Existing works can be categorized into methods based on domain- invariant representation learning (Muander et al., 2013; Li et al., 201 8b;d),,, meta-learning (Li et al., 2018a; Balaji et al., 2018), data augmentation (Zhou et al., 2020), to name few. Another recent stream research from a causal perspective includes invariant risk minimization (Arjovsky et al., 2019), invariant causal prediction (Peters al., 2016), and causal representation learning (Schölkopf et al., 2021). The overall motivation here is to learn the representation that is robust to domain-speciffe spurious correlations. In other words, it is postulated that "causal" features are the right kind of invariance that will enable OOD generalization. Despite the successful applications, DG remains a challenging research gap.
We differentiate our work from existing ones as follow. First, we postulate the existence of domain- invariant structure at the distributional level rather than the data representation, which is a common assumption in DG. This is motivated by theoretical results (Mansour et al., 2009; Hoffman et al., 2018a), stating that a distribution- weithhhed combination of source hypotheses represents the ideal hypothesis. Furthermore, our distributional assumption, as we argued in Section 2, generalizes previous work that proposes to use domain-specifie knowledge to tackle the problem of DG from more elementary setting. For example, approaches such as Piratla et al. (2020); Monteiro et al. (2021) can be compared to our GDUs as domain- -ssecfff- predictors, in the special case, where each elementary domain represents single source domain. However, GDUs do not assume the existence of a single common classifier for all the domains, providing a combination of multiple common classifiers shared between different source domains.
Second, we incorporate the I.E.D. assumption directly into our model's 's architecture, as shown in Figure 2. Designing effective architectures for DG has been largely neglected (Zhou et al., 2020, Sec. 4.1). Last, we do not assume access to domain information. Although obtaining such information can be difficult in practice, see our short discussion in Appendix C.4 (Niu et al., 2017), DG methods that can deal with their absence (e.g., Huang et al. (2020); Carlucci et al. (2019); Li et al. (2018c)) are yet scarce (Zhou et al., 2020, Sec. 4.2).
EXPERIMENTS
Since ERM is one of the strongest baselines in DG (Gulrajani & Lopez-Paz, 2020; Koh et al., 2021), we, first, compare our approach compared to ERM and ensemble learning (Table 2 and Appendix C.I). Second, we benchmark our approach state-of-the-art DG (e.g., CORAL, LISA, IRM, FISH, Group DRO) methods focusing on image, graph, and text data (Table 3 and Appendix C.4). Third, we analyse the GDUs robustness gainst DS that occurs daily clinical practice (Table 12 and Appendix C.3). Finally, in Appendix C.2, we conduct an ablation study focusing on the representation learned during training (Appendix C.2.2). In our experiments, we distinguish two modes of training the DG layer: fine tuning (FT), where we extract features using pre-trained model, and end-to-end training (E2E), where the FE and the DG layer are jointly trained'.
5.1 PROOF-OF-CONCER BASED ON DIGITS CLASSIFICATION
Following Feng et al. (2020) among others, we create multi-source dataset by combining five publicly available digits image datasets, namely MNIST (Lecun et al., 1998), MNIST-M (Ganin & Lempitsky, 2015), SVHN (Netzer al., 2011), USPS, and Synthetic Digits (SYN) (Ganin &
All source code made available GitHuh.

---

Under review as a conference paper at ICLR 2023
Lempitsky, 2015). The task is to classify digits between zero and nine. Each of these datasets is considered an out-of-training target domain which is inaccessible during training, and the remaining four are the source domains. Details are given in Appendix C.1. Table 2 summarizes the results for the most challenging out-of-training target domain, namely MNIST-M. In Appendix C.I, we provide the results on the remaining target domains in Table 7 and discussion heuristics for choosing hyperparameters for our GDUs, Our method noticeably improves for all datasets mean accuracy and decreases the standard deviation in comparison to the ERM and ensemble baselines, making the results more stable across the ten iterations reported.
We also compare our methods with related work that uses domain information and data augmentation, based on the results of Li et al. (2021) (Table 9 in Appendix C.I). Although data augmentation in DG is a comparatively strong approach and, the same time, we do not use domain information; we obtain comparable results to the baselines reported by Li et al. (2021).
Table 2:
Results Digits experiment. The mean (standard deviation) ac- curacy for ten runs is reported. Best results are bold.
MNIST-M
Ablation study We chose the digits dataset to analyze each component of our DG layer in 1st paragraph in Appendix C.1 and C.2. We (A) vary M, .V on Figure 9, and the strength of the regularization terms on Figure 6, Figure 7, and Figure 8 to assess the sensitivity of the DG layer to the choice of hyper- parameters, (B) visualize the output of the FE (Figure 11). Our ablation study in (A) reveals stable results across different sets of hyper-pann paramearraettrs. While the layer is not sensitive to the choice of regularization strength, we recommend not to omit the regularization completely, although the computational expenses decrease without the orthogonal regularization. As illustration in (B), we project the output of the FE trained with a dense layer (ERM) and with the DG layer by t-SNE (t-distributed stochastic neighbor embedding). The GDU-trained FE yields more concentrated and bounded clusters in comparison to the one trained by ERM. Hence, we observe a positive effect on the representation learned by the FE.
ERM
Single
63.00 (3.20)
Ensemble
62.87 (1.50)
CS
(0.80)
MMD
68. 62.62 (0.70)
PROJECTION
68.56 (0.91)
CS
69.25 (0.61)
R2E
MMD
69.04 (0.83)
PROJECTION
68.67 (0.98)
5.2 WILDS BENCHMARK
To challenge the I.E.D. assumption and the OOD generalization capabilities of the GDUs, we use WILDS, a curated set of real-world experiments for benchmarking DG methods (Koh et al., 2021). Further, WILDS is semi-synthetic dataset set that operates under similiar assumptions as the source component shift (Koh et al., 2021). We consider the following eight datasets: Camelyon17, FMoW, FMoW, Amazon, iWildCam, and RxRx/, OGB-MMlPCHA, Civil-Comments, and PovertyMap, which represent the task of real-world DG. We closely follow Koh et al. (2021) for the experiments. Details on datasets and benchmark methods are given in Appendix C.4. We present our benchmarking in Table 3. Our results are achieved out-of-the-bor (i.e., default parameters) since hyperparameter optimization has substantial impact on the generalization performance (Gulrajani & Lopez-Paz, 2020), and we aim to highlight the improvements solely attributable to our GDUs.
First, we observe the strengths and weaknesses of the benchmarks in the different data sets, all of which are lower than FRM at least once. In contrast, although GDUs show similar behavior across the datasets, performing very well for some datasets (e.g.. FMoW, Poverty Map), they, however, do not fall below ERM across all GDU experiments conducted. In addition, the baselines require domain information. Our approach requires less information, yet, achieving comparable results the benchmarks.
5.3 ECG EXPERIMENT
The PhysioNet/Computing in Cardiology Challenge 2020 (Perez Alday et al., 2021: Goldberger et al., 2000; Perez Alday et al., 2020) aims to identify clinical diagnoses from 12-lead ECG recordings from 6 different databases. This publicly available pooled dataset contains 13,101 recordings sampled with various sampling frequencies and lengths. Each recording is labeled having or more of 24 cardiac abnormalities; hence, the task is to perform a multi-label binary classification. For our experiment, we iterate over the databases, taking one at time the test domain while utilizing
8

---

Under review as a conference paper at ICLR 2023
Table 3: Results on WILDS benchmarking tasks. Our results are achieved out-of-the-box (i.e., default parameters) without hyperparameter optimization. We use grey background to highlight methods using no domain information for DG. We compute the metrics following Koh al. (2021) and report the mean (standard deviation). Best benchmark and GDU results are bold.
CAMELYON17
FMoW
AMAZON
IWILDCAM
RXRXI
OGB- MoLPCBA AVG WORST- REGION Acc
CIVIL COMMENTS WORST- GROUP Acc
POVERTY
MAP WORST-UUR R
AvG AcC
REG AcC
WORST-
10% ACC
MACRO F1
AVG Acc
KRM
(6.4)
31.3 (0.17)
(3.8(0.8)
(1.3
(0.4)
27.2
(3.66
0:45 (0.06)
(0.4)
(0.3)
26.9
55.6 (0.8)
0.52 (0.08)
CORAL
(7.3
32.8 (0.66)
(0.9
32.8 (0.1)
28.4 (0.3)
17.9 (0.5)
65.6 (1.3)
0:44 (0.07)
FISH
(3.1)
34.6 (0.18)
53.3 (0.0)
22.0(1.8)
75.3 (0.6)
IRM
(8.1))
(4.9)
0.1)
0:43 (0.07)
GROUP DRO
(7.3)
31.1 (1.66)
(0.0)
(2.9
22.5 (0.3)
22.4 (0.6)
68.4 (7.3)
0.39 (0.05)
LISA
77.1 (6.9)
35.5 (0.81)
$4.7 (0.0)
31,9 (1.0)
72.9 (1.0)
CGD
(3.9)
(2.26)
69.1 (1.9)
0.43
(0.54)
(2.8)
31.2 (0.1)
CS
(8.3)
(1.2)
54.2 (0.8)
31.2 (0.8)
29.9 (0.3)
(0.2)
(6.0)
0:46 (0.07)
MMD
31,9 (8.0)
54.2 (0.8)
31.2 (0.8)
29.9
27.5 (0.2)
55,9 (3.7)
0.50 (6.06)
66.7 (8.9)
(1.0)
(0.8)
(1.4
(0.3)
27.4 (0.2)
(2.9)
0.49 (0.07)
CS
(8.9)
(1.9)
$3.9 (0.7)
27.8
29.7 (0.4)
26 (0.1)
$5.9 (0.8)
0.46 (0.07)
E2E
MMD
(6.7)
34.4
(0.2)
0.50 (0.06)
PRO
(9.5)
32.9 (0.8)
53.8 (0.8)
30.1 (1.2)
(0.2)
26.6 (0.3)
(2.1)
0.49 (0.07)
the remaining five databases for training. The performance was measured according to the original score assigns partial credit to
PhysioNet challenge score. This generalized intersection-over-ume misdiagnoses that result in similar treatments or outcomes. The score is then adjusted for solution that always selects the normal/majority class and normalized for the perfect solution. Therefore, the score can have negative values and best possible score of 1.
Table reports results for the ECG experiments (see Appendix C.3 for details). For this clinical time-series data, we observe an improvement in mean score and a reduction in standard deviation over the ERM and ERM ensemble baselines across all DG tasks. We attribute poorer performance for the PTB dataset to the fact that it contains considerably longer recordings than other datasets (except for INCART which, however, contains only 75 samples) and a higher sampling rate (1000Hz vs. 500Hz and 257Hz). The negative challenge score for the PTB-XL dataset is due the presence of previously unobserved labels in other datasets as well as a considerably smaller amount of data for training since the PTB-XL dataset comprises the majority of all samples (21,837 out of 43,101).
Table 4: Results ECG experiment. All experiments were repeated five times and the mean (standard deviation) challenge metric is reported. Higher is better. Best overall results are highlighted bold.
CPSC
CPSC-EXTRA
INCART
PTB
PTB-XL
G12EC
Single
0.0840 (0.0220) 0.2715 (0.0270)
0,2290 (0.0059)
-8,8206 (0,3908)
-0.3373 0403)
0,2011 (0.0015)
FRM
Encemble
0. 0699 (0.0346)
0.2488 (0.0079)
0.2456 (0.0109)
-8,9115 (023)
-0.4136 (0.0780)
(0.0161)
CS
0.1830 (0.0061)
0.2950 (0.0035)
0.1595 (0.0313)
88,802 (0.1069)
-0.1932 (0.0168)
0.1853 (0.0036)
FT
MMD
1877 (0.0077) 0.3011 (0.0035) 0.2100 (0.0413)
-88008 (0. (458)
-0. 1567 (0.0211)
0.1919 (0.0036)
PROJECTION
0,1941 (0.0050) 0.3135 (0.0015)
10.1041 (0.0015)
888817 (0.0478)
00.2166 (0.0191) 0.2409 (0.0042)
CS
0.1067 (0.0170) 0.2866 (0.0146) 0,2539 (0.0289)
-9.2947 (0.3004)
-0.1651 (0.0494)
0.1927 (0.0080)
E2E
MMD
1.34 (0.0143) 0.2834 (0.0228) 0.2398 (0.0257) -9.0600 (0.3100)
-0.1433 (0.0293)
0.1925 (0.0067)
PROJECTION 0.1411 (0.0269) 0.2962 (0.0065) -0.1467 (0.0513)
-8.5904 (0.3310)
-0.0178 (0.0291)
0.2947 (0.0117)
CONCLUSION AND DISCUSSIONS
We introduced the I.E.D. assumption, postulating that real-world distributions are composed of elementary distributions that remain invariant across different domains and showed that implies an invariant structure in the solution space that enables knowledge transfer to unseen domains. Empirical results based on real-world data support the practicality of the L.E.D. assumption and that we can leamn such representation. Further, presented modular neural network layer consisting of Gated
9

---

Limitations. A major limitation of our L.EDD. assumption is to provide theoretical evidence that this assumption holds in practice. We aim to expand the scope the theoretical understanding of the I.E.D. assumption and the GDUs. In addition, the particular theoretical setting of Albuquerque et al. (2019) (i.e., each elementary domain represents a source domain) seems promising to extend their generalization guarantee to cases where our I.E.D. assumption holds. Second, our GDU layer induces additional computational overhead due to the regularization and model size that increases function of the number of elementary domains. Noteworthy, our improvement is achieved with a relatively small number of elementary domains indicating that the increased complexity is not consequence of Under review as a conference paper at ICLR 2023
Domain Units (GDUs) that leverage the LE.D. assumption. Our GDUs substantially improve the downstream performance of learning machines in real-world DG tasks. Across our experiments, we observed that for some datasets FT is better than E2E and vice versa. In E2E training, the feature extractor (encoder) is jointly trained with GDUs. Hence, the latent representation is stochastic during training, meaning that we have variability in the representation fed into GDUs between epochs. In contrast, in FT, the feature extractor is pretrained and always produces the same embedding. Especially with large feature extractors such ResNet-50, learning the elementary domains can be more effective when we avoid any stochasticity in the latent representation.
coercive consequence of applying the DG layer. Also, the results achieved increased complexity, as the ensemble baseline shows.
Future work We expect the L.ED, assumption and GDUs be adapted, yielding novel applications that tackle DG. For example, we suggest dynamically increasing the number of elementary domains during learning until their distributional variance reaches plateau as measure of their heterogeneity. Hence, one would learn the number of elementary domains instead of fixing the number of elementary domains prior to training.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H. Falk, and Ioannis Mitliagkas. Generalizing unseen domains via distribution matching. ArXiv Prepr. ArXiv191100804, 2019.
Martin Arjovsky, Léon Bottou, Ishean Gulrajani, and David Lopez-Paz. Invariant risk minimization. 2019. URL http::///// arxiv. org/abs/////eeeooo.ee/..ee//eaaa /1907 C2893. arxiv:1907.028933
Mahsa Baktashmotlagh, Mehrtash Harandi, and Mathieu Salzmann. Distribution-matching embed- ding for visual domain adaptation. J. Mach. Learn. Res., 17:1-30, 2016. ISSN 15337928.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. MetaReg: Towards domain gen- eralization using neta-regularization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018
N.
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regu- larizations in training deep networks? In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Mach. Learn. 19(--2) 2010. ISSN 15730565.
doi: 10/c5bhdk.
Alain Berlinet and Christine Thomas-Agnan. Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic Publishers, 2004.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from Several Related Classification Tasks to New Unlabeled Sample. In Advances in Neural Information Processing Systems, volume 24, pp. 9, 2011.
Karsten M Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by Kernel Maximum Mean Discrepancy. Bioinformatics, (2(14))e4--e57 July 2006. ISSN 1367-4803. doi: 10/czwvqs.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain Separation Networks. In D Lee, M Sugiyama, Luxburg, Guyon, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Fabio M. Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain Generalization by Solving Jigsaw Puzzles. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2224-2233 Long Beach, CA, USA, June 2019. IEEE. doi: 10/gh7pq4.
ISBN 378-1-7281--293-8.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification Presents Challenges for Credibility in Modern Machine Learning. November 2020. doi: 2011 1. 1.03395v2.
Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 129-136. JMLR Workshop and Conference Proceedings, 2010.
Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan. and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11996-12007 Curran Associates, Inc., 2020.
11

---

Under review as a conference paper at ICLR 2023
Hao-Zhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, and Chen Wei. KD3A: Unsupervised Multi- Source Decentralized Domain Adaptation via Knowledge Distillation. ArXiv Prepr. ArXiv201109757 2020. doi: 2011.09757.
Kenji Fukumizu, Francis R. Bach, and Michael Jordan. Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. Journal of Machine Learning Research, 3:73-99, 2004.
Yaroslav Ganin and Victor Lempitsky. Unsupervised Domain Adaptation by Backpropagation. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 1180-1189. PMLR, June 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-AAversarral Training of Neural Networks. Adv. Comput. Vis. Pattern Recognit., 1707883195834441188-2999 May 2015. ISSN 21916594, doi: 10/gige2j.
S. Gao, I. W. Tsang, and I.. Chia. Sparse Representation With Kernels. IEEE Transactions on Image Processing, 22(2):423-434 February 2013. ISSN 1941-0042. doi: 10/f4nhfq.
Shenghua Gao, Ivor Wai-Hung Tsang, and Liang-Tien Chia. Kernel Sparse Representation for Image Classification and Face Recognition. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios (eds.), Computer Vision ECCV 2010. pp. 1-14. Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642-15561-11
Yingbo Gao, Christian Herold, Weiyue Wang, and Hermann Ney. Exploring Kernel functions in the softmax layer for contextual word classification. arXiv, 2019. ISSN 23318422.
Muhammad Ghifary, W. Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recognition. Lect. Notes Comput. Sci. Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinforma. 8862:898 1089,904 2014. ISSN 16113349, doi: 10/gige2m.
A. L. Goldberger, L. A. N. Amaral, L. Glass, M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation, 101(23):e215-e2D3 June 13 2000. doi: 10.1 10.1161/01.CIB R0L///..CIR..11..1 101.230.33 URL http::/circe ahwrc..aajouraaa tronic Pages: http:://ctccc.cc wtwp://c......uua..oooooooo////a
full. Circulation Elec- PMID:1085218:
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13:723-773, 2012. ISSN 15324435.
Ishaan Gulrajani and David Lopez-Paz. Paz. In Search of Lost Domain Generalization. ArXiv200701434 Cs Stat, July 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. ISBN 1063-6919. doi: 10/gdcfkn.
C. Hillermeier. Generalized homotopy approach to multiobjective optimization. Journal of Optimiza- tion Theory and Applications, 110(3)557-583, 2
C. Hillermeier. Nonlinear Multiohjecttive Optimization: Generalized Homotopy Approach. Interna- tional Series of Numerical Mathematics. Springer Basel AG, 2001b.
Judy Hoffman, Mehryar Mohri, and Ningshan Zhang. Algorithms and theory for multiple-source adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018a.
Judy Hoffman, Eric Tzeng. Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. CyCADA: Sycle-Consistee Adversarial Domain Adaptation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1989-1998, Stockholmsmässan, Stockholm Sweden, 2018b. PMLR.
12

---

Under review as a conference paper at ICLR 2023
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2029-2037 PMLR, July 2018.
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. 2017 IEEE Conf. Comput. Vis. Patlern Recognit. CVPR, pp. 2261-2269, 2017. doi: 10/gfhw3n.
Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challeninng Improves Cross-Domain Generalization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020. pp. 124 140, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58556-5.
Geewook Kim, Akifumi Okuno, Kazuki Fukui, and Hidetoshi Shimodaira. Representation learning with weighted inner product for universal approximation of general similarities. In Proceedings of the trrentyEiighth International Joint Conference Artificial Intelligence, IJCAI-19 19, pp. 5031- 5038. International Joint Conferences on Artificial Intelligence Organization, July 2019. doi: 10/gigdb3.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1865, International Convention Centre, Sydney, Australia, 2017. PMLR
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A Bench- mark of in-the- Wild Distribution Shifts. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139. pp. 5637-5664, Proceedings of Machine Learning Research, 2021. PMLR.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, November 1998. ISSN 1558-2256. doi: 10/d89c25.
Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19, MIT Press, 2007.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Deeper, Broader and Artier Domain Generalization. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5543-5551, Venice, October 2017. IEEE. ISBN 978-1-5386-1032-, doi: 10/gfx4ss.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain Generalization with Adversarial Feature Learning. In 2018 IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5400-5409, Salt Lake City, UT, June 2018b. IEEE. ISBN 978- 1188--668--6420-9, doi: 10/ghv245.
Pan Li. Da Li, Wei Li. Shaogang Gong, Yanwei Fu, and Timothy M. Hospedales. A simple feature augmentation for domain generalization. In Proceedings the IEEE/CVF International Conference the
on Computer Vision (ICCV), pp. 8886-8895, October 2021.
W. Li, Z. Xu, D. Xu, D. Dai, and L. Van Gool. Domain Generalization and Adaptation Using Low Rank Exemplar SVMs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(5): 1114-1127, May 2018c. ISSN 1939-3539, doi: 10/gde517.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep Domain Generalization via Conditional Invariant Adversarial Networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 647-663, Cham, 2018d. Springer International Publishing. ISBN 978-3-030-01267-0.
13

---

Under review as a conference paper at ICLR 2023
Yiying Li, Yongxin Yang. Wei Zhou, and Timothy Hospedales. Srature-critie networks for het- erogeneous domain generalization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3915-3924. PMLR, June 2019.
Ming- Yu Liu and Oncel Tuzel. Coupled Generative Adversarial Networks. In D Lee, M Sugiyama, U Luxburg, I Guyon, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. 32nd Ini. Conf. Mach. Learn. ICML 2015, 1:97-105, 2015.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, Fesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
Pingchuan Ma, Tao Du, and Wojciech Matusik. Efficient continuous pareto exploration in multi-task learning. In Proceedings of the 37th International Conference on Machine Learning. JMLR.org. 2020.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation with multiple sources. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2009.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Multiple Source Adaptation and the Renyi Divergence. Proc. 25th Conf. Uncertain. Artif. Intell. UAI 2009, pp. 367-374, May 2012.
Joao Monteiro, Xavier Gibert, Jianqiao Feng, Vincent Dumoulin, and Dar-Shyang Lee. Domain Conditional Predictors for Domain Adaptation. In Luca Bertinetto, João F. Henriques, Samuel Albanie, Michela Paganini, and Gül Varol (eds.), NeurIPS 2020 Workshop on Pre-registration in Proceedings of Machine Learning Research, 2021.
Machine Learning, volume 148, pp. PMLR.
Saeid Motiaan, Marco Piccirilli, Donald Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In IEEE International Conference on Computer Vision (ICCV), 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain Generalization via Invariant Feature Representation. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning. pp. 18-18. PMLR, February 2013.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel Mean Embedding of Distributions: A Review and Beyond, volume 10. May 2016.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schölkopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 10(1-2):1-14:) 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
L. Niu, W. Li, D. Xu, and J. Cai. Visual Recognition by Learning From Web Data via Weakly Supervised Domain Generalization. IEEE Transactions on Neural Networks and Learning Systems, 1889 September 2017. ISSN 2162-2388. doi: 10/gbtxs5.
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Res., 37(23):331 1-3325, 1997. ISSN 0042-6989. doi: 10/dn4knd.
X. Peng and K. Saenko. Synthetic to Real Adaptation with Generative Correlation Alignment Networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1982-1991, March 2018. doi: 10/gige2k.
14

---

Under review as a conference paper at ICLR 2023
X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment Matching for Multi-Source Domain Adaptation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1405-1415, 2019. ISBN 2380-7504. doi: 10/ghfhmt.
E. Perez Alday, Gu, A. J. Shah, Robichaux, A. I. Wong, Liu, F. Liu, A. B. Rad, A. Elola, S. Seyedi, Q. Li, A. Sharma, G. D. Clifford, and M. A. Reyna. Classification of 12-lead ECGs: the PhysioNet/Computing
Cardiology Challenge 2020. Physiol Meas, 41(12):124003, 01 2021.
Erick A Perez Alday, Annie Gu, Amit J Shah, Chad Robichaux, An-Kwok Ian Wong, Chengyu Liu, Feifei Liu, Ali Bahrami Rad, Andoni Elola, Salman Seyedi, Qiao Li, Ashish Sharma, Gari D Clifford, and Matthew A Reyna. Classification of 12-lead ECGs: The PhysioNet/Computing in Cardiology Challenge 2020. Physiol. Meas., 41(12):124003, December 2020. ISSN 0967-3334, 1361-6579. doi: 10. 1088/ 11188/1361-666 6611166666b/966
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947-1012, 2016.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient Domain Generalization via Common-Specific Low-Rank Decomposition. In Hal Daumé III and Aarti Singh (eds.), Proceed- ings of the 37th International Conference on Machine Learning, volume 119, pp. 7728-7738. Proceedings of Machine Learning Research, 2020. PMLR.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Focus on the common good: Group distributional robustness follows, 2021. URL https: / /arxiv. wttp//iirrr////ooooo wttg/bs/////ooooooooot...tt..sst abss//100 2110. 02919.
J. Quinonero-Candela, M. Sugiyama, A. Schwaighofer, and N.D. Lawrence. Dataset Shift in Machine Learning. Neural Information Processing Series. MIT Press, 2022. ISBN 978-0-262-5458777.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally Robust Neu- ral Networks for Group Shifts: On the Importance of Regularization for Worst- Case Generalization. In International Conference on Learning Representations 2020, pp. 19, 2020.
Bernhard Schölkopf. Causality for Machine Learning. arXiv, pp. 1-20, 2019, ISSN 23318422.
Bernhard Schölkopf, Ralf Herbrich, and Alex J. Smola. A Generalized Representer Theorem. In G. Goos, J. Hartmanis, J. van Leeuwen, David Helmbold, and Bob Williamson (eds.), Computa- tional Learning Theory, volume 2111, pp. 416-426. Springer Berlin Heidelberg, Berlin, Heidelberg, doi: 10. (000077--50044481--222
2001. ISBN 978-3-540-42343-J 978-3-540-44581-4
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, (09(5):612-634 2021.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optmizatiin. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 525-5555550 536, Red Hook, NY, USA, 2018. Curran Associates Inc.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing Across Domains via Cross-Gradient Training. In International Conference on Learming Representations, pp. 12, 2018.
Yuge Shi, Jeffrey Seely. Philip S. Torr, N. Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient Matching for Domain Generalization. arXiv, 2021. doi: 2104.09937.
Alexander J. Smola, Arthur Gretton, Le Song, and Berhhard Schölkopf. A Hilbert space embedding for distributions. In Proceedings of the 18th International Conference on Algorithmic Learning Theory (ALT), pp. 13-31. Springer- Verlag, 2007.
15

---

Under review as a conference paper at ICLR 2023
Bharath Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Gert Lanckriet, and Bernhard Schölkopf. Injective Hilbert space embeddings of probability measures. In The 21st Anmual Conference on Learning Theory (COLT), pp. 111-122. Omnipress, 2008
Bharath K Sriperumbudur, Gretton Arthur, Fukumizu Kenji, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metrics on probability measures. J. Mach. Learn, Res., 11:1517-1561, 2010.
Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaplation. The MIT Press, March 2012. ISBN 978-0-262-01709-11
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. 30th AAAI Conf. Artif. Intell. AAAI 2016, (Figure 1):2058-2065, 2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt.
Measuring Robustness to Natural Distribution Shifts in Image Classification. Si2i::0007.006444 September 2020.
E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial Discriminative Domain Adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2962-299277 July 2017. ISBN 1063-6919. doi: 10/gf5br3.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep Domain Confusion: Maximizing for Domain Invariance. January 2014.
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
Tomas Vicar, Jakub Heje, Petra Novotna, Marina Ronzhina, and Oto Janousek. Ecg abnormalities recognition using convolutional network with global skip connections and custom loss function. In 2020 Computing in Cardiology, pp. 1 1-4, 2020. doi: 10.22489/Cin.22020.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. CoRR, abs/2103.03097, 2021. URL https: //arxiv. oogg/////aaoo hbs///b///0/ / 2103. 03097.
Pengtao Xie, B. Póczos, and E. Xing. craooorthogonality regularization in kernel methods. In UAI, 2017.
Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching using sparse coding for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1794-1801, 2009. doi: 10/d5279q.
Huaxiu Yao, Yu Wang, Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving Out-of-Distribttion Robustness via Selective Augmentation, June 2022.
Z. Yi, H. Zhang, P. Tan, and M. Gong. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2868-2876, 2017. ISBN 2380-7504. doi: 10/ggv876.
Kun Zhang, Mingming Gong, Petar Stojanov, Biwei Huang, QINGSONG LIU, and Clark Glymour. Domain adaptation as a problem inference on graphical models. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4965-4977
Curran Associates, Inc., 2020.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning adapt domain shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Adv. Neural Inf. Process. Syst. volume 34, pp. 23664-23677 Curran Associates, Inc., 2021.
Han Zhao, Shanghang Zhang, Guanhang Wu, José M F Moura, Joao P Costeira, and Geoffrey Gordon. Adversarial Multiple Source Domain Adaptation. In S Bengio, H Wallach, H Larochelle, K Grauman, N Sesa-Bianchi, and R Gamett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
16

---

Under review as a conference paper at ICLR 2023
S. Zhao, X. Yue, S. Zhang, B. Li, H. Zhao, B. Wu, R. Krishna, E. Gonzalez. A. L. Sangiovanni- Vincentelli, A. Seshia, and K. Keutzer. A Review of Single-Source Deep Unsupervised Visual Domain Adaptation. IEEE Transactions Neural Networks and Learning Systems, pp. 1-21, 2020. ISSN 2162-2388. doi: 10/ghtzh..
Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Deep Domain-Adversarial Image Generation for Domain Generalisation. AAAI, 34007)13025-13333 April 2020. doi: 10/gj4mr2.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain Generalization: A Survey, ArXiv210302503 Cs, March 2021a.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to Generate Novel Domains for Domain Generalization, March 2021b.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain Generalization with MixStyle. In International Conference on Learning Representations, pp. 15, 2021c
J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired Image-to-lmage Translation Using Cycle- Consistent Adversarial Networks. 2017 IEFE International Conference on Computer Vision (ICCV), pp. 2242-2251, 2017. ISBN 2380-7504. doi: 10/gfhw33.
F. Zhuang, Z. Qi, K. Duan, Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. A Comprehensive Survey on Transfer Learning. Proceedings of the IEEE, 109(1):33776, January 2021. ISSN 1558-2256. doi: 10/ghesqr.
17

---

Under review as a conference paper at ICLR 2023
Appendices
Table of Contents
A Proofs
19
A.1 Proof of Lemma 1.
19
A.2 Proof of Proposition 3.1
19
B Details on the Gated Domain Units
20
Real-world example: Visualizations
................................
20
B.2 Detailed View of the Regularization Term N2L5
................................
20
B.3 Visualization of DG Layer
22
C Experiments
22
C.I Digits Experiment
23
C.2 Ablation Study
26
ECG Experiment
28
C.4 WILDS Benchmarking Experiments
31
18

---

Under review as a conference paper at ICLR 2023
A PROOFS
A.1 PROOF OF LEMMA
Proof. The result holds trivially for K 1. For K 2 2 and by the LE.D assumption, write the risk functional
(xXXY) Pjx,))((X,y)) for some E AK. Then, [y. f())) di v)) a RS) where R2 F + R4 is the elementary risk arg min R(f) arg min FeT
K
for each f E as R(f) f(x, (P))) ΣS- 0j J C(v,S(a) dP (1, Σÿ= oj
functional associated with the elementary distribution P2 Y). Hence, the Bayes predictors satisfy
K Σ af) fEF
f E arg
(A.3)
Since the rhs of equation A.3 corresponds to the linear scalarization of a multi-objective function
Rx) (Ri...., ), its solution (i.e., stationary point) Pareto-optimal with respect to these objective functions (Ma et al., 2020, Definition 3.1); see, also, (Hillermeier, 2001a;b). That is, the Bayes predictors for the data distribution that satisfies the L.E.D assumption must belong to the Pareto set
F == [ f = arg min E5= a E A C.F
A.2 PROOF OF PROPOSITION 3.1
Proof. Suppose we have a representation.
M
PP
ByNY,
(nv, = 0'0 #
(A.1)
i.e. (,v,...,,,y... are pairwise orthogonal. We want to minimize the MMD by minimizing
1u Σ 1=1
Byuv, 11 (up, UPYH
22
3juv)N
Bun
Byuv, nH
(A.2)
j=1
5=1
===
M
M
M
-I1up|1 2 Σ
(MP, MV,, +
B,B3
(uvg, MN, Big Auyluvg yro
(A.3)
j=1
M
=11ue|l
>>
By (pp, uv,)
83 llv, 11
(A.4)
5=1
j=1
By defining
M
((3) = ||pp-
Σ
B By 11
(A.5)
we can simply find the optimal
Bg by using the partial derivative
0Φ
22pe,, Pvy + 2,3 Mlv,, 11ke
(A.4)
03,
+ 3,lllv, 11kc = (HeP MN;
(A.5)
(VE,NV,) + 37 Ilvv 12
(A.6)
Please note that the function + is convex.
n
19

---

Under review as a conference paper at ICLR 2023
Task: Forecast glucose values from past observations
OOD generalization
CH
AT
"right" invariance!
US
Motivation
DE
CH
AT
Problems:
Elemantary distributions
strecture
Domsins ane oetinecitoo too broad (countree vs supporuter.on with countries)
Figure 3: Visualization and motivation of our invariant elementary distribution assumption and how they can be instantiated with Gated Domain Units.
B DETAILS ON THE GATED DOMAIN UNITS
B.I REAL-WORLD EXAMPLE: VISUALIZATIONS
As written in Section 2.1, we postulate that the elementary domain bases are the invariant subspaces that allow us to generalize to unseen domains. In practice, the question arises if and when elementary domains evolve. Consider that we aim to learn predict the risk of developing Diabetes from laboratory data from Europe and then infer the risk from data from the United States of America. Naturally, factors influencing the data-generating process may change, such as the level of physical activity and nutritional habits. While, to certain degree, these common factors remain invariant across continents, each of these factors' contributions may differ. In terms our assumptions, we model each of these factors with corresponding elementary distribution. Figure 3 depicts assumption and how differs from existing works".
To exploit this assumption in out-of- distribution (OOD) generalization, developed modular neural network layer that consists of so-called Gated Domain Units (GDUs). In Figure 4, we visualized the fundamental concept of the GDUs. Each GDU learns embedding of individual elementary domain that allows us to encode the domain similarities during the training. During inference, the GDUs compute similarities between observation and each of the corresponding elementary distributions, which are then used to form a weighted ensemble of learning machines. In other words, for previously unseen individual, we aim to determine the coefficients and quantify each factor's contribution without any information about the individual's 's origin.
B.2 DETAILED VIEW OF THE REGULARIZATION TERM DQLS
First, consider the following single term (lØ().) AM BosHy, 11 that can be expressed
M
M By
(lo(0:)
Biy 11 = [10(x))11 -2 (6(a.).
Σ
ByyMY +
ll
(B.1)
1=1
j=1
j=1
(2)
(3)
AD (1):
We begin with Term (1) and write [o as -o evaluate this term using the kernel function & for each data point in the batch b. However, since this
(Ó(ā.), o(ā))( k(ū;, We could
of note, Figure 3 is complete fictive example, and we do not want to make medical implications in any way.
20

---

Under review as a conference paper at ICLR 2023
During Training
During Inference
nottorr the set unzosenedd elmmentoyy diarbissionss that recses conanant
ue cmaate weighte ensemble usng acoteriuu dstmoun smsssnn observatón wiemertory
Oil
Gated
Figure 4: Visualization of the concept of the Gated Domain Unit and how they are leveraged to build distributionally weighted ensembles of learning machines.
term does not depend the the elementary domains .V...... VM is unnecessary to compute this value to minimize the penalty. Thus, we obtain similar result by minimizing the penalty without considering (o(;;))l lI in the regularization.
AD (2):
Term (2) can be expressed as
M
M
(ő(T;), Σ Biguny H Σ
Big (d(T), Nvz)
(B.2)
Implemeenattion-wiss, the evaluation of this term requires the calculation of the inner product (o(z,), Mv,)H. Since our CS and projection-baset methods involve this inner product to deter- mine the coefficients Biy, we pre-compute the inner product (o(#;), (vy H once for a mini-batch and store these information during training to avoid multiple calculations of the same term.
Moreover, the projection- based method does not apply softmax and has linear form. Therefore, the term (2) can be simplified even further:
(p(T)), Beguer H
Bey (ob(d(()))) Mv3))
(B.3)
M
(d(T:), mV3)
(0(())) HV)
(B.4)
luv; 1l
M
(d(T:)) pV,
(B.5)
Hvv l
AD (3):
Last, we express the term (3) as follows
M
M
M
By 11
Σ
Buy uve
(B.6)
j=1
21

---

Under review as a conference paper at ICLR 2023
and calculate the inner product of the domains (NVg, me by
1
(uv, (vv,
(o(x)) ((1)))
(B.7)
N
N
Σ
(uu vk) = Kgkn
(B.8)
N2
m=
where represents the number of vectors per domain basis. Note that this term does not depend on the input data 2; and, hence, matrix Kjs can be calculated once at the beginning of the optimization step and stored to re-used for all the data point of a batch.
Combining Equation B.6 and Equation B.8 yields
M
M
M
Bey PV3 11
=
Big Bix (uv,, HVL)H
(B.9)
==1
1=1 Am
M
M
N
N
Big Pik
Σ
(B.10)
j=1 k=1
1=1
m=11
Beg Ki
(B.11)
8f KinBi.
(B.
As final step, we use the results for Term (1), (2), and (3) obtain the desired regularization term
2ors
(1((z.)
- Σ
B 11
(B.13)
i=1
1=1
b
M
M Bo + Σ Bopyy 11 ==1
(1o(E))1 - 2(0(z)),
Σ
(B.14)
As mentioned above, [ is independent from the elementary domains, and, thus a constant in the regularization. Hence, we can exclude this term, which avoids additional computational effort.
B.3 VISUALIZATION OF DG LAYER
Figure depicts the layout of our DG layer.
C EXPERIMENTS
In this section, we provide a detailed description of the DG experiment presented in Section 5. Our Digits and ECG experiments are implemented using TensorFlow 2.4.1 and TensorFlow Probability 0.12.1. For the WILDS benchmarking we use our PyToorch (version 1.11.0). All source code will be made available on GitHub httpps://g thub. com/ (TensorFlow) and https: / thub. com/ (PyTorch). Overall, our experiments aim to show the validity of the invariant elementary distribution (L.E.D.) assumption and the Gated Domain Units (GDUs).
For the DG layer, we considered two modes of model training: fine tuning (FT) and end-to-end training (E2E). In FT scenario, we first pre-train the FE in the ERM single fashion. Then, we extract features using the pre-trained model and pass them to the DG layer for training the latter. For the E2E training, however, the whole model including the FE and DG layer is trained jointly from the very beginning.
22

---

Under review as a conference paper at ICLR 2023
Hy spantpy / 1.11... 1...MM)
MV
BiWW + b))
M13
Ba ba)
he
Σ
2(5.y)
+
PV5
( W NX bar)
(x, (,.))
(1x)
(6(x), AVA))
Vus
[(Var)
Figure 5: Domain generalization (DG) layer.
C.I DIGITS EXPERIMENT
Our experiment setup is closely related Peng et al. (2019); Feng et al. (2020); Zhang al. (2020); Zhao et al. (2018). Each dataset, except USPS, is split into training and test sets of 25,000 and 9,000 images, respectively. For USPS, we take whole dataset for the experiment since contains only 9,298 images". Our experimental setup regarding datasets, data loader, and FE are based on existing work (Feng et al., 2020; Peng et al., 2019). The structure of the FE summarized in Table and subsequent learning machine is dense layer.
In the Empirical Risk Minimization (ERM) single experiment, we add a dense layer with 10 outputs (activation=tanh))
classifier to the FE. In the Empirical Risk Minimization (ERM) ensemble experiment, we add M classification heads (a dense layers with 10 outputs and tanh activation each) to the FE and average their output for the final prediction. This sets a baseline for our DG layer to show performance gain against the ERM model with the same number of learning machines.
For training, we resorted the Adam optimizer with learning rate of 0.001. We used early stopping and selected the best model weights according to the validation accuracy. For the validation data, we used the combined test splits only of the respective source datasets. The batch size was set to 512. Although the DG layer requires more computation resources than the ERM models, all digits experiments were conducted on a single GPU (NVIDIA GeForce RTX 3090).
Heuristics for main parameter of DG layer From a practical perspective, our layer requires choosing two main syper-parameters. the number of elementary domains M and since we use the characteristics Gaussian kernel the corresponding parameter The parameter M determines the size of the ensemble of learning machines and, thus, for deep learning models, their overall network size. As heuristic choose M, suggest cluster the output of pre-trained FE. In the following, we provide an example. We pre-trained the FE for the test domain MNIST-M and pass the source data through this FE, which we cluster with the k-means algorithm. Subsequently. we analyse three different metrics (Calinski Harabasz score, Davies Bouldinn score, and Silhouette score) to select the optimal number of clusters as the basis to choose M. All scores yielded an accordance between four to five clusters. Therefore, we set M to five and observed in Table 2 in Section 5 strong results in the generalizing to the unseen test domain MNIST-M.
We used the digits data from https: / //gi thub. com/ Fenggz/ KD3A [last accessed on 2022-05-17, available under MIT License.] published in Feng et al. (2020).
23

---

Under review as a conference paper at ICLR 2023
Table Feature Extractor used for the Digits Experiment
PEATURE EXTRACTOR
LAYER TYPE
OUTPUT SHAPE
ZD-CONVOLUTIONAL LAYER
(32, 32, 64)
BATCH NORMALIZATION
(32, 32, 64)
MAXPOOLING 2D
(16, 16, 64)
2D-CONVOLUTIONAA LAYER
(16, 16, 64)
BATCH NORMALIZATION
(16, 16, 64)
MAXPOOLING 2D
88, 64)
CONVOLUTIONAL LAYER
(8.8, 128)
BATCH NORMALIZATION
(8, 8, 128)
MAXPOOLING 2D FLATTEN
(4,4, 128)
(2048)
DENSE LAYER
(3072)
BATCH NORMALIZATION
(3072)
DROPOUT
(3072)
BATCH NORMALIZATION
(2048)
DENSE LAYER
(2048)
As for the parameter dr resort the median heuristic proposed in (Muandet et al., 2016) that is i, = =1, ,nt.. While both heuristics require a pre-trained act reasonable alternative. The hyper-parameters relevant for the DG
g² = median || ã; 3j 1|a FE, cross-validation layer are summarized in Table 6. In the FT setting. we applied the median heuristics presented above estimate of the Gaussian kernel function, where the estimator denoted as ô. Since median heuristic is not applicable for the E2E scenario, a was fixed to 7.5 for E2E.
Note that our approach choose the relevant parameters was kept very general to show the feasibility of the I.E.D. assumption and the generalization ability of GDUs and, most importantly, to provide mesy-to-reproduee results. During training, additional epoch metrics can be subscribed using our custom DG layer callbuck, which may help to choose the model parameters. Furthermore, we observed that the elementary domains become naturally orthogonal during the experiments, and thus, set λORTH relatively small. Since the orthogonal regularization puts additional computational burden, one could omit this term completely to speed up training.
Table 6: Parameters for DG Layer in Digits and Digit-DO Experiments for the Fine Tuning (FT) and End-to-end training (E2E) Settings. In case of Projection, we chose the spectral restricted isometry property (SRIP) as the orthogonal regularization ΩD
EXPERIMENT
M
N
dL
AOLS
AORTH
d
K
CS
5
10
Ie-3
1e-3
à
2
FT
MMD
5
10
le
le
à
2
PROJECTION
5
10
le 3
le
le
à
CS
5
10
16-3
le-3
7.5
2
E2F
MMD
5
10
le
le
7.5
2
PROJECTION
5
10
le
le
1e
7.5
Digit-DG Benchmark In previous research, the aforementioned digits data is not only used for domain adaptation (DA), but also for domain generalization (DG) methods. For the latter, Zhou et al. (2021b) and Li et al. (2021) introduced Digit-DG dataset and the evaluation protocol to benchmark seven DG methods and ERM 4. Unlike the Digits experiment described above, Digit-DG dataset from Zhou et al. (2021b) and Li et al. (2021) consists of only four datasets (without USPS) and a different FE summarized in Table 8. Therefore, we follow their instructions conduct fair comparison and ensure reprodueibility. For the hyper-parameters however, we kept the same values that we used for the Digits experiment, see Table 6.
'Results were reported by Zhou al. (2021h) and Li et al. (2021). Of note, both authors did not report the standard deviation on their results.
24

---

Under review as a conference paper at ICLR 2023
Table Results Digits experiment. All experiments were repeated ten times and the mean (standard deviation) accuracy is reported. Best results according to the mean accuracy are highlighted in bold.
MNIST
MNIST-M
SVHN
USPS
SYN
Single
97.98 (0.34)
(3.00)
70.18 (2.74)
93.70 93.70
83.62 (1.47)
Ensemble
98:21 (0.39)
(1.50)
72.01 (3.59)
95.16 (0.89)
(1.80
CS
98,33 (0.16)
68.55 (0,80)
78.90 (1,41)
95.83 (0.50)
88.39 (0,82)
FT
MMD
98,60 (0.08)
68.62 (0.70)
(2.01)
96.24 (0.71)
88.27 (0.41)
PROJECTION
98.57 (0.17)
68.56 (0.91)
79.34 (0.72)
96.24 (0.71)
88.58 (0.53)
CS
98.62 (0.19)
69.25 (0.61)
79.42 (1.27)
96.17 (0.52)
87,92 (0.84)
MMD
98.58 (0.16)
69.00 (0.83)
79.20 (0.90)
96.00 (0.44)
18 (0.86)
PROJECTION
98.67 (0.12)
68.67 (0.98)
78.56 (1.68)
96.24 (0.77)
88.77 (0.48)
Table 8: Feature Extractor used for the Digit-DG Benchmark Experiment
FEATURE EXTRACTOR
LAYER TYPE
OUTPUT SHAPE
2D- D2--CONVOLUTIONES LAYER
(32, 32, 64)
MAXPOOLING 2D
(16, 16, 64)
2D AD-CONVOLUTIONAL LAYER
(16. 16, 64)
MAXPOOLING 2D
8. 8. 64)
CONVOLUTIOOAAL LAYER
(8, 64)
MAXPOOLING 2D
(4, 128)
2D- CONVOLLUIIOO LAYER
(8, 8, 64)
MAXPOOLING 2D
(4, 4, 128)
RD-CONYOLUTIONAL LAYER
(4, 4, 64)
MAXPOOLING 2D FLATTEN
(2, 64)
(256)
As a first method, we consider the CCSA (Classification and Contrastive Semantic Alignment) method, which learns a domain-invariant representation by utilizing the CCSA loss (Motian et al., 2017). Second, MMD-AAE (Maximum Mean Diserepancy- Adverserial Autoencoders) extends adverserial autoencoders by a maximum mean discrepancy regularization to learn a domainninvariant invariant feature representation (Li et al., 2018b). CrossGrad (Cross-Gradient) augments data by perturbating the input space using the cross-gradients of a label and domain predictor (Shankar et al., 2018) 8).. Another augmentation-basee DG method is L2A-OT (Learning Augment by Optimal Transport) (Zhou et al., 2021b). Specifically, data generator trained to maximize the optimal transport distance between source and pseudo domains, is used to augment the source data. All aforementioned methods rely on the availability of domain information such as domain labels. To benchmark our layer to a method for DG without domain information, we resort to the JiGen (Jigsaw puzzle based Generalization) method (Carlucci et al., 2019). JiGen introduces an auxiliary loss for solving jigsaw task during training. Further, we use the adaptive and non-adaptive stochastie feature augmentation (SFA-S and SFA-A, respectively) method proposed by Li et al. (2021). In principle, both method augment the latent feature embedding of a FE using random noise.
Our results summarized in Table 9, As noted by Li et al. (2021), it is challenging to outperform augmentation- based DG methods. addition, SFA-A and SF-S are computationally light (i.e., only adding random noise to the feature embedding) and do not require domain information (Li et al., 2021). Nevertheless, our layer achieves competitive results even against the strongest baselines in all DG tasks without requiring domain information.
25

---

Under review as a conference paper at ICLR 2023
Table Results of the Digits-DG experiment. All experiments were repeated ten times. Methods are classified into ougmentation-basel (A) and nnn-augmeetation-baeeeee (B) as well as DG with (v) and without (X) domain information according Li al. (2021). Best results according to the mean accuracy are highlighted in bold.
MNIST MNIST-M
SVHN
CATEGORIE
Resultx reported by Lá al. (2021)
ERM
95.8
58,8
61.7
78.6
B
CCSA (MOTIIAN ET Al., 2017)
95.2
58.2
65.55
79.1
B
MMD-AAE (LLET AL., 20188)
96.5
58.4
65.0
78.4
B
CroszGrad (SHANKAR ET AL., 2018)
96.7
61.1
63.3
80.2
(ZHOU Al.,
96.7
SPA-S ET Al., 2021)
96.7
66.3
68.8
85.1
SEA-A (LI ET AL... 2021)
96.5
66.5
70.3
85.0
A
JiGnn (CARLUCCI AL., 2019)
96.5
61.4
63.7
74.0
B
Gated Domain Units (GDUs)
CS
74.0
FT
MMD PROJECTION
97.6
69.0
74.2
84.3
B
X
97.7
69.1
74.1
86.1
B
CS PROJECTION
97.6
69.4
75.9
86.5
B
X
97.8
66.7
73.4
84.0
B
C.2 ABLATION STUDY
C.2.1 MAIN COMPONENTS OF THE GATED DOMAIN UNIT
We chose the Digits dataset conduct ablation study, which is organized as follows: (1) ablation of the regularization terms presented in Section 3, (2) effect of the orthogonal regularization for projection-based generalization, and (3) affect on the FE's output.
As a reminder, we introduced the regularization to be dependent on the form of generalization (i.e.. domain similarity measures or projection- generalization in Section 3). For the domain similarity measure case, the regularization is
((llll)) AoLs (19ll)) + AΣ, 20 (lhrl),
(C.1)
where AOLS, Xz, 2 0. In the case of projection, the regularization is given by
(Iool)) Aous (IIal))) AORTHRD (Illlla)))))
(C.2)
with ÀOLS, λORTH 0. Although one additionally choose the sparse regularization in projection- based generalization. we set the focus in the ablation study on the two main regularization terms that are the OLS and orthogonal regularization. For (1) vary in Equation C.1 and Equation C.2 the corresponding weights A and Aa in the interval of [0; 0.1] 1] and display the mean classification accuracy for the most challenging classification task of MNSIT-M in the form of heatmap. In Figures 6-8, we see that the classification accuracy remains on an overall similar level which indicates that the DG layer is not very sensitive the hyper-parameter change for MNIST-M as the test domain. Nevertheless, we observe that ablating the regularization terms by setting the corresponding weights zero decreases the classification results and the peaks in performance occur when the regularization is included during training of the DG layer.
Applying the DG layer comes with additional overhead, especially the regularization that ensures the orthogonality of the elementary domain bases. This additional effort raises question whether ensuring the theoretical assumptions outweigh the much higher computational effort. Thus, second step, analyze how the orthogonal regularization affects the orthogonality of the elementary domain bases (i.e., spectral restricted isometry property (SRIP) value) and the loss function (i.c., categorical cross-entropy).
In Figure 10, we depict the mean and standard deviation of the SRIP value and loss over five runs for 40 epochs. The SRIP value can be tracked during training with the DG layer's callback functionalities. First, we observe that the elementary domains are almost orthogonal when initialized. Training the layer leads in the first epochs decrease in orthogonality. This initial decrease happens because
26

---

Under review as a conference paper at ICLR 2023
70.
69
0.01
0.03
97.02
Adks
68.0
0.003
68.61
66.
0,003
69.
69.34
20.10
07.5
67.
69.13
58:40
70.19
68.30
90.43
69.47
68. 68.66
67.19
66.5
001
0.00
0.1
0.001
0.01
0.1
A1
Figure 6: Classification results for varying AL, and AOLS in the interval of [o; 0.1] for FT (left) and E2E (right) CS on MNIST-M.
29.75
89.
50.30
09.25
0.01
68.55
68.88
70.18
68.31
69.0
0.0.
57.94
69 54
60.82
68.26
09 00
Apis
00.5
0.001
69.44
57.32
68.44
68.16
00022
88.62
69 71
60.16
12
62.
60.25
6/.59
58. 36
67.30
b7:14
06.
65 71
68.23
1s C
00.00
0,001
0.01
0.001
0.01
0.1
A,3
Asa
Figure 7: Classification results for varying dL, and AOLS in the interval of [0; 0.1] for FT (left) and E2E (right) MMD on MNIST-M.
cross-entropy has stronger influence on the optimization than regularization in the first epochs. After five epochs, the cross-entropy decrease to threshold when the regularization becomes more effective and the orthogonality of the elementary domain bases increases again. In Figure 10, we also observe that ablating the orthogonal regularization, while leading to better orthogonality of the domains, does not significantly affect the overall cross-cntropy during training.
Finally, we project the output of the FE trained with dense layer (ERM) and with the DG layer by t-SNE (I-distributed stochastic neighbor embedding) in Figure 11. The GDU-trained FE yields more concentrated and bounded clusters in comparison to the trained by ERM. Hence, observe positive effect on the representation learned by the FE.
C.2.2 INTERPRETATION OF THE ELEMENTARY DOMAINS
We analyze the learned elementary domains the digits experiment based on two visualizations, and choose the maximum mean discrepancy (MMD) as the similarity measure and MNIST-M the test domain. The first visualization depicts the MMD between the datasets (i.e., MNIST, MNIST-M, SVHN, USPS, and Synthetic Digits (SYN)) and the learned elementary domains (i.e., heatmap (see Figure 12 (left)). The heatmap indicates that the source and test domains are close to one another in terms of the MMD. Hence, we expect that their closeness reflects in the learning the elementary domains. In other words, we expect that each elementary domains contributes similarly to the source and test domains (i.e., the coefficients / are similar for each these domains). In Section 3.1, we derive the coefficients by applying kernel softmax function to the negative MMD distances. Since the MMD distances between the source test domains and the elementary domains are similar, the coefficients will be similar too. We conclude that the learned elementary domains represent the same distributional characteristics that existed among the source and test domains.
V/3)
In the second visualisation, show the t-SNE (t-distributed stochastic neighbor embedding) of the feature extractor output for each source and test domain alongside the elementary domains in Figure 12 (right). First, observe that the learned elementary domain bases form distinctive clusters. We
27

---

Under review as a conference paper at ICLR 2023
7012
06. 09
68.35
05.05
08:27
69.5
0.01
0.1
69.0
Adks
68.3
0.003
66.58
89.20
0.002
66.0
68.23
89.80
69.14
69:02
07.0
70.02
67.38
68.52
68.32
0.001
0.00
0.1
0.001
0.01
0.1
Ausio
Figure 8: Classification results for varying λORTH and ÀOLS in the interval of [0; 0.1] for FT (left) and E2E (right) Projection on MNIST-M.
Projection
0.72
6.47
0.09
30
6.71
0.09
0.00
30
20
30
20
10
Figure 9: Mean and standard deviation of classification accuracy over 10 runs for varying number of elementary domains (M, upper panel) and varying number of vector for each domain basis (NV lower panel) for MNIST-M dataset.
see these clusters as a validation our hypothesis that each GDU learns to mimic samples generated from corresponding elementary distribution pointed out in Section 2.2. However, we can not answer whether and where these elementary distributions occur in the real world. Moreover, these elementary distributions yet lack interpretability.
In summary, the MMD heatmap and t-SNE embeddings the learned elementary and source domains on Figure 12 indicate that the GDUs learn represent distributional structures in the dataset.
C.3 ECG EXPERIMENT
adopted the task multi-label binary classification 12-lead electrocardiogram (ECG) signals combined from 6 different sources introduced in the PlystoNettCompputig in Cardiology Challenge 2020 (Perez Alday et al., 2021; Goldberger et al., 2000; Perez Alday et al., 2020). Each ECG recordings is annotated with 24 binary labels indicating whether not certain cardiac abnormality present. The data is aggregated from different databases and contains 43,10 101 recordings sampled
Shttps: /tt::://hhhy onet.chal lenges wrg. /220//20////// [last accessed on 2021-03-10, available under Cre- ative Commons Attribution 4.0 International Public License].
28

---

Under review as a conference paper at ICLR 2023
Labe!
1.0
033
0.5
FTaggeed::
Filgosdd
ITolers
005
897
0.4
Epochs
Epochs
Figure 10: Effect of omitting the orthogonal regularization term 22 Spectral restricted isometry property (SRIP) (left) and categorical cross-entropy (right) with and without orthogonal regularization and their evolution during training for MNSIT-M dataset. The mean and standard deviation presented for End-to-end (E2E) and Fine-tuning (FT) training scenarios are calculated over 10 runs.
ERM
MMD
Projection
Figure 11: Visualization of t-SNE Embedding on unseen Synthetic Digits Dataset. Colors encode true label.
with various sampling frequencies, number of subjects, and lengths. Table 10 summarizes most important details about the data sources for this experiment.
According to the original challenge score, we measure the performance in terms of the generalized Intersection-over- Unioon (IoU) score where partial credit is assigned to misdiagnoses that result in similar treatments or outcomes. The score defined as
y² W
score
(C.3)
y U
where y, E {0, 1 represent actual labels and predicted labels and W stands for the partial credit-assignment matrix provided as a part of the challenge description. Note that in case of identity matrix W the score is exactly the Intersection-over Union (IoU) score. The score is then adjusted for a solution ymajoritys which always predicts the normal/maaority class, and is moreover normalized
HNDD Bestrop Ser 990
"NE endiniiing Fr: NVD
200
+
UPPE ON
Figure 12: MMD heatmap (left) and t-SNE embedding (right) for the test domain MNIST-M.
29

---

Under review as a conference paper at ICLR 2023
Table 10: ECG Data Sources Details.
DATASET
NUMBER
ECG LENGTH [SFC]
FREQUENCY
COUNTRY
CPSC
6,877
CHINA
CPSC-EXTR
3,453
TO 60
500 Hz.
CHINA
INCART
1,800
257 Hz
RUSSIA
PTB
549
UP TO 120
1.000 Hz
GERMANY
PTB-XL
21.837
500 11Z
GERMANY
G12EC
10,344
500 HZ
USA
for the perfect solution y. Therefore, the final score can have negative values and the best possible score of and is formalized as
scorely, ú) score(y: Ymajority) score(y, scorely, Umajority)
adjusted score
(C.4)
As pre-processing step, we down-sampled all the signals 125 Hz and applied Z-score, random amplification and random stretching according to Vicar et al. (2020). For that we partially adopted the code provided by the authors Additionally, we cropped each signal to its first 15,000 points if the signal was too long (mostly applied to INCART database). Each dataset was randomly split into train and validation parts with 3:1 ratio. During each experiment, we used the train splits of databases for training and utilized the validation splits of the training databases for early stopping. The hold-out 6-th database was used for inference and testing only.
Table 11 describes the architecture of FE used for the task. Since the provided ECG recordings have different lengths, we used TensorFlow padded batching, which is padding all the recordings a batch to the length of the longest sequence in the batch. Therefore, input from different batches can have different lengths so the spatial dimensions of the ID-Convolutional layers are not predefined and are presented as *
Table 11: Feature Extraction Architecture used for the ECG Experiment is an adapted Version of LeNet Architecture for ID input Signals. Note that ECG recordings have variable lengths, therefore, the spatial dimension not defined and denoted
PEATURE EXTRACTOR
LAYER TYPE
OUTPUT SHAPE
ID-CONVOLUTIONAL LAYER (KURNEL SIZE=], ACTIVATION==EL
(7,
BATCH NORMALIZATION
(*, 32)
MAXPOOLING ID (POOL SIZE=2, STRIDES=2)
(%, 32)
ID-CONVOI UTIONAL LAYER (KERNEL, S17R22, ACTIVATION
(*,
BATCH NORMALIZATION
(%, 64)
GLOBAL AVERAGE POOLING
(64)
DENSE LAYER (ACTIVATIO=<eell)
(100)
DENSE LAYER (ACTVAATIO==eea))
(100)
We used the Adam optimizer to optimize weighted binary cross-entropy loss defined - ( log (1 ÿ). Positive weights wpss are defined per class based on the training split data inversely proportional
y log ú) + (1 11) y)
the frequency of positive labels for each class. A learning rate was initially set to 0.001 and during the training reduced by the factor of 0.2 if the training loss was improving for 10 epochs. We also applied early stopping and restored model weights the best model according to the validation accuracy after the training end. Since each input samples for this experiment have larger size than the previous one, we decreased the batch size 64. Each ECG experiment was performed on single GPU (Nvidia GTX 1080 Ti). The parameters relevant for the DG layer are summarized Table 12. We have to emphasize that we did not perform extensive hyper-parameter tuning since our goal was to show the fcasibility of the I.E.D. assumption and GDUs while keeping the experiments reproducible.
https: / / thub. www.//tomasvicar//uutss [last accessed on 2022-05-17, available under BSD 2-Clause License].
30

---

Under review as a conference paper at ICLR 2023
Table 12: Parameters for DG Layer in ECG experiments for the Fine Tuning (FT) and End-to-end training (E2E) Settings.
EXPERIMENT
M
d
AOLS
AORTH
R
CS
10
10
le
le-3
5.5
FT
MMD
10
10
le
1e-3
5,5
PROJECTION
10
10
le
le
3
le"
5,5
CS
10
le
le
5.5
E2E
MMD
10
10
le
le-s
5.5
PROJECTION
10
10
1e
le-3
1e-6
5.5
C.4 WILDS BENCHMARKING EXPERIMENTS
For comparison of our approach and benchmarking, we followed the standard procedure of WILDS experiments, described in Koh et al. (2021). As a technical note, all WILDS experiments have been implemented in Pytorch (version >= 1.7.0) based the specifications made in Koh al. (2021) and their code published on https: //gitthub com coomp-llalee http/pplambba//// /wilds [last accessed on 2022-05-17, available under MIT License]. The results for the benchmarks were retrieved from the official leaderboard https:/ / /wi lds. stan eeeeeeedeeeeeeeeeeeeeeeeeeeeeeee leaderboard/ [last accessed on 2022-09-26).
Camelyon17 In medical applications, the goal is to apply models trained on comparatively small set of hospitals to larger number of hospitals. For this application, we study images of tissue slides under microscope to determine whether a patient has cancer or not. Shifts in patient populations, slide staining, and image acquisition can impede model accuracy in previously unseen hospitals. Camelyon 17 comprises images of tissue patches from five different hospitals. While the first three hospitals are the source domains 2302,436, examples), the forth and fifth are the validation (34,904 examples) and test domain (85,054 examples), respectively.
We deviate from the specifications made in (Koh et al., 2021) in terms of the FE. We use the FE from Feng et al. (2020); Peng et al. (2019) since we observed a higher mean accuracy and faster training than with the by Koh et al. (2021) originally proposed DenseNet-122 121 FE (Huang et al., 2017). We trained the FE from scratch. Both, ERM and the DG were trained over 250 epochs with early stopping, learning rate of 0.001, which is reduced by factor of 0.2 if the cross-entropy loss has not improved after 10 epochs. All results were aggregated over ten runs.
FMoW Analyzing satellite images with machine learning (ML) models may enable novel possibili- ties in tackling global sustainability and economic challenges such population density mapping and deforestation tracking. However, satellite imagery changes over time due to human behavior (e.g., infrastructure development), and the extent of change is different in each region. The Functional Map of the World (FMoW) dataset consists of satellite images from different continents and years: training (76,863 images; between 2002-2013), validation (19,915 images; between 2013 and 2016), and test (22, 108 images, between 2016-2017). The objective is to determine one of 62 building types (e.g., shopping malls) and land-use.
As instructed in Koh et al. (2021), we used the DenseNet- 121 pre-trained on ImageNet without L2-regularization. For the optimization, we use the Adam optimizer with learning rate of Ie-4, which is decayed by factor of 0.96 per epoch. The models were trained for 50 epochs with early stopping and a batch size of 64. Additionally, we report the worst-region accuracy, which is specific metric used for FMoW. This worst-region accuracy reports the worst accuracy across the following regions: Asia, Europe, Africa, America, and Oceania (see Koh et al. (2021) for the details). Again, report the results over three runs.
Amazon Recent research shows that consumer-facing machine learning application large perfor- mance disparities across different set of users. To study this performance disparities, WILDS (Koh et al., 2021) leverages variant of the Amazon Review dataset. The Aamazon WIILDS dataset is composed of data from 3,920 domains (number reviewers) and the task is a multi-class sentiment classification, where the model receives review text and has predict the rating from one to five.
31

---

Under review as a conference paper at ICLR 2023
To split this dataset, between training, validation, and test disjoint set of reviewers is used: training (245,502 reviews from 1,252 reviewers), validation (100,050 reviews from 1,334 reviewers), test (100,050 reviews from 1,334 reviewers).
For the experiments and baseline models, we use the specifications made in Koh et al. (2021). As for the FE, used BattlBBBER-base-ccasse models. For ERM, we use batch size of 8, learning rate le-5, L.2 regularization of 0.01, 3 epochs with early stopping and 512 as the maximum length of tokens. For training the DG layer, we used the same specifications as made for ERM. The performance is measured in 10th percentile accuracy.
iWildsCam Wildlife camera traps offer an excellent possibility to understand and monitor biodiver- sity loss. However, images from different camera traps differ in illumination, color, camera angle, background, vegetation, and relative animal frequencies. We use the iWildsCam dataset consisting of 323 different camera traps positioned in different locations worldwide. In the dataset, we refer to different locations of camera traps as different domains, in particular 243 training traps (129,809 images), 32 validation traps (14,961 images), and 48 test traps (42,791 images). The objective is to classify one of 182 animal species.
Following the instructions by Koh et al. (2021), we used again the ResNet50 pre-trained on ImagNet (He et al., 2016). For ERM, used learning rate of 3e-5 and no L2-regularization. The models were trained for 12 epochs with a batch size of 16 with the Adam optimizer. In addition to the accuracy, report the macro Fi-score to evaluate the performance rare species (see Koh et al. (2021) for details). All results were aggregated over three runs.
RxRx1 In biomedical research areas such as genomics or drug discovery, high-throe throughutthh screening techniques generate vast amount of data in several batches. Because experimental designs cannot fully mitigate the effects of confounding variables like temperature, humidity, and measurements across batches, this creates heterogeneity the observed datasets (commonly known as batch effect). The RxRx1 dataset comprises images obtained by fluorescent microscopy from 51 domains (disjoint experiments): training (40,612 images, 33 domains), validation (9,854 images, domains), and test (34,432 images, 14 domains). The aim is to classify one of 1,139 genetic treatments. All results were aggregated over three runs.
We conducted the RxRx1 experiments in accordance with the specifications made in (Koh et al., 2021). As for the FE, we, thus, used the ResNer50 pre-trained on ImagNet (He et al., 2016). We trained the models using AdamW with default parameters 31 = 0.9 and 83 = 0.999 using learning rate of lc-4 and a L2-regularization with strength le-5 for 90 epochs with a batch size 75. We scheduled the learning rate to linearly increase in the first ten epochs and then decreased it following a cosine rate. For trainingthe DG layer, we chose the same parameters as for the ERM. All results were aggregated over three runs.
In biomedical research, machine learning has the potential to accelerate drug discovery while reducing the experimental overhead due to lowering the number of experiments required. However, leverage the potential of machine learning, the models need generalize to molecules structurally different from those seen during training. To study this OOD generalization across molecule scaffolds, we use the MGB-MolPCBA dataset. This dataset is split into the following subsets according to the scaffold structure: training (44,930 domains), validation (31,361 domains), and test (43,739 domains). The task is to classify the presence/abseene of 128 biological activities based on graph representation of a molecule.
OBG-MoIPCBA
In line with Koh et al. (2021), we use Graph Isomorphism Network (GIN) combined with virtual nodes as the FE. For training ERM and our DG, we use the default parameters: five GNN layers with a dimensionality of 300 and a learning rate of 0.001. We train for 100 epochs using early stopping. As for the performance, we report the mean and standard deviation of the average precision across all scaffolds (domains) over three runs.
CivilComments In the last decades, users have generated a vast amount of text on the Internet, some of which contain toxic comments. Machine learning has been leveraged for automatic text review to flag toxic comments. However, the models are prone to learn spurious correlations between toxicity and information on demographics in the comment, which causes the model performance to
32

---

Under review as a conference paper at ICLR 2023
drop in specific subpopulations. To study this OOD task, we leverage the modified CivilComment dataset from Koh et al. (2021). Based on text imput, the task is to predict a binary label, toxic or non-toxic. The domains are defined according to cight demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black, and White. All comments were randomly split into a disjoint training (269,038 comments), validation (45, 1551800 comments), and test (133,78) 782 comments) set.
model with the following parameters: batch size = 16, learning rate = le-5, AdamW optimizer, number of epochs = 5, L2 regularization 0.01, and the maximum number of tokens of 300. We use these default parameters for training our DG layer. The performance is measured in the worst-group accuracy and we report mean and standard deviation across five runs.
Again, we follow Koh al. (2021) and use DistillBERT-bas--uucaaa
PovertyMap As the FMoW example shows, satellite images in combination with machine learning models can been used monitor sustainability and economic challenges on global scale. Another application of these satellite images is poverty estimation across different spatial regions. However, there exists a lack of labels for developing countries since obtaining the ground truth is expensive, which makes this application attractive for machine learning models. To study the OOD generalization to unseen countries, we use a modified version of the poverty mapping dataset of WILDS (Koh et al., 2021). The task is to predict a real-valued aset wealth index between I and 5 based on multi-spectral satellite image. The domain refers to the country and whether the the the image is from a rural or urban are. In contrast to the other datasets, this dataset is split in five different folds, whereby in cach fold the the training, valdiation and test set contains a disjoint set of countries, however, data from both rural and urban regions. The avergae size of each set across the 5 folds is for the training 100,000 images (13- 14 countries), -4,000 images (4-5 different countries), and for the test set -4,000 images (13-14 countries).
On the challenge of obtaining domain labels. In the example of hospitals (e.g. Camelyon17 7 dataset), domain labels come, in fact, for free. However, other examples, such as the CivilComments dataset, show the opposite. This dataset requires additional annotations (i.e., demographic identities), which can tedious to obtain in practice. Some algorithms need these domain annotations to achieve superior performance on each subgroup. Furthermore, the task of subgroup detection in itself is difficult and relevant problem. Coming back to our hospital example, even people from the same hospital might belong to different subpopulation (e.g. gender, race, age) and these demographic subgroups are often more relevant for diagnosis than which hospital a patient comes from. This information, however, is not always available (due to anonymization standards, for instance) and, therefore, the relevant domain annotation might be hard to obtain.
We follow Koh al. (2021) and use pre-trained R8 model minimizing the sqarred error loss. For the optimization we rely on the Adam optimizer with the following parameters: learning rate of 1e-3 with a decay of 0.96 per epoch, batch size of 64 and early stopping based on the OOD evaluation score. For evaluation, we report the Pearson correlation (r) between the predicted and actual asset wealth indices across the five different folds.
General benchmark methods Following the WILDS benchmarking procedure (Koh et al., 2021), )), we compare our proposed DG layer the following baselines. First, empirical risk minimization (ERM), which minimizes the average training loss over the pooled dataset. Second, a group of DG algorithms provided by the WILDS benchmark, namely, Coral, Fish, IRM, and DRO. The Coral algorithm introduces penalty for differences in means and covariances of the domains feature distributions. The Fish algorithm achieves DG by approximating an inter-domain gradient matching objective, i.e., maximizing the inner product between gradients from different domains (Shi et al., 2021). Conceptually, Fish learns feature representations that are invariant across domains. Invariant risk minimization (IRM) introduces a penalty for feature distributions with different optimal classifiers for each domain (Arjovsky et al., 2019). The idea is to enable OOD generalization by learning domain-invariant causal predictors. Lastly, group distributionally robust optimization (DRO) explicitly minimizes the training loss on the worst-case domain (Sagawa et al., 2020; Hu et al., 2018).
In addition to the baselines originally presented in Koh al. (2021), consider the following more recent DG baselines. First, we describe LISA, which instead of regularizing the internal representations for generalization, seeks to learn domain-invariant predictors with selective data
33

---

Under review as a conference paper at ICLR 2023
Table 13: Parameters for DG Layer in WILDS experiments for the Fine Tuning (FT) and End-to-end training (E2E) Settings.
EXPERIMENT
N
AL
AoLS
AORTH
X
CS
10
le
le-3
2
FT AND E2E
MMD
10
le
le
4
2
BENCHMARK
PROJECTION
10
le
le
8
16
augmentation Yao et al. (2022). Common Gradient Descent (CGD), introduced by Piratla et al. (2021), based on Group-DRO. However, proposes to focus not on groups with the worst regularization but on common groups that enable generalization. Last, Adaptive Risk Minimization using batch normalization (ARM-BN) by Zhang et al. (2021) is different from the methods presented since adapts to previously unseen domains during test time using unlabeled observations from this test domain.
34

---

Under review as a conference paper at ICLR 2023
Table 14: Detailed results on RxRx1 dataset.
RXRX1
VAL
IID ACCURACY
OOD ACCURACY
ACCURACY
ERM
19.4 (0.2)
(0.4)
29.9 (0.4)
CORAL
(0.4)
(0.3)
28.4 (0.3)
FISH
IRM
(.640.4)
(19.4.4)
8.2 (1.1)
GROUP
(0.1)
LISA
20.1 (0.4)
41.1 (1.3)
(1.0)
ARM-BN
20.9)0..2
(0.2)
31.2 (0.1)
CS
(0.9)
(0.4)
29.7 (0.4)
FT
MMD
(0.2)
(0.2)
29.6
PRO
(2.5)
35.3(0.3)
(0.2)
(0.5)
(0.4)
E2E
MMD
19.5 (0.5)
36.2
29.9 (0.3)
(0.5)
(0.5)
(0.3)
Table 15: Detailed results FMoW dataset.
FMoW
VAL
TEST
REGION AcC
REGION ACC
ERM
$9.2 (0.07)
49.8 (0.36)
52.7 (0.23)
31.3 (0.17)
CORAL
32.8
FISH
(0.15)
49.5 (2.34)
51.8 (0.32)
34.6 (0.18)
IRM
(6.6()
49.7 (0.97)
(0.75)
32.8 (2.09)
GROUP DRO
(U.700
48.7 (0.92)
$2.8 (1.15)
31.1 (1.66)
LISA
58.7
48.7 (0.92)
52.8 (1.15)
35.5 (0.81)
CGD
(1.03)
49.8 (1.04)
50.6 (1.39)
32.0
ARM-BN
(0.66)
38.9 (2,17)
42.1 (0.26)
24.4 (0.54)
CS
(0.2()
50.4 (0.50)
53.1 (0.22)
31.8 (1.24)
FT
(U.24)
(1.17)
PRO
59.2
50.0 (0.76)
52.7 (0.14)
31.8 (1.08)
CS
59.3 (0.33)
52.1 (0.80)
$3.4 (0.25)
34.4 (1.86)
E2E
MMD
58.5 (0.4))
52.9 (1.67)
52.7 (0.45)
34.4 (0.71)
PRO
58.5 (0.43)
50.9 (0.85)
52.7 (0.68)
32.9 (0.78)
35


Reviews:
[{"review_id": "rJRRl7zd2c5", "reviewer": "Reviewer_13Xi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This contribution tackles the domain generalization problem by assuming that every real-world distribution is composed of a mixture of elementary distributions, which remain invariant across different domains. This assumption was named I.E.D. (Invariant Elementary Distribution). The authors presented a lemma to support the theoretical soundness of making such an assumption and then introduced a practical way to leverage the I.E.D. assumption when training neural networks to enable generalization on unseen domains containing the same elementary distributions. The proposed approach roughly consists of an ensemble of predictors weighted by the similarity between an instance and each such elementary distribution. The main contribution of this work lies in computing such similarities with the introduced Gated Domain Unit (GDU), a module to be employed along with neural networks, responsible for encoding each elementary distribution into an embedding that can be trained via backpropagation with the parameters of the ensemble. Empirical evaluation of the proposed approach was carried out on multiple benchmarks comprising, for example, computer vision tasks and time-series classification. Results showed the proposed approach outperforms the selected baselines with respect to accuracy on unseen domains at training time.", "summary_of_the_review": "This work tackles the domain generalization setting and assumes that problems within this setting can be seen as if each domain is a mixture of the same distributions (elementary distributions). The authors attempted to provide in Lemma 1 a theoretical motivation to support the introduced assumption, but I have concerns and questions about this result.  The authors then proposed architectural changes to deep neural networks by introducing Gated Domain Units to allow generalization to unseen domains for which the introduced assumption holds. Even though I appreciate the practical relevance of the problem being tackled by this submission, I found that the manuscript lacks in multiple points (please refer to the previous sections of this review for comments and suggestions), especially in the motivation for the introduced assumption and empirical evaluation in terms of considered baselines, reported metrics, as well as the aspects investigated in the experiments. All in all, I believe this submission requires multiple improvements in order to be considered for publication.", "strength_and_weaknesses": "Strengths:\n- The paper tackles a relevant and open problem for the machine learning community;\n- The proposed approach is theoretically grounded and its algorithmic instantiation seems practical. Also, it does not rely on domain labels;\n- Empirical evaluation was extensive in the sense that multiple datasets and tasks were taken into account;\n- Results show that the introduced method presents advantages in comparison to the selected baselines in terms of performance on unseen domains at training time. \n \nWeaknesses:    \n- Lack of soundness in the theoretical motivation, more specifically in Lemma 1 (see the following section of the review for more details);\n- The manuscript contains multiple claims and statements that are not well-supported / clear (see the following section of the review for more details);\n- It is not clear, from both theoretical and empirical perspectives, what is the impact on out-of-distribution generalization of selecting a much higher / lower number of elementary distributions;\n- Experimental evaluation lacks in many aspects:\n  - In the main paper, the proposed approach is only evaluated in terms of performance on unseen domains. This type of evaluation does not explain the actual source of the observed improvements;\n  - The authors did not take into account closely related approaches as baselines in the empirical evaluation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qShs1xjkXZ3", "reviewer": "Reviewer_FZYk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for domain generalization. The core assumption is that every domain can be decomposed to some elementary domain (something like a base domains) which are invariant. Every domain is a linear combination of these base domains. With this assumption, we can have elementary domain prediction functions and we just need to figure out how to linearly combine their predictions. For a given input example, this is done by measuring the similarity of the input example with each base domain. ", "summary_of_the_review": "The paper is theoretically interesting. Im not sure how applicable it is to practical applications for example generalizability outside of the convex hull of the source domains. ", "strength_and_weaknesses": "The idea is theoretically interesting and it certainly is versatile within the convex hull of the base domains. It relates to some of the existing work on recycling neural networks. I suggest the authors to cite those papers and if possible compare with those baselines. \n\n\nSome questions for the authors:\n- in figure 7 of the supplementary material, how come adding elementary domains does not improve results? \n- The base domains are based on the domains observed from source data. Therefore we can only assume the generalizability of this approach extends to test domains which can be considered an interpolation of source domains. How can the generalizability of the approach extrapolate outside the convex hull of the source domains?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MV-uOqBCs_", "reviewer": "Reviewer_1an7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Based on an assumption about invariant elementary distributions, the authors proposed a method, namely Gated Domain Units, to recover those elementary distributions and leverage them for the domain generalization problem.", "summary_of_the_review": "I would like some clarification on the following issues:\n- The authors claim that \"Lemma 1 shows the theoretical appeal of the I.E.D. assumption\". Can the authors elaborate? Even if we know the I.E.D.s and their coefficient in the mixture (which we would never know, but just assume here), then still it is very hard to find the set of Pareto optimal solution $f$. Then why Lemma 1 would show the theoretical appeal of this assumption or motivate us to find such elementary distributions?\n\n- The practicality of the I.E.D. assumption: can the authors give an example of the I.E.D. in some of the datasets used in the paper? The authors give one example in section 2.1 but I don't think it is concrete enough. For me, an ideal way to do this for 2 domain $p_s$, $p_t$ is to somehow decompose them into $p_s = \\alpha_s^1.p_s^1 + ... + \\alpha_s^k.p_s^k$ and $p_t = \\alpha_t^1.p_t^1 + ... + \\alpha_t^k.p_t^k$ such that a discriminator can't distinguish between $p_s^i$ and $p_t^i$ well.\n\n- The empirical evaluation is also not quite convincing to me. The method needs to process data with multiple Gated Domain Units, thus increasing the training and inference complexity. Therefore, ensemble is a natural baseline for this method. And indeed, the authors do include this baseline for mnist and ECG. But I wonder why the authors do not consider this baseline for WILDS? This would make the empirical evaluation much stronger.", "strength_and_weaknesses": "In my opinion, the main weaknesses of the paper are the assumption of I.E.D (which the authors did discuss briefly) and the evaluation of the method. Details are in the section below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject