Gradient Matching for Domain Generalization
Authors: Yuge Shi
Jeffrey Seely
Philip Torr
Siddharth N
Awni Hannun
Nicolas Usunier
Gabriel Synnaeve
Venue: ICLR 2022 Poster (2022)
Keywords: Domain generalization
multi-source domain adaptation

Abstract: Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an _inter-domain gradient matching_ objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive --- it requires computation of second-order derivatives –-- we derive a simpler first-order algorithm named Fish that approximates its optimization. We perform experiments on the Wilds benchmark, which captures distribution shift in the real world, as well as the DomainBed benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks.

Published as a conference paper at ICLR 2022
GRADIENT MATCHING FOR DOMAIN GENERALIZA- TION
Yuge Shi'
Jeffree Seely Meta Reality Labs jsee y fb. com
Philip H.S. Torr University of Oxford philip torréeng ox ac...ac.uu uk
University of Oxford ysh @robots. ac.uk
N. Siddharth
Awni Hannun'
Nicolas Usunier Facebook Al Research
The University of Edinburgh & The Alan Turing Institute
Facebook Al Research awni@ fbb. com
usunier0 fb. com
n. s w.ssidhhartheee ac .uk
Gabriel Synnaeve
Facebook AI Research gabê fb. com
ABSTRACT
Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement such systems in the real world is their ability to generalize unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive we derive simpler
requires computation of second-order derivatives first-order algorithm named Fish that approximates its optimization. We perform experiments on the WILDS benchmark, which captures distribution shift the real world, as well the DOMAINBED benchmark that focuses more synthetic- to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across wide range of domain generalization tasks. Code is available
com/ YugeTen. fish
INTRODUCTION
at lnttl point
The goal of domain generalization is to train models that performs well on unseen, out-of out-of-distribution data, which is crucial in practice for model deployment in the wild. This seemingly difficult task made possible by the presence of multiple distributions/domainss at train time. As we have seen in past work (Arjovsky et al., 2019 Gulrajani and Lopez-Paz 2020: Ganin et al. (016). key aspect of domain generalization is to learn from features that remain invariant across multiple domains, while ignoring those that are spuriously correlated to label information (as defined in Torralba and Efros (20TT); Stock and Cisse (2017)) ). Consider, for example, a model that is built to distinguish between cows and camels using photos collected in nature under different climates. Since CNNs are known to have bias towards texture (Geirhos et al. 2018 Brendel and Bethge| 2019). if we simply try to minimize the average loss across different domains, the classifier is prone to spuriously correlate "cow" with grass and "camels" with desert, and predict the species using only the background. Such classifier can be rendered useless when the animals are placed indoors or in a 2oo. However, if the model could recognize that while the landscapes change with climate, the biological characteristics of the animals
Traledary
Trjlcker duauiing TRM ninging
Gradent of the ices for domain
podid
Goddier theires demeii
MAY Eals
Domans
Figure 1: Isometric projection of train- ing with ERM (blue) vs. our objective (dark blue), using data from Figure 2
Work done during internship at Facebook Al Research.
Now at Zoom.

---

Published as a conference paper at ICLR 2022
(e.g. humps, neck lengths) remain invariant and use those features to determine the species, we have a much better chance at generalizing to unseen domains.
Similar intuitions have already motivated many approaches that consider learning "invariances" across domains as the main challenge of domain generalization. Typically, lot of these work focus on learning invariant representations directly by removing the domain information (Ganin al.] 2016; [Snn and Saenko, 2016 Li et al.) 2018). In this work, we propose an inter-domain gradient matching (IDGM) objective. Instead of learning invariant features by matching the distributions of representations from different domains, our approach does so by encouraging consistent gradient directions across domains. Specifically, our IDGM objective augments the loss with an auxiliary term that maximizes the gradient inner product between domains, which encourages the alignment between the domain-specific gradients. By simultaneously minimizing the loss and matching the gradients, IDGM encourages the optimization paths to be the same for all domains, favouring invariant predictions. Figuuee T] illlustrates a motivating example described in [Section 3.2) given 2 domains, each containing one invariant feature (orange cross) and one spurious feature (yellow and red cross). While empirical risk minimization (ERM) minimizes the average loss between these domains at the cost of learning spurious features only, IDGM aligns the gradient directions and is therefore able to focus on the invariant feature.
While the IDGM objective achieves the desirable learning dynamic in theory, naive optimization of the objective by gradient descent is computationally costly due to the second-order derivatives. Leveraging the theoretical analysis of Reptile, meta-learning algorithm (Nichol et al., (018), propose to approximate the gradients of IDGM using a simple first-order algorithm, which we name Fish. Fish is simple to implement, computationally effective and we show in our experiments, functionally similar to direct optimization of IDGM.
Our contribution is a simple but effective algorithm for domain generalization, which exhibits state-of- the-art performance on 13 datasets from recent domain generalization benchmark WILDS (Koh et al. 2020) and DOMAINBED (Gulrajani and Lopez-Pat Paz 2020). The strong performance of our method on variety of datasets demonstrates that it is broadly applicable in different applications/subgenee of domain generalization tasks. We also perform detailed analysis in [Section 4.4 explain the effectiveness of our proposed algorithm.
2 RELATED WORK
Domain Generalization In domain generalization, the training data is sampled from one many source domains, while the test data is sampled from a new target domain. We will now discuss the five main families of approaches domain generalization:
1. Distributional Robustness (DRO): DRO approaches minimize the worst-case loss over set of data distributions constructed from the training domains. Rojas-Caruus Carulla et al. (2015) proposed DRO to address covariate shift (Gretton et al., 2209a]bb) b), where P(Y[X) remains constant across domains but P(X) ) changes. Later work also studied subpopulation shift, where the train and test distributions are mixtures of the same domains, but the mixture weights change between train and test (Hu et al. 2018 Sagawa et al. 2019);
2. Nomain-invarian representation learning: This family of approaches to domain generalization aims learning high-level features that make domains statistically indistinguishable. Prediction is then based on these features only. The principle is motivated by a generalization eтror bound for unsupervised domain adaptation (Ben-David et al. [2010] Ganin al. (0166) but the approach readily applies to domain generalization (Gulrajani and Lopez-Paz, 2020 Koh et al. 2020). Algorithms include penalising the
power of the model (Ganin et al.
2016
Wang al. 2019 Huang et al., 2020), aligning domains through contrastive loss (Motiaan et al. 2017), matching mean and variance of feature distributions across domains (Sun and Saenko 2016). learning useful representations by solving Jigsaw puzzles (Carlucci et al. [2019), using the maximum mean diserepancy to match the feature distributions (Li et al. 2018b) or introducing training constraints across domains using mixup formulation (Yan et al. 2020).
3. Invariant Risk Minimization (IRM): IRM is proposed by AArjossyy et al. (2019), which learns an intermediate representation such that the optimal classifiers (on top of this representation) of all domains are the same. The motivation is to exploit invariant causal effects between domains while
2

---

Published as a conference paper at ICLR 2022
reducing the effect of domain-specific spurious correlations. From optimization perspective, when IRM reaches its optimal, all the gradients (for the linear classifier) has to be zero. This is why IRM's solution won't 't deviate from ERM when ERM is optimal for every domain, which is not the case for our proposed IDGM objective due to the gradient inner product term.
4. Data augmentation: More recently, approaches that simulates unseen domains through specific types of data augmentaatio/normaiiaaatt has been gaining traction. This includes work such as Zhou et al. ( Volpi and Murino (2019); lise et al. (2021), as well as Seo et al. (019)) which utilises ensemble learning.
5. Gradient alignment: Two concurrent work - -Koyama and Yamaguchi (2021) and Parascandolo al. 2021 utilise similar gradient-alignmert principle for domain generalization. Koyama and Yamaguchi (202T) proposes IGA, which learns invariant features by minimizing the variance of inter-domain gradients. The key difference between IGA and our objective is that IGA is completely identical to ERM when ERM is the optimal solution on every training domain, since the variances of the gradients will be zero. While they achieve the best performance on the training set, both IGA and ERM could in some cases, completely fail when generalizing to unseen domains (see Secciion 3.2 for such an example). Our method, on the contrary, biases towards non-ERM solutions as long as the gradients are aligned, and is therefore able to avoid this issue. Parascandolo et al. (202T) on the other hand, proposes to mask out the gradients that have opposite signs for different domains. Unlike their work that prunes gradients that are inconsistent, our approach actively encourage gradients from different domains to consistent by maximizing the gradient inner product. Additionally, in Lopez- Popez-Paz&one Paz and Ranzato (2017) also the application of gradient- alignment, however in this case it is applied under the continual learning setting to determine whether a gradient update will increase the loss of the previous tasks.
Apart from these algorithms that are tailored for domain generalization, well-studied baseline in this area is ERM, which simply minimizes the average loss over training domains. Using vanilla ERM is theoretically unfounded (Hashimoto et al. 2018 Blodgett al. 2016 Tatman 2017) since ERM is guaranteed to work only when train and test distributions match. Nonetheless, recent benchmarks suggest that ERM obtains strong performance in practice, in many case surpassing domain generalization algorithms (Gulrajani and Lopez-Paz 2020) Koh et al. 2020). Our goal is to fill this gap, using an algorithm significantly simpler than previous approaches.
Connections to meta-learning There are close connections between meta-learning (Truun and Pratt 1998) and (multi-source) domain adaptation. In fact, there are few works in domain generaliza- tion that are inspired by the meta-learning principles, such as Li et al.) (2018)); Balaji et al. (2018) : Li et al. )2019); Dou et al. 2019). Specifically, [i et al. (2020) also proposes to adapt Reptile for domain generalization tasks, however they study their method under the sequential learning setting, whereas our method can be trained on all domains and therefore learns faster, especially when the number of domains is large. In Ren et al. 2018), we also see the leveraging of gradient inner product in meta-learning, where it is used to determine the importance weight of training examples. We discuss the connection between our proposed algorithm to leaanig in more details in Apppeddix A.
Note that our proposed algorithm Fish is similar to the Mean Teacher method (Tarvainen and Valpola 2017]. where a teacher model (cquivalent to 0 in Algorithm I} is computed using a moving average of the student model (equivalent to 0 in Algorithm
3 METHODOLOGY
3.1 GOALS
Consider training dataset Der consisting of S domains D., = (Di, DDs), where each domain
characterized by a dataset D, := {(rf yf )= containing data drawn i.i.d. from some probability distribution. Also consider a test dataset D,e consisting of T domains Dze = (Ds41, DD54 }.
where Der n Dre = 0. The goal of domain generalization is to train a model with weights that generalizes well on the test dataset D,, such that:
arg argmin min EDWD, EODND [((,,, 0);
(1)
0
3

---

Published as a conference paper at ICLR 2022
((7g)
p(Pa)
D
0
0
comaais
D
0
D3
P( 1
PlY
[2))
[ps) 0.3
Figure 2: All 3 domains (rows) consist of types of inputs (columns): 1) 21, left: makes up for 50% each domain, label always O, an is always [0, 0, 0, 0]; 2) æ2, middle: makes up for 40% of each domain, label is always 1, 22 changes for each domain; 3) æt right: makes up for 10% of each domain, labels are randomly assigned with 30% of y = and 70% of y = 23 is always [1, 0, 0, 0].
where (((x, 0) is the loss of model evaluated on v).
naive approach is apply ERM, which simply minimizes the average loss Drr, ignoring the discrepancy between train and test domains:
Lama Per 0) ED~D., (x,,)>> [[((z, y):
The ERM objective does not exploit the invariance across different domains in Der and could perform arbitrarily poorly on test data. We demonstrate this effect with the following simple linear example.
3.2 THE PITFALL ERM: LINEAR EXAMPLE
Consider binary classification setup where data (ar, v) E BA IB, and a data instance is denoted x [fi, f2. fa, f4], y. The train domains are {Di, D2). J. and test domain is D3. The goal is to learn a linear model W x b = y, W E Rd, b E E on the train data, such that the error the test domain is minimized. The setup and dataset of this example illustrated in Figure
As we can see in Figure 2] S1 is the invariant feature in this dataset, since the correlation between Si and y is stable across different domains. The relationships between y and fz. fa and fa changes for D1. D2, D3. making them the spurious features. Importantly, if we consider one domain only, the spurious features f2. fa and fa are a more accurate indicator of the label than the invariant feature fi- For instance, using f2 predict y can give 97% accuracy Di, while using fa only achieves 93% accuracy.
Table 1: Performance comparison on the linear dataset.
Method
train acc.
test acc.
W
ERM
97%
57%
[2883.3, 3.3,0.00000
-2.7
IDGM
93%
93%
0.4, 0.2.2...2,0..0 0.0
-04
Fish
93%
93%
0.,4, 0.2,
-0.4
The performance of ERM this simple example is shown in Table (first row). From the trained parameters W and b, we see that the model places most of its weights on spurious features fz and f3- While this achieves the highest train accuracy (97%), the model cannot generalize to unseen domains and performs poorly on test accuracy (57%).
3.3 INTER-DOMAIN GRADIENT MATCHING (IDGM)
To mitigate the problem with ERM. we need an objective that learns from features that are invariant across domains. Let us consider the case where the train dataset consists of S = 2 domains D.r = (Di, D2). Given model and loss function 1, the expected gradients for data in the two domains is expressed as
Ol((,y):0) = G
G2 ED, ((((,)))0)
00
20
The direction, and by extension, inner product of these gradients are of particular importance to goal of learning invariant features. If G, and G2 point in a similar direction, i.c. G, G2 > 0, taking gradient step along G1 or G2 improves the model's performance both domains, indicating that
4

---

Published as a conference paper at ICLR 2022
the features learned by either gradient step invariant across (D1, D2). This invariance cannot be guaranteed G1 and G2 are pointing in opposite directions, i.e. G1 G2 S 0.
To exploit this observation, we propose maximize the gradient inner product (GIP) to align the gradient direction across domains. The intended effect is to find weights such that the input-output correspondence is close as possible across domains. We name objective inter-domain gradient matching (IDGM), and is formed by subtracting the inner product of gradients between domains G from the original ERM objective. For the general case where 5
2 2, we can write
2 Lidgm Lem (Dtr: S(S - 1) ijes
EG. G.
(4)
GIP, denote as
where 7 is the scaling term for G. Note that GIP can be computed in linear time as 1 -
Σ, || (ignoring the constant factor). We can also compute the stochastic estimates of Equation (4)
by replacing out the expectations
the entire dataset by minibatches.
We test this objective on our simple linear dataset, and report results in the second row of Table Note that avoid exploding gradient
the normalized GIP during training. The model has lower training accuracy compared to ERM (93%), however its accuracy remains the same on the test set, much higher than ERM. The trained weights W reveal that the model assigns the largest weight to the invariant feature f1. which is desirable. The visualization in Figure also confirms that by maximizing the gradient inner product, IDGM is able to focus on the feature that is common between domains, yielding better generalization performance than ERM.
3.4 OPTIMIZING IDGM WITH FISH
The proposed IDGM objective, although effective, requires computing the second-order derivative of the model's parameters due to the gradient inner product term, which can be computationally prohibitive. To mitigate this, we propose a first-order algorithm named Fish that approximates the optimization of IDGM with inner-loop updates. In AnAAlgrri [w present Fish. As comparison, we also present direct optimization of IDGM using SGD in Algorithm
Algorithm Fish.
Algorithm 2 Direct optimization of IDGM.
1: for itcrations = 1,2, ... do 8-
1: for itcrations = 1,2,- do
for D; E permppeeuuee). ((D1,, D.....)) do
3: for D, E permute((D. D.. (( ,D,) do
4: Sample batch di ~ D; al((2, y):
4: Sample batch d3 ~ D,
al((,, 0):
5:
9 = Edd
Grad wrt Ø
5: 9: =
Edt
// /Gaa wrt
00
00
Update ē agj
end for
GIP (utch)
is
2
end for
Jar
S(S
1)
9r -.-9 9j
Update 0 + 0 e(0 0)
a
iJES
8: Update +- (g - 100000/000000
end for
9:
end for
Fish performs S' inner-loop (13-17) update steps with learning rate a on clone of the original model 0, and each update uses minibatch d, from the domain selected in step i. Subsequently, 0 is updated by a weighted difference between the cloned model and the original model e(0 0).
To see why Fish is an approximation to directly optimizing IDGM, we can perform Taylor-series expansion its update in 18, [Agoriihhm Doing so reveals two leading terms: g: averaged gradients over inner-loop's minibatches (effectively the ERM gradient); 2) 0g/00: gradient of the minibatch version of GIP. Observing of Algorithm 2, we see that ğ and 9 are actually the two gradient components used in direct optimization of IDGM. Therefore, Fish implicitly optimizes
Following the convention of naming this style of algorithms after classes of vertehrates (animals with backhones).
5

---

Published as a conference paper at ICLR 2022
IDGM by construction (up constant factor), avoiding the computation of second-ordd derivative 0g/09. We present this more formally for the full gradient G in Theorem 3.1
Theorem 3.1 Given wwice-differentaable model with parameters 0 and objective Let us define the
following:
G B[([ 00] aas-
Fish update o.S-ERM grad
Gg -G/
grad of max (G)
where G == Ex=1 G. and is the full gradient of ERM. Then we have
G5 Gg
lim a a-
|Gy GG
Note that the expectation in Gy is over the sampling of domains and minibatches. Theorem 3.J indicates that when a sufficiently small, if we remove the scaled ERM gradient component G from Fish's update, we are left with a term Gy that in similar direction the gradient of maximizing the GIP term in IDGM, which was originally second-order. Note that this approximation comes at the cost of losing direct control over the GIP scaling ? - we therefore also derived smoothed version of Fish that recovers this scaling term, however we find that changing the value of y does not make much difference empirically. See/Appeddix for more details.
The proof to Theorem 3.1 can be found in Appendix A We follow the analysis from Nichol et al. (2018), which proposes Reptile for model-agnostic meta-learning (MAML), where the relationship between inner-loop update and maximization of gradient inner product was first highlighted. Nichol iet al. (2018) found the GIP term in their algorithm to be over minibatches from the same domain, which promoted within-task generalization; in Fish we construct inner-loop using minibatches over different domains - it therefore instead encourages across-domale generalization. We compare the two algorithms in further details in Appendix A.1
We also train Fish on our simple linear dataset, with results in Table I and see it performs similarly to IDGM the model assigns the most weight to the invariant feature /1, and achieves 93% accuracy on both train and test dataset.
4 EXPERIMENTS
4.1 CDSPRITES-N
Dataset We propose simple shape-color dataset CDSPRITES-N based on the DSPRITES dataset (Matthey et al. 2017), which contains a collection of white 2D sprites of different shapes, scales, rotations and positions. CDSPRITES-N contains N domains. The goal sprites, and there is a shape-color deterministic matching that is specific per domain. This way we have shape the invariant feature and color the spurious feature. See Figure for an illustration.
classify the shape of the
To construct the train split of CDSPRITES-N NN we take a subset of DSPRITES that contains only 2 shapes (square and oval). We make N replicas this subset and assign 2 colors each, with every color corresponding to one shape (e.g. yellow block in Figure 3a pink + squares, purple + oval). For the test split, we create another replica of the DSPRITES-N subset, and randomly assign one of the 2N colors in the training set to each shape in the test set.
We design this dataset with CNN's texture bias in mind (eiihoos et al., 2018 Brendel and Bethge 2019). If the value of N is small enough, the model can simply memorize the N colors that correspond to each shape, and make predictions solely hased on colors, resulting in poor performance on the test set where color and shape are longer correlated. Our dataset allows for precise control over the features that remains stable across domains and the features that change as domains change; we can also change the number of domains N easily, making it possible to examine the effect N has on the performance for domain generalization.
Results We train the same model using three different objectives including Fish, dicrect optimization of IDGM and ERM on this dataset with number of domains ranging from 5 to 50. Again, for direct optimization of IDGM, we use the normalized gradient inner product avoid exploding gradient.

---

Published as a conference paper at ICLR 2022
Accurauc (%)
Accurpony
Fish
Flsb
Direct
Bireet
Number
30
Nunder
25
30
(a) Train
(b) Test
(a) Train
(b) Test
Figure 3: CDSPRITES-N visual- Figure 4: Performance of Fish, IDGM and ERM on CDSPRITES- ization. Each 3x3 grid (e.g. yel- N, with N e 50]
low square) is one domain.
We plot the average train, test accuracy for each objective over runs against the number of domains N in Figure 4. We can see that the train accuracy is always 100% for all methods regardless of N (Figure 4ah, while the test performance varies: Figure 4b shows that direct optimization of IDGM (red) and Fish (blue) obtain the best performances, with the test accruacy rising to over 90% when N 10 and near 100% when N 2 20. The predictions ERM (yellow), on the other hand, remain nearly random on the test set up until N = 20, and reach 95% accuracy only for N 2 40.
This experiment confirms the following: the proposed IDGM objective have much stronger domain generalization capabilities compared to ERM; 2) Fish is an effective approximation of IDGM, with similar performance to its direct optimization. We also plot the gradient inner product progression of Fish ERM during training in Figure 9a| showing clearly that Fish does improve the gradient inner product across domain while ERM does not; 3) we also observe during training that Fish about 10 times faster than directly optimizing IDGM, demonstrating its computational efficiency.
4.2
WILDS
Datasets We evaluate our model on the WILDS benchmark (Koh et al. 2020). which contains multiple datasets that capture real-world distribution shifts across diverse range of modalities. We report experimental results on 6 challenging datasets in WILDS, and find Fish to outperform all baselines on most tasks. A summary of the WILDS datasets can be found in (pppendix C For hyperparameters including learning rate, batch size, choice of optimizer and model architecture, we follow the exact configuration as reported in the WILDS benchmark. Importantly, we also use the same model selection strategy used in WILDS to ensure a fair comparison. See details in Appendix D
Table 2: Results on WILDS benchmark.
POVERTYMAP
CAMELYON17
FMoW
CiVI.COMMENTS
IWILDCAM
AMAZON
Worst-U/R Pearson
Avg.
Worst
Worst (9)
Macro Fl
10-th per.
Fish
0.30 (1t-e)
74.7 (171)
34.6 (1018)
75.3 (ine)
22.0 (2.9)
53.3 (1:0)
(11w2)
(181)
(1137)
66.3 (12.)
(249)
(10a)
Coral
0.44 (#6022
39.5 (+77)
31.7 (#124)
65.6 (=1.))
32.8(+0.)
$2.9 (#00)
Reweighted
(1n9)
(100)
GroupDRO
0.39 (#6022)
68.4 (#72)
30.8 (sost)
(=220)
23.9 (=2J)
$3.3 (#0o)
ERM
0.45 (#te-2)
(#644
32.3
56.0 (=20)
31.0
53.8 (soa)
ERM (ours)
0.29 (#12-2)
70.5 (=122.)
30.9 (#1.5)
58.1 (#1.7)
25.1 (=02)
53.3 (#04)
Results See summary of results in Table where we use the metrics recommended in WILDS for each dataset. Again, following practices in WILDS, all results reported over random seed runs, apart from CAMELYONI) 7 which uses 10 random seeds and CiVILCOMMENTS which uses 5, We included additional results as well as in-depth discussion on each dataset in Appendix CJ and an ablation studies on Fish's hyperparameters in Appendix F anddd Appendix E. We make the following observations:
Strong performance across datasets: Considering results on all datasets, Fish is the best performing algorithm on WILDS. It significantly outperforms baselines on 3 datasets and achieves similar level of performance to the best method on the other (AMAZON and IWILDCAM). Fish's strong performance different types of data and architectures such as RESNET (He et al. (2016). DENSENET (Huang et al. 2017) and DISTILBERT (Sanh et al., 2019) demonstrated it's capability generalize diverse variety of tasks;

---

Published as a conference paper at ICLR 2022
Table 3: Test accuracy (%) on DOMAINBED benchmark.
ERM
IRM
GrouDDRO
Mixup
MLDG
Coral
MMD
DANN
CDANN
Fish (ours)
CMNIST
51.5
(2a.))
52.1 (100)
52.1 (10z)
51.5 (des)
51.5 (au)
(10:)
51.5 (104)
51.7 (197)
51.6 (1s.)
RMNIST
98.0 (+:00 97.7 (#az)
98.0 (#00)
98.0 (=0,)
(=
(-a) 97.9 (=0.)
97.8 (#0.)
97.9 (=:7)
98.0 (#30)
VLCS
77.5 (eo4)
78.5 (+0.5)
76.7 (280)
(=00)
77.2 (204)
78.8 (=06) 77.5 (=09)
(+94)
77.5 (227)
77.8 (+6.2)
PACS
(1.5)
(190)
84.4 (3n4)
(2.6(200)
84.9 (1:d)
(anr)
(zar)
83.6 (194)
82.6 (199)
85.5 (3A
OfficeHome
66.5 (ecJ) 64.3 (#22)
66.0
68.1 (=0.7
66.8 (=0.)
68.7 (=07)
66.3 (=aJ)
65.9 (496)
65.8 (=:3)
68.6 (#6.4)
TerraInc
46.1 (1:79) 47.6 (1n4)
43.2 (27.
47.9 (2n4)
47.7 (200)
42.2 (21o)
(10c)
45.8 (27A)
45.1 (a:1)
DomainNet
40.9 (10/)
33.9 (126)
33.3 (102)
39.2 (Iav)
(1nu)
41.5 (10))
23.4
38.3 (Ln))
38.3 (19.5)
42.7 (162)
Average
66.6 65.4
64.8
66.7
66.7
67.5
63.3
66.1
65.6
67.1
2. Strong performance on different domain generalization tasks: We make special note the CIVILCOOMENTS dataset captures subpopulation shift problems, where the domains in test are a subpopulation of the domains in train, while all other WILDS datasets depicts pure domain generalization problems, where the domains in train and test are disjointed. As a result, the bascline models for CIVILCOMMENTS selected by the WILDS benchmark are different from the methods used in all other datasets, and are tailored to avoiding systematic failure on data from minority subpopulations. We see that Fish works well in this setting too without any changes or special sampling strategies (used for baselines on COMMENTMMN,S see more in Table 10), demonstrating it's capability to perform in different domain generalization scenarios;
3. Failure mode of domain generalization algorithms: We noticed that on IWILDCAM and AMA- ZON, ERM is the best algorithm, outperforming all domain generalization algorithms except for Fish on AMAZON. We believe that these domain generalization algorithms failed due to the large number of domains in these two datasets- 324 for IWIIDCAAM and 7,676 for AMAZON. This is a common drawback of current domain generalization literature and is direction worth exploring.
4.3 DOMAINBED
Datasets While WILDS is a challenging benchmark capturing realistic distribution shift, to test our model under the synthetic-to-real transfer setting and provide more comparisons to SOTA methods, we also performed experiments on the DOMAINBED benchmark (Gulrajani and Lopez-Pat Paz 2020).
See summary of DOMAINBED in Amppeedix H
Results Following recommendations in DOMAIN we report results using training domain as validation set for model selection. See results in Table 3 reported over 5 random trials. Averaging the performance over all 7 datasets, Fish ranks second out of 10 domain generalization methods. performs only marginally worse than Coral (0.1%), and is one of the three methods that performs better than ERM. This showcases Fish's 's effectiveness on domain generalization datasets with stronger focus to synthetic-to- -e-rell transfer, which again demonstrates its versatility and robustness on different domain generalization tasks.
4.4 ANALYSIS
We show extensively through empirical evaluation that Fish is very effective for a variety domain generalization tasks. In this section, we perform analysis to validate that Fish's strong performance is due to inter-domain gradient inner product maximization.
Does Fish maximize gradient inner product (GIP) empirically? In Figure 5) we plot the progression of GIP during training using different objectives. We train both Fish (blue) and ERM (yellow) on CDSPRITES- until convergence while tracking the normalized GIP between minibatches from different domains used in each inner-loop. To ensure a fair comparison, we use the exact same sequence data for Fish and ERM (see Appendix I| for more details).
Chadkent Nura po0dr1
EtM
From Figure 5| it is clear that during training, the normalized GIP of Fish increases, while that for ERM stays at the same value. The observations here shows that Fish is indeed effective in increas- ing/maintaining the level of inter-domain GIP.
Figure 5: Gradient inner prod- uct values during the training for CDSPRITES-N (N=15).
8

---

Published as a conference paper at ICLR 2022
Table 4: Test accuracy on four datasets, in three partitions are 1) two baselines Fish and ERM, 2) Fish with random grouping strategy (Fish, RG) and 3) ERM with domain grouping strategy (ERM, DG, *)
FMOW
VLCS
PACS
OfficeHome
Fish (ours)
34.3 (10.)
77.6 (10.)
85.5 (10.))
68.6 (2099
ERM
31.7 (=10)
77.5 (404)
85.5 (102)
66.5 (#0.2)
Fish, RG
33.4 (117)
77.7 (10J)
83.9 (107)
66.5 (L7.0)
ERM, DG, prop. Ir
32.1 (#0.9
72.7 (#:4)
83.2 (#07)
58.5 (z0/)
ERM, DG, 0.1x prop. Ir
29.9 (1077)
73.9 (106)
84.2 (10.4)
65.1 (10.1)
ERM, DG, x.01 prop.
29.8 (#04) 74.7 (+04) 84.0 (#06)
63.7 (=02)
We conduct the same GIP tracking experiments for the WILDS datasets we studied as well to shed some lights on its efficiency - see AAppendix Gor results.
Does maximizing GIP between random batches help domain generalization? In this part of the analysis, we examine the effectiveness of another element of our algorithm -th construction of minibatches. We conduct experiments where data are grouped randomly instead of by domain for the inner-loop update. By doing so, we are still maximizing the inner product between minibatches, however not strictly between domain. We therefore expect the results to be slightly worse than Fish, and the bigger the domain gap is, the more advantage Fish has against the random grouping strategy.
We show the results for random grouping (Fish, RG) in Table 4 As expected, the random grouping strategy performs worse than Fish on all datasets. The experiment demonstrated that our algorithm does benefit from the domain grouping strategy, and that maximising GIP between random batches of data, while still achieving strong results, does not achieve the same domain generalization performance as Fish.
Differences between Fish and ERM We have shown through empirical evaluation that Fish and ERM are sufficiently different. Most notably, they converge to completely distinet minimum in our linear toy example and CDSPRITES experiment, resulting in large decrepencies in test accuracy.
However, some may notice that Fish first-order method shares algorithmic similarities to ERM. In fact, if we fix the meta learning rate of Fish e = 1 and use domain sampling for ERM, the two algorithms are equivalent. Although the optimal es found through hyperparameter search are much smaller than 1, one might wonder if we can offset the increased meta learning rate e by reducing the learning rate a proportionally, and achieve similar performance to Fish using simply ERM with domain grouping.
In Table 4 (ERM, DG), we verify that this is infeasible empirically. We train an ERM model with domain grouping strategy (equivalent to Fish with 1) using 3 different learning rates: 1) prop. Ir, learning rate a is lowered proportionally to keep ae constant; 2) 0. xprop. ir and 3) 0.001 xprop. which further reduces the learning rate. We see that on all datasets, all three configurations result in worse performance than ERM and Fish. This shows that one cannot improve the results of ERM to match that of Fish by simply adjusting the learning rate/adopting domain grouping strategy, and that Fish's gain on domain generalization tasks is not due to its proximity to HRM.
In Appendix A.2 we also show that setting c is equivalent to setting S as the total number of iterations during training, which causes the effect of maximizing GIP to diminish. This provide further support to our empirical finding in this analysis.
Does maximising GIP help domain generalization? In Parascandolo et al. (222)) authors show that by masking out the gradients that have opposite signs in different domains, they able to trade-off some "learning speed" for prioritizing learning the invariances. Our principle for proposing the IDGM objective identical, except in this case we actively encourage these gradients to be of consistent signs. This unfortunately, does result in computing the costly second-order derivative, which makes it difficult for us to empirically evaluate the IDGM objective on non-toy datase We believe that the large performance gain of IDGM over ERM on the linear and CDsPRITES datasets, along with the detailed discussion on the relationship between gradient sign consistency and invariant representation, is enough to justify for IDGM's ability to improve domain generalization.
For context, training an IDGM model on VLCS dataset with hatch size 32 and standard RESNET 8 backbone does not fit NVIDIA V100 (32GB) GPU.
9

---

Published as a conference paper at ICLR 2022
5 CONCLUSION
In this paper we presented inter-domain gradient matching (IDGM) for domain generalization. To avoid costly second-order computations, approximated IDGM with simple first-order algorithm, Fish. We demonstrated our algorithm's capability to learn from invariant features (as well as ERM's failure do so) using simple datasets such as CDSPRITES-N and the linear example. We then evaluated the model's performance on WILDS and DOMAINBED, demonstrating that Fish performs well on different subgenres of domain generalization, and surpasses baseline performance on a diverse range of vision and language tasks using different architectures such as DenseNet, ResNet-50 and BERT. Our experiments can be replicated with 1500 GPU hours on NVIDIA V100.
Despite its strong performance, similar to previous work on domain generalization, when the number of domains large Fish struggles to outperform ERM. We are currently investigating approaches by which Fish can be made to scale to datasets with orders of magnitude more domains and expect to report on this improvement our future work.
10

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
We believe there are no ethical concerns within this work. None of the datasets used involve human identities, and our motivation in proposing this work in no way concerns any surveillanc/mmiitaryy use. As with many other domain generalization methods, our algorithm aims at improving the performance of machine learning systems deployed in the wilds, which can be used many ways that will benefit the society.
REFERENCES
M. Andrychowiez, M. Denil, Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford. and N. De Freitas. Learning to leamn by gradient descent by gradient descent. arXiv preprint 1606. 04474, 2016.
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 19X.009077.0299 2019.
Y. Balaji, S. Sankaranarayanan, and R. Chellappa. Metareg: towards domain generalization using meta-regularizztion.. In NIPS 18 Proceedings of the 32nd International Conference on Neural Information Processing Systems, volume 31, pages 1006-1016, 2018.
Beery, V. Horn, and P. Perona. Recognition terra incognita. In Proceedings of the European 2018.
Conference on Computer Vision (ECCV), pages
Ben-David, J. Blitzer, K. Crammer, Kulesza, Pereira, and J. Vaughan. A theory of learning from different domains. Machine learning, 79()):511-15,.0000
Blodgett, Green, and B. T. O'Connor. Demographic dialectal variation in social media: A case study of african- english. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pages 1119-11300 2016.
W. Brendel and M. Bethge. Approximating cnns with bag-of- waa-llocll-fatte -eaturrss models works surprisingly well on imagenet. In International Conference on Learning Representations (ICLR), 2019.
F. M. Carlucci, A. D'Innocente, S. Bucci, B. Caputo, and T. Tommasi. Domain generalization by solving jigsaw puzzles. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2229-2238. 2019.
Q. Dou, D. C. de Castro, Kamnitsas, and B. Glocker. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, volume 32, pages 6450-6461, 2019.
C. Fang, Y. Xu, and D. N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. 2013 IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
C. Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML' 17 Proceedings of the 34th International Conference on Machine Learning Volume 70, pages 1126-11 1035, 2017.
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Nomain-adversarit training neural networks. The journal of machine learning research, 17(1):2096200 2030,220002222 2016.
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICIR), 2018.
M. Ghifary, W. B. Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2551-2559 2015.
11

---

Published as a conference paper at ICLR 2022
A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, and B. Schölkopf. Covariate shift and local learning by distribution matching, pages 131-160. MIT Press, Cambridge, MA, USA, 2009a.
A. Gretton, A. Smola, J. Huang, M. Schmittfull, K. Borgwardt, and B. Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009b.
I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. arXiv preprint 2011:20077.01434, 2020.
T. B. Hashimoto, M. Srivastava, H. Namkoong, and P. Liang. Fairmess without demographics in repeated loss minimization. In International Conference on Machine Learning, pages 1929-1938, 2018.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
W. Hu, G. Niu, Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers. In International Conference Machine Learning, pages 2029-2037, 20-00037 2018.
W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? International Conference on Machine Learning, pages 2029-2037. PMLR, 2018.
G. Huang. Z. Liu, L. van der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-2269, 2017.
Z. Huang, H. Wang, E. P. Xing, and D. Huang. Self-challenging improves cross-domain generalization. In European Conference on Computer Vision, pages 124-140, 2020.
M. Ilse, J. Tomczak, and P. Forré. Selecting data augmentation for simulating interventions. In ICML 2021: 38th International Conference Machine Learning, pages 4555 4562, 2021.
P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, S. Beery, J. Leskovec, A. Kundaje, E. Pierson, S. Levine, C. Finn, and Liang
Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:20/2.07421, 2020..
M. Koyama and S. Yamaguchi. ditt----iistibution generalization with maximal invariant predictor.
In arXiv e-prints, 2021.
D. Li, Y. Yang, Y.-Z. Song, and M. Hospedales. Deeper, broader and artier domain generalization. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 5543-555 2017.
D. Li, Y. Yang. Y.-Z. Song, and T. Hospedales. Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018a.
D. Li, J. Zhang, Y. Yang, C. Liu. Y.-Z. Song, and T. Hospedales. Episodic training for domain generalization. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1446-1455. 2019.
D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales. Sequential learning for domain generalization. In European Conference on Computer Vision, pages 603-619, 2020.
H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5400-5409 2018b.
Y. Li, X. Tian, M. Gong, Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference Computer Vision (ECCV), pages 647-663, 2018.
D. Lopez-Paz Paz and M. Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, volume 30, pages 6467-6476. 2017.
12

---

Published as a conference paper at ICLR 2022
L. Matthey, I. Higgins, D. Hassabis, and A. Lerchner. dsprites: Disentanglement testing sprites dataset. https://github..omm hotpp//ssoooeeemmmm/pppstttsssse dataset// 2017.
Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto. Unified deep supervised domain adaptation and generalization. In 2017 IEEE International Conference Computer Vision (ICCV), pages 5716-5726, 2017.
A. Nichol, J. Achiam, and J. Schulman. On first-order meta-learning algorithms. arXiv: Learning. 2018.
G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Schölkopf. Learning explanations that hard to vary. In ICLR 2021: The Ninth International Conference on Learning Representations. 2021.
X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. Moment matching for multi-source domain adaptation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 1406-1415, 2019.
M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pages 4333, 2018.
M. Rojas-Carulla B. Schölkopf, R. Turner, and J. Peters. A causal perspective on domain adaptation. star, 1050:19, 2015.
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.0877 2019.
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: /911:89911.0731 2019.
V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
S. Sco, Y. Suh, D. Kim, G. Kim, Han, and B. Han. Learning to optimize domain specific normalization for domain generalization. In European Conference Computer Vision, pages 68-83, 2019.
Stock and M. Cisse. Convnets and imagenet beyond accuracy: Explanations, bias detection, adversarial examples and model criticism. arXiv preprini 171.:17111.14433 2017.
B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 450- Springer, 2016.
A. Tarvainen and H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In ICLR (Workshop), 2017.
R. Tatman. Gender and dialect bias in youtube's 's automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 53-59, 2017.
Thrun and L. Pratt, editors. Learning to Learn. Kluwer Academic Publishers, USA, 1998. ISBN 0792380479.
A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '11, page 1521-1528, USA, 2011. IEEE Computer Society. ISBN 9781457703942. doi: 10.1 2091
URL
https://doi.org// www.:/..11..........
L. van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (186))2579-260) 2008
H. Venkateswara, J. Eusebio. S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5385-5394, 2017.
13

---

Published as a conference paper at ICLR 2022
R. Volpi and V. Murino. Addressing model vulnerability distributional shifts over image trans- formation sets. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7980-7989, 2019.
H. Wang, S. Ge, Z. C. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, volume 32, pages 10506-10518, 2019.
S. Yan, H. Song, Li, L. Zou, and L. Ren. Improve unsupervised domain adaptation with mixup training, arXiv preprint arXiv:2001.00677, 2020.
K. Zhou, Y. Yang, T. M. Hospedales, and T. Xiang. Learning to generate novel domains for domain generalization. In European Conference on Computer Vision, pages 561-578, 2020.
14

---

Published as a conference paper at ICLR 2022
A TAYLOR EXPANSION OF REPTILE AND FISH INNER-LOOP UPDATE
In this section we provide proof to Theorem 3.1 We reproduce and adapt the proof from Nichol et al. (2018) in the context of Fish, for completeness.
demonstrate that when the inner-loop learning rate a small, the direction of Gy aligns with that of Gg. where
G 16 (0- [0] as G.
(5)
Gg = 06000
(6)
Expanding G Gg is the gradient of maximizing the gradient inner product (GIP).
2
a
--- S(S- - 1,jES
G. G.GGG G3
00
Expanding Gy To write out Gj, we need to derive the gradient update of Fish, 0 8. Let us first define some notations.
For each inner-loop with S steps of gradient updates, we assume loss functions i as well sequence of inputs [d: where d; := {ra: yb 1 denotes a minibatch at step i randomly drawn from one of the available domains in (DI,... Ds). For reasons that will become clearer later, take extra note that the subscript here denotes the index of step, rather than the index of domain. We also define the following:
Ol((x,,):0..
9i Ed;
(gradient step wrt 0:)
(8)
00,
0,+1 0,
agi
(sequence of parameters)
(9)
El((I,, 0)
91 Ed.
(gradient at step i, wrt 01)
(10)
00,
(0I(((, 01)
H; Ed,
(Hessian at initial point)
(11)
007
In the following analysis omit the expectation F., and input (ar, y) to and instead denote the loss at step i as 4. Performing second-order Taylor approximation to gi yields:
g. = ((()))
(12)
= (0,) + (0:) - 01) + -:
(13)
=0(a²)
Di + H&(0; - 01) + 0
(14)
D: H.H
Jj + Oaa²).
(15)
Applying first-order Taylor approximation to gj gives
9 = 9, O(a),
(16)
plugging this back to Equation (15) yields:
g 9. all, Σ 93 O
(17)
For simplicity reason, let us consider performing two steps in inner-loop updates, i.e. S 2. We can then write the gradient of Fish 0
0 = a [91 + (2)
(18)
= 0r 91 0g + 92) a² H2g +0(a3).
(19)
15

---

Published as a conference paper at ICLR 2022
Furthermore, taking the expectation of f - 0 under minibatch sampling gives (assuming indepen- dence between 91 and 92)
E1, 1 + 92] = G1 + 22
E1
[221 - E1 1192
(interchanging indices)
2 F H2291 + H192]
(averaging last two eqs)
Mgg 92)
E122
2
00,
1
D
G2)
2
001
Note that the only reason steps in the inner loop rather than index of domains. Plugging ①. 2 in Equation (19) yields:
interchange the indices in 2 is because the subscripts represent
a² D G2) + O(a²)
0 - 0] (G1 + G2) 2 00,
(20)
We can also expand this the general case where S2
8
itj
(G; Gj)
a G ==1
+ O(a²).
(21)
5(
00,
JJES
The second term in Equation i5) G, which the full gradient of ERM defined as follow:
8
=
G..
(22)
Plugging Equation (21) and Equation (22) to Equation yields
G = E - 0 asG
(23)
a
0 G, Gj
(24)
S(S 1)
00,
J.ES
Comparing Equation (7) to Equation (24) we have:
G Gg
lim
1.
G
A.1
FISH AND REPTILE: DIFFERENCES AND CONNECTIONS
As we introduced, our algorithm Fish is inspired by Reptile, a model agnostic meta-learning (MAML) algorithm.
Meta-lcarning aims at reducing the sample complexity of new, unseen tasks. A popular school of thinking in meta-learning MAML, first proposed Fini al. (2017); (ndrryhowwc et al. (2016). The key idea is to backpropagate through gradient descent itself to learn representations that can be easily adapted unseen tasks. There are close connections between meta-learning (Thrun and Pratt 1998) and (multi-source) domain adaptation. In fact, there a few works in domain generalization that are inspired by the learnne principles, such as Li et al. (2018a); Balaji et al. (2018); Li et al. (2019; Dou et al. (2019). In Ren et al. (2018), also the leveraging of gradient inner product in meta- meta-leerning, where is used to determine the importance weight of training examples.
Even though meta learning and domain generalization both study N-way, K-shot problems, there are some distinct differences that set them apart. The most prominent one is that in meta learning, some examples in the test dataset will be made available test time (K 0), while in domain
16

---

Published as a conference paper at ICLR 2022
Algorithm 3 Black fonts denote steps used in both algorithms, colored fonts are steps unique to
Fish or Reptile.
1: for i ... do
2: 0-0
3: Sample task Dj 1D,,... Dr}
4:
for s ... ..) } or Dr {D1, ,r) do
5:
Sample batch d, De
6:
9: =
7: Update 0 -- 0 09t
8: end for
9: Update 0 + + c(O 0)
10: end for
Peptte
Fan
Toplic
conois
Number
(a) Train
(b) Test
Figure 6: Performance CDSPRITES-N, with N E [5,
generalization no example in the test dataset is seen by the model (K 0); another important difference that while domain generalization aims to train models that perform well on an unseen distribution of the same task, meta-learning assumes multiple tasks and requires the model to quickly learn an unseen task using only K examples.
Due to these differences, it does not make sense in general to MAML framework in domain generalization. As it turns out however, the idea of aligning gradients to improve generalization is relevant both methods- The fundamental difference here that MAML algorithms such as Reptile aligns the gradients between batches from the same task Nichol et al. (0188 while Fish aligns those between batches from differen/ tasks.
To see how this is ahiceved, let us have a look at the algorithmic comparison between Fish (blue) and Reptile (green) [Algorithm As we can see, the key difference between the algorithm of Fish and Reptile is that Reptile performs its inner-loop using minibatches from the same task, while Fish uses minibatches from different tasks (14-8). Based on the analysis in Nichol et al. (2018) (which we reproduce in Appendix A), this is why Reptile maximizes the within-task gradient inner products and Fish maximizes the across-task gradient inner products.
A natural question to ask here is how does this affect their empirical performance? In Figure 6 we show the train and test performance Fish (blue) and Reptile (green) on CDSPRITES-N, We can see that despite the algorithmic similarity between Fish and Reptile, the two methods behave very differently on this domain generalization task: while Fish's test accuracy goes to 100% at N 10, Reptile's test performance is always 50% regardless of N. Moreover, we observe dip in Reptile's 's training performance early on, with the accuracy plateaus at 56% when N > 20. Reptile's poor performance this dataset is to be expected since its inner-loop is designed encourage within-domain generalization, which is not helpful for learning what's 's invariant across domains.
A.2 WHEN =
When we set the meta learning rate of Fish e = 1, Fish reduces to ERM with a data-loader that samples from only domain in each minibatch. Note that with this change, the inner-loop step S now represents the total training iterations, and as a result it increases from a constant value (S 10 in our experiments) orders of magnitude larger (up to 10 for some datasets that use). We
17

---

Published as a conference paper at ICLR 2022
demonstrate here that for this reason, the effect of maximizing inter-domain gradient inner product is significantly less prominent when =
To this, let revisit Equation (21) where demonstrate that the expectation to Fish's 's update E 0 be written the sum of the following three terms via Taylor series expansion,
a
a(G.
E|0 0
Σ
G
+ O(aa))
(25)
5(S 1)
00,
1,jES
(1)
(3)
where (1) is the sum of gradients over S steps of SGD updates (equivalent to ERM), (2) the gradient inner product over S steps of gradients and (3) is the higher order terms of Taylor series expansion.
For both Fish and Reptile, the higher order terms are ignored under two conditions: 1) constant/non- infinite inner-loop step S and 2) small learning rate However, when 1, inner-loop step S oo. As result, the higher order terms cannot be ignored, and we can no longer conclude that gradient inner product plays an important rule in the model's updates.
B SMOOTHFISH: A MORE GENERAL ALGORITHM
B.1 DERIVATION
We conclude in Appendix A that a component of Fish's update G Ei0 a G is in the same direction as the gradient of GIP, Gg. It therefore possible to have explicit control over the scaling of the GIP component in Fish, similar to the original IDGM objective, by writing the following:
Gam = aS G (B10 0 as. a.S.c..
(26)
By introducing the scaling term 7, we have better control on how much the objective focus on inner product average gradient. Note that recovers the original Fish gradient, and when y 0 the gradient Gsmm is equivalent to ERM's gradient with learning rate aS. We name the resulting algorithm SmoothFish. See Algorrihh 4
B.2 RESULTS
We rum experiments on datasets in WILDS using SmoothFish, with y ranging in 0.2, 0.5, 0.8, 1, 10). Note that when y 0, SmoothFish is equivalent ERM and when it is equivalent to Fish. See results in Figure 7 The other hyperparameters including a, meta steps, used here the same the ones used in main experiments section.
[0, 0.2.
18

---

Published as a conference paper at ICLR 2022
Algorithm 4 Smoothed version of Fish, which allows to get approximate gradients for the general form of Equation (4)
1: for iterations = 1,2, do
2: 0-0 0
3: for D; E permutel ( .. ;D)).,
4: Sample batch dj D)
Øl((x, y);
5:
Ed,
/rar wrt
00
al((x, 0):
6: Di = Ed,
//rad wrt 0
00
7:
Update 0 1 agi
8:
end for
5
9i Ism = aSg + 7 (0-0) 0) aSg
9:
g
10:
Update 0 +- + cJsm
11: end for
Val
72
Vbl
60-
02-
Procision (%6
<c-
00
32-
20-
12-
35
601
0.1
0.2
0.5
0.8
3.1
0.
(a) CAMELYON17
(b) CIMIECONMENTS
0.50
0.c
Precisin
0.45
Precision
0.40
Test
Test
02
0.5
(c) FMoW
(d) POVERTY
Figure 7: Results on WILDS using SmoothFish with y ranging from to
C DISCUSSIONS AND RESULTS ON WILDS
We provide a more detailed summary of each dataset Table 5
Some entries in # Domains omitted because the domains in each split overlap. Note that in this paper we report the results on WILDS vl the benchmark has been updated since with slightly different dataset splits. We currently working updating results to v2 of WILDS.
19

---

Published as a conference paper at ICLR 2022
Table Details of the WILDS datasets we experimented on.
Domains (it domiins)
Data
Target
# Examples
Domains
FMoW
Time (16), Regions (5)
Sanelitee images
and classes)
76,863
19.915
22,108
11.-
3,-
POVIRTY
Countries (25). Urbanvraml (2)
Sanellite images
Asset (real valued)
10,000
4,000
4,000
13.-
5,-
5,-
CAMELYON17
Hospitals (5)
Thaaue slides
Tumur (2 classes)
302.436
34,904
85,054
Civin COMMENTS
Denogrpphes (8)
Online comments
Toricity classes)
269,038
45,180
133,782
WILDCAMM0000
Trap locatices,
Photos
Animal species (186
142,200
20.784
AMAZON
Reviewers (7.676)
Product reviews
rating classes)
1,000.124
100,050
100,000
Table 6: Results on POVERTYMAP- WILDS.
Table 7: Results on CAMELYON
Method
Val. Worst-U/Rr
Test Worst-U/R
Method
Val. Accuracy (%)
Test Accuracy (%)
Fish
0.47 (=00))
0.30 (#a0r)
Fish
83.9 (=1.)
74.7 (=2.)
IRM
0.53 (2905)
0.43 (2:00)
ERM
84.9 (=3/)
70.3 (=64)
ERM
0.80 (1.or)
0.78 (10.0)
ERM (ours)
84.1 (12.4)
70.5 (1/2.)
ERM (ours)
0.48 (=a7)
0.29 (#am)
IRM
86.2 (=14)
64.2 (=x1)
Coral
0.51 (#906)
0.45 (20.0)
Coral
86.2 (#14)
59.5 (#77)
C.I MAP-R.YMA
Task: Asset index prediction (real-valued). Domains: 23 countries
The task is predict the real-valued asset wealth index of an area, given its satellite imagery. Since the number of domains considered here is large (23 countries), instead of looping over all S domains in each inner-loop, we sample N << S domains in each iteration and perform inner-loop updates using minibatches from these domains only to speed up computation. For this dataset we choose N by hyper-parameter search.
Worst-U/R Pearson Correlation (r). Following the practice in WILDS benchmark, we compare the results by computing the worst region Pearson correlation (r) between the predicted and ground-truth asset index over 3 random seed runs.
Evalutaion:
Results: We train the model using ResNet- (He et al. 2016) backbone. See Table
We see that Fish obtains the highest test performance, with the same validation performance as the best baseline. The performance is more stable between validation and test, and the standard deviation is smaller than for the baselines. We also report the results of ERM models trained in our environment as "ERM (ours)", which shows similar performance to the canonical results reported in the WILDS benchmark itself ("ERM").
C.2 CAMELYON WWWLSDS
Task: Tumor detection (2 classes). Domains: hospitals
The HOMELYON1-WWILE dataset contains 450,000 lymph-node scans from 5 hospitals. Due to the size of the dataset, instead of training with Fish from scratch, we pre-train the model with ERM using the recommended aaperpparaeetes in Koh et al.] (2020), and fine-tune with Fish. For this dataset, we find that Fish performs the best when starting from a pretrained model that has not yet converged, achieving much higher accuracy than the ERM model. we provide an ablation study on this in Appendix E
Evaluation: Average accuracy. We evaluate the average accuracy of this binary classification task. Following Koh et al. (2020), we show the mean and standard deviation of results over 10 random seeds runs. The number of random seeds required here is greater than other WILDS datasets due to the large variance observed in results. Note that these random seeds not only applied during the fine-tuning stage, but also to the pretrained models to ensure a fair comparison.
Following the practice in WILDS, we adopt DenseNet-121's (Huang al. 2017) architecture for models trained on this dataset. See results in Table 7
Results:
The results show that Fish significantly outperforms all baselines its test accuracy surpasses the best performing baseline by 6%. Also note that for all other baselines, there is a large gap between validation and test accuracy (11% 27%). This is because WILDS chose the hospital that is the most
20

---

Published as a conference paper at ICLR 2022
difficult to generalize to the test split make the task more challenging. Surprisingly, observe in Table 7 the discrepancy between test and validation accuracy of Fish is quite small (3%). The fact that it is able to achieve similar level of accuracy the worst-performing domain further demonstrates that Fish does not rely on dopain-specific information, and instead makes predictions using the invariant features across domains.
To demonstrate that Fish's strong performance on CAMELYON 7 selected test set is not merely coincidental, randomly chose ways of assigning the 5 domains train/test/val (3/1/1 domains) splits. See results collected over 10 random seeds in Table 8
Notably, these different splits of CAMELYON17 7 Fish still average outperforms ERM (with only 2 exceptions), demonstrating that Fish can generally achieve good performance on the Camelyon17 dataset.
Table 8: Results CAMELYON WILLS with shuffled splits.
Train/Val/Test ID
Val, ERM (%)
Val, Fish (%)
Test, ERM (%)
Test, Fish (%)
012/3/4
71.3 (=6.3
71.9 (#577)
80.9 (#86)
88.9 (#6.9)
124/0/3
72.3 (#5.))
74.1 (+4.3
63.4 (#66)
65.9 (4:19)
234/0/1
83.9 (#2.)
84.1 (+2.3)
75.6 (42.8
73.9 (4:15)
034/1/2 (original)
84.3 (=2.))
82.5 (1:12)
73.3 (190)
79.5 (16.0)
Table 10: Results on CIVI.COMMENTS- -WOMMEENSSWIIDS.
Table 9: Results on FMOW-WILDS.
Val. Accuracy (%)
Test Accuracy (%)
Method
Val. Accurncy (%)
Test Accuracy (%)
Average
Worst
Average
Method
Fish
57.8 (=019)
49.5
51.8
34.6
Average
Worst
Average
Worst
ERM
59.5 (_a.r)
48.9 (1nez
53.0 (Loss)
32.3 (L:25)
Fish
(LoA) 70.5 (L
(9.4
(LAx)
ERM (ours)
59.9 (=a22)
47.1 (#x21)
52.9 (saus)
30.9 (a23))
Group DRO
89.6 (2:3) 68.7 (a10)
(99.))
70.4 (#27)
IRM
57.4 (Lar))
47.5 (1xs5)
50.8 (Loes
30.0 (12s)
Reweighted
(Las)
67.9 (L73)
88.9 (104)
(197)
Coral
(=025)
47.1 (a04s)
50.5 (s226)
31.7 (6224)
ERM
92.3 (12a)
53.6 (396))
92.2 (196)
(112)
ERM (ours)
(#05)
54.1 (+04)
92.5 (+0J)
58.1 (#17)
C.3 FMOW-WILDS
Task: Infrastructure classification (62 classes). Domains: 80 (16 years 5 regions)
Similar to CAMELYON 7- WILDS, since the number of domains is large, we sample = 5 domains for each inner-loop. To speed up computation, with Fish; different from Appendix C.2, we find the best-performing models are acquired when using converged pretrained models (see details in Appendix EJ.
also use pretrained ERM model and fine-tune
Evaluation: Average & worst-region accuracies. Following WILDS, the average accuracy evaluates the model's ability to generalize over years, and the worst-region accuracy measures the model's performance across regions under time shift. We report results using 3 random seeds.
Results: Following Koh al. (2020),
DenseNet-121 pretrained on ImageNet for this dataset. Results in Table 9 show that Fish has the highest worst-region accuracy on both test and validation sets. ranks second in terms of average accuracy, right after ERM. Again, Fish's performance is notably stable with the smallest standard deviation across all metrics compared to baselines.
C.4 COVI.COMMENTS. WII.SS
Task: Toxicity detection in online comments (2 classes). Domains: 8 demographic identities.
The COVILCOMENTS- -ILLD contains 450,000 comments collected from online articles, each annotated for toxicity and the mentioning of demographic identities. Again, we use ERM pre-trained model speed up computation, and sample N = 5 domains for each inner-loop.
Evaluation: Worst Worst-group accuracy. To study the bias of annotating comments that mentions demographic groups as toxic, the WILDSS benchmark proposes to evaluate the model's performance
21

---

Published as a conference paper at ICLR 2022
by doing the following: 1) Further separate each of the 8 demographic identities into 2 groups by toxicity - for example, separate black into black, toxic and black, not toxic; 2) measure the accuracies of these x 2 = 16 groups and use the lowest accuracy as the final evaluation of the model. This metric is equívalent to computing the sensitivity and specificity of the classifier on each demographic identity, and reporting the worse of the two metrics over all domains. Good performance the group with the worst accuracy implies that the model does not tend to use demographic identity as an indicator of toxic comments.
Again, following Koh et al. 2020) we report results of 3 random seed runs.
Results: We compare results the baselines used in the WILDS benchmark over random seed runs in Table 10 All models are trained using DistilBERT (Sanh et al. 2019).
The results show that Fish outperforms the best baseline by 4% and 7% on the test and validation set's worst-group accuracy respectively, and is competitive in terms of average accuracy with ERM (within standard deviation). The strong performance of Fish on worst-group accuracy suggests that the model relies the least on demographic identity as an indicator of toxic comments compared to other baselines. ERM, on the other hand, has the highest average accuracy and the lowest worst-group accuracy. This indicates that it achieves good average performance by leveraging the spurious correlation between toxic comments and the mention of certain demographic groups.
Note that different from all other datasets in WILDS that focus pure domain generalization (i.e, no overlap between domains train and test splits), COVLEENMNTTS- WINSSS is subpopulation shift problem, where the domains in test are subpopulation of the domains in train. As a result, the baseline models used in WILDS for this dataset are different from the methods used in all other datasets, and are tailored to avoiding systematic failure data from minority subpopulations. Fish works well in this setting too without any changes or special sampling strategies (such as * and + in Table TOD. This further demonstrates the good performance of algorithm different domain generalization scenarios.
C.5 IWILDCAM-WII WILDS
Task: Animal species (186 classes). Domains: 324 camera locations.
The dataset consists of over 200,000 photos of animal in the wild, using stationary cameras across 324 locations. Classifying animal species from these heat or motion-activated photos is especially challenging: methods can easily rely on the background information of photos from the same camera setup. Fish models pretrained with ERM till convergence, and for each inner loop sample from N = 10 domains.
Evaluation: Macro Fl score. Across the 186 class labels, report average accuracy and both weighted and macro F1 scores (F1-w and F1-m, respectively, in Table 11). We run 3 random seeds for each model.
Results: All models reported in Table 7 are trained using a ResNet-50. We find Fish to outperform both metrics over the
baselines on both test accuracy and weighted F1, with a 1% improvement best performing model (ERM). However, this comes at the cost of lower macro Fl score, where Fish performs 1% worse than ERM models that we trained and 3% than the ERM reported in WILDS. This suggests that Fish is less good at classifying rarer species, however the overall accuracy on the test dataset improved.
Although Fish did not outperform the ERM baseline on the primary evaluation metric proposed in Koh et al. (2020), we found the improvement of Fish in both accuracy and weighted F1 be robust across a range of hyperparameters. See more details on this in Appendix D
C.6 AMAZON-WILDS
Task: Sentiment analysis (5 classes). Domains: 7,676 Amazon reviewers.
The dataset contains 1.4 million customer reviews on Amazon from 7,676 customers, and the task is to predict the score (1-5 stars) given the review. Similarly, pretrained the model with ERM
22

---

Published as a conference paper at ICLR 2022
till convergence, and due to the large number of domains (S 5008 in train) we sample N reviewers for each inner loop.
10th percentile accuracy. Reporting the accuracy of the 10th percentile reviewer helps us assess whether the model performance is consistent across different reviewers. The results in Table are reported over 3 random seeds.
Evaluation:
The model trained using DISTILBERT (Sanh et al. 2019) backbone. While Fish has lower average accuracy compared to ERM, its 10th percentile accuracy matches that of ERM, outperforming all other baselines.
Results:
Table 11: Results on IWILDCAM- WILDS.
Method
(%)
Avg (%)
Test Macmo (%)
Avg (%)
Fish
40.3 (#0.)
73.8 (#az)
(+1.8)
64.7 (=26)
GroupDRO
37.5 (=1.7)
71.6 (#27)
23. (=2.1)
72.7 (=20)
ERM
47.0 (#24)
75.7 (#ed)
(1.0(
71.6 (=2.)
Coral
43.5 (+15)
73.7 (#04)
32.8 (=0.)
73.3 (=4,7)
IRM
22.4 (277)
59.9 (3x
15.1 (1:99
59.8 (2177
Table 12: Results on AMAZON-WILDS.
Accuracy (%)
Accurcy (%)
Method
Average
10-th per.
Average
10-th
Fish
72.5 (=00)
(=9.)
(=ao) 53.3 (=0a)
ERM
72.7
58.2
71.9 70.5 (Ler)
5388 (204)
IRM
71.5 (Lai) 54.2 (1ox)
52.4 (1ox)
Reweighted
69.1
52.1
68.6
52.0 (2.0
D HYPERPARAMETERS
In Table 13 we list the hyperparameters we used to train ERM. The same hyperparameters were used for producing ERM baseline results and as pretrained models for Fish. In val. metric we report the metric on validation set that is used for model selection, and in cut-off we specify when to stop training when using ERM to generate pretrained models.
Table 13: Hyperparameters for ERM. We follow the hyperparameters used in WILDS benchmark. Note that we did not use a pretrained model for POVERTY, therefore its cut-off condition is not reported.
Dataset
Model
Learning rate
Batch size
Weight decay
Optimizer
Val. metric
Cut-off
CAMELYON17
Densenet- 121
lc-3
SGD
acc. avg.
iter
COMMCOMMENTS
DistilBERT
le-5
Adam
acc. wg.
Best metric
FMOW
Densenet- 121
lc-4
64
Adam
uvg.
Best val. metric
IWILDCAM
Resnet-50
lc-4
16
0
Adam
F1-macro (all)
Best val. metric
POVERTY
Resnet- 18
1e-3
64
Adam
Worst-U/R Pearson (r) 10th percentile acc.
AMAZON
DistilBERT
2c-6
8
0.01
Adam
In Table 14 we list out the hyperparameters we used to train Fish. Note that we train Fish using the same model, batch size, val metric and optimizer ERM these not listed in Table 14 to avoid repetitions. Weight decay is always set as 0.
E ABLATION STUDIES ON PRE-TRAINED MODELS
In this section we perform ablation study on the convergence of pretrained ERM models. Note that the pretraining is done the same domain generalization dataset that Fish is trained on later, not ImageNet. We study the performance Fish with the following three configurations of pretrained ERM models:
23

---

Published as a conference paper at ICLR 2022
Table 14: Hyperparameters for Fish.
Dataset
Group by
a
# domains
Meta steps
CAMELYON17 17
Hospitals
le-3 0.01
3
3
CIVILCOMMENTS
Demographics x toxicity
lc-5 0.05
16
5
FMOW
time regions
lc-4 0.01
5
WWILDCAM
Trap Incations
le-3 324
le-4
10
POVERTY
Countries
5
AMAZON
Reviewers
2c-6 0.01
7,676
5
1. Model is trained on 10% of the data (epoch 1);
2. Model is trained 50% of the data (epoch 1);
3. Model at convergence.
By comparing the results between these three settings, we demonstrate how the level of convergence affects the Fish's ' training performance. See results in Table 13 Note that POvERTY is excluded here because the dataset is small enough that are able to train Fish from scratch.
Table 15: Ablation study pretrained ERM models.
Model
FMow
CAMELYONI7
IWILDCAM
CIVILCOMMENTS
Test Avg Ace
Test Avg Acc
Test Macro FI
Test Worst Ace
10%
21.7 (#25)
79.1 (s122)
13.7 (sa3)
71.8
50% data
31.0 (#24)
(=123)
(9.0))
74.2
Converged
32.7 (#12)
(182)
23.7 (209)
73.8 (+1x)
We see that CIVI.COMMENTS sustain good performance using pretrained models different convergence levels. FMoW and IWILDCAM on the other hand seems to have strong preference towards converged model, and the results worsen the amount of data seen during training goes down. CAMELYON17 17 achieves the best performance when only 10% of data is seen, and the test accuracy decreases while training with models with higher level of convergence.
F ABLATION STUDIES ON HYPERPARAMETERS
and c We study the effect of changing Fish's inner loop learning rate and outer loop learning rate c. To make the comparisons more meaningful, we keep a constant while changing their respective values. See results in Figure 8
Meta steps N For most the datasets we studied (all apart from CAMELYON17 where T = 3) we sample N-sized subset of all T domains available for training (see Table 14 for T of each dataset). Here we study when N = 5, 10, 20.
Table 16: Ablation study on meta steps N.
FMoW
POVERTY
IWILDCAM
COMMENTSENNTS
N
Test Avg Ace
Test Pearson
Test Macro Fl
Test Worst Acc
33.0 (116)
80.3 (+17)
23.7 (=09)
74.3 (+15)
10
32.7 (+12)
80.0 (#08)
23.7 (=05)
73.4 (#10)
20
33.3 (12.)
77.7 (12.))
23.7 (109)
(12.9)
In general altering these hyperparameters don't have huge impact the model's performance, however it does seem thet when N = 20 the performance on some datasets (POVERTY and CIVIL- COMMENTS) degrade slightly.
24

---

Published as a conference paper at ICLR 2022
2owtsfer Lird
2200.02
20202
10:0
300
300 CONO
a CAMBLYON17
(b) CIMIECOMMENTS
(c) FMOW
0
Soan 194
220
23025404 3642.00
194
(6123e 8843 1964
(d) IWILDCAM
(e) POVERTY
Figure 8: Ablation studies on a and e. Note that a x e remains constant in all experiments, and the midpoint of each plot is the hyperparameter we chose use to report our experiment results.
eller
Fal, DIES befonn
Fab.
TIHM beforn
Credeet bsee prodect
BBM, befom
STAR
BRM, after
Bersdooe
(a) CDSPRITES-N
(b) CAMELYON17
(c) CIVILCOMMENTS
SON, Inbir KIAN oRKE
B104 Infore
ERM,
E1D4 after
EIM aber
Berolota
(d) FMOW
(e) POVERTY
IWILDCAM
Figure 9: Gradient inner product values during the training for CDSPRITES-N (N=15) and 5 different
WILDS datasets.
G TRACKING GRADIENT INNER PRODUCT
In Figure 9 we demonstrate the progression of inter-domain gradient inner products during training using different objectives. We train both Fish (blue) and ERM (yellow) untill convergence while recording the normalized gradient inner products (i.c. cosine similarity) between minibatches from different domains used each lnner-loop. The gradient inner products are computed both before (dotted) and after (solid) the model's update. To ensure a fair comparison, use the exact same sequence data for Fish and ERM (see Appendix I for more details).
Inevitably. the gradient inner product trends differently for each dataset since the data. types of domain splits and the choice of architecture are very different. In fact, the plot for CDSPRITES- N and POVERTY significantly different from others, with dip in gradient inner product at the beginning
25

---

Published as a conference paper at ICLR 2022
of training this is because these the two datasets that train from scratch. On all other datasets, the gradient inner products are recorded when fine-tuning with Fish.
Despite their differences, there some important commonalities between these plots: if we compare the pre-update (dotted) to post-update (solid) curves, we can see that ERM updates often result in the decrease of gradient inner product, while for Fish it can either increase significantly (Figure 9c Figure 9e) or at least stay at the same level (Figure 9a Figure 9b] Figure result of this, can that the post-update gradient inner product of Fish is always greater than ERM at convergence.
Figure 9tp. As
The observations here shows that Fish is effective in increasing/mainnaiiing the level of inter-domain gradient inner product and sheds some lights on its efficiency on the datasets we studied.
H SUMMARY OF DOMAINBED
DOMAINBED is a testbed for domain generalization that implements consistent experimental proto- cols across SOTA methods ensure fair comparison. contains 7 popular domain generalization
datasets, including Colored MNIST (Arjovsky et al. 2019), Rotated MNIST (Ghifary et al. 2013).
VLCS (Fang et al.,| (013). PACS et al. 2017). OfficeHome (Venkateswara et al. [011 Terra Incognita (Beery et al. 2018) and DomainNet (Peng et al., 2019), and offers comparison to variety of SOTA domain generalization methods, including IRM (Arjovsky al.] 2019), Group DRO (Hu
et al. 2018 [Sagawa et al. 2019). Mixup (Yan et al. (020), MLDG (Li et al. (0T8a)). Coral (Sun and
Saenko 2016). MMD (Li et al. 2018b), DANN (Ganin et al. 2016) and CDANN (Li et al. 2018
I ALGORITHM FOR TRACKING GRADIENT INNER PRODUCT
To make sure that the gradients we record for ERM and Fish are comparable, we use the same sequence of S-minibatches to train both algorithms. See Algorithm 6 for details.
Algorithm 5 Function GIP
1: function GIP (d1. d2, ... dv3.0) 0):
2: for d,, E {dj, d2. dN} } do
3: In = al(d, 0/0
4: end for
5: 9 3(S-1) Zijes
6: return
J T-SNE VISUALISATION OF LEARNED REPRESENTATIONS
In this section we visualize the representation learned by Fish and ERM with t-SNE (van der Maaten and Hinton 2008) for PACS, VLCS and OfficeHome datasets. See Figure 10
We can see that Fish and ERM are both capable of forming distinctive label-clusters PACS (first row) and VL.CS (second row), however with ERM we can observe within each label cluster that sub-clusters of domains are forming. This is particularly the case for the the red and yellow cluster in Figure 10a and cyan cluster in Figure 10cl The domain clustering phenomena is not observed for Fish. On the other hand, for the OfficeHome dataset where Fish outperforms ERM by more than 1%. we clearly see that Fish exhibits better label clustering performance than ERM.
26

---

Published as a conference paper at ICLR 2022
Algorithm Algorithm of collecting gradient inner product g for Fish and ERM both before and
after updates. See GIP in Algorithm 5.
Initialize Fish f + 8, ERM 0g 8
2: for 1, ... do
3: //Get
minibatehes
4: for DD, E (D1, D2,
DN} do
5: Sample batch d, Dn.
6: end for
7: //GracInnerProd before update
8: 9F SIP((d1,d2..
(.)
9: DE = GIP ((d1 6.
Fish training
11:
O - 0f
12:
for d, d2: dN) do
In = 0l ; /00
Update 0 0 0gn
15:
end for
16:
0y + 0f + <(0 0,)
17:
//Rarrange minibatches
18:
d= chuufeee(ccnn ... ,xx))
19:
dx) split(d)
//RM training
21:
for dn E (dl, d2. ,dx) do
22:
In = : 0.0
23:
Update 0g +- 0g agn
24:
end for
25:
//GradInnerProd after update 26: OF. GIP (d1, d2, dN }. 0f)
27: gEa GIP (di, d2, ... dy w. 0g)
28:
end for
29:
Return Trh, FF. gEb gEa
27

---

Published as a conference paper at ICLR 2022
(a) PACS, ERM
(b) PACS, FISH
(c) VLCS, ERM
(d) VLCS, FISH
ERM, OFFICEHOME
(f) FISH, OFFICEHOME
Figure 10: t-SNE plot for PACS, VLCS and OfficeHome. Colors represent labels, markers (shape of each datapoint) represent domain.
28


Reviews:
[{"review_id": "hVmHwudptbg", "reviewer": "Reviewer_KYNz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is well written and is easy to follow\n- Multiple benchmarks are used to evaluate the method \n\nConcerns:\n- the claim that feature learnt are domain invariant is not really backed by a theoretical explanation or empirical. One way would be to show using tsne plot, that indeed features align as the gradient have same sign.\n- Algorithm 1 formulation looks similar to the Mean Teacher[1] formulation where \\tilde{\\theta} plays the role of student and \\theta as of teacher. \n- Regarding just empirical results Fish seems competitive but it is not clear in what cases it the best. Is just the low number of source domain it shines? If that's the case, more explanation is needed to highlight the reason.\n- Related Work section is needs to be improved, some works with which it is compared isn't discussed, for eg. Coral. \n- Why the data augmentation based methods are left out in the comparisons?\n\n[1] https://arxiv.org/abs/1703.01780", "summary_of_the_paper": "The work tries to tackle the problem of domain generalisation in multi-source setting. The main claim of the paper is that by maximising inner product  between gradients from different domains leads to better learning of domain invariant features. The provide a meta-learning inspired algorithm Fish to approximate the second-order derivates. The results on several domain generalisation dataset is shown. ", "summary_of_the_review": "The paper presents gradient alignment to induce feature invariance. Several comparisons are presented but paper lacks explanation on the central idea, that why invariance is achieved. Also, some clarifications mentioned in the main review are need. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fP95-B7MwJH", "reviewer": "Reviewer_b5oJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper address an important problem in OOD generalization. The paper is well-written and the contributions are stated clearly. I have a few questions and concerns.\n\nPositive Points:\n\n1. The motivation and methodology is stated clearly. I can understand the approach by only read the paper once. I like figure 1 and section 3.2, which make the formulation itself very clear.\n\n2. Despite the simplicity of Fish, it achieves non-trivial improvement on WILDS benchmark, outperforming important baselines such as IRM. It good to see some approaches finally outperform ERM by a large margin, e.g., CAMELYON17.\n\n3. I like the extensive analysis in the experimental section, which provide valuable insights for future study.\n\nQuestions:\n\n1. I think objective (4) shares similar motivation to IRM in the sense that we want the model to update in directions (use features) that are beneficial for all domains. It would be great to provide more discussion and comparison w.r.t. IRM.\n\n2. I wonder whether the inner product of gradient in equation (4) could be related to bi-level / meta learning? For instance, check equation (12) in [1]. One may be able to explain the success of IDGM from the perspective of optimization, and further relates it to IRM. In particular, IRM defines its objective by measuring optimality, which is highly related to optimization. Note that this is different from analyzing approximation of IDGM such as Thm 3.1.\n\n3. The measure drawback of the paper is it lacks of rigorous analysis about why and how IDGM works. For instance, IRM performs a careful study to analyze the proposed objective from a causal perspective. One may also investigate the problem from information theory [2]. It would be great if the paper can have at least a few paragraphs for analyzing IDGM.\n\nOverall, there is no significant flaw in this paper. Although the proposed approach is not well analyzed, the empirical results are still impressive. \n\n[1] Ren et al., Learning to Reweight Examples for Robust Deep Learning, ICML 2018\n\n[2] Huszár, Invariant Risk Minimization: An Information Theoretic View, 2019", "summary_of_the_paper": "To improve the generalization in unseen domains, the paper proposes a regularizer based on align ing the gradient direction between training environments. The proposed approach achieve non-trivial improvement on bench marks such as WILDS.", "summary_of_the_review": "Overall, I think this is a well-written paper with good empirical results. I did not observe obvious flaw in the paper.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WdFre7VZAiK", "reviewer": "Reviewer_t4Z6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \n\n1) Authors motivate a proposed method clearly with their illustration in figure 2 and explanation in section 3.3\n\n2) Contribution is a novel combination of existing ideas. Authors worked out the Taylor series expansion of the gradient and came up with a linear approximation. Authors modified the Reptile algorithm which works on improving the GIP inside a particular task (or domain) to improve the GIP across different tasks (or domains)\n\n3) By the virtue of their simplification, their model update process is much cleaner than it would have been had they used IDGM itself\n\n4) This linear approximation also helps to not worry about saving intermediate gradients of all the past steps as might be the case in Reptile.\n\nWeakness: \n\n1) It is not clear if the improvements given by the proposed method are statistically significant. It is also not clear why this method performs better in certain datasets and worse on others. \n\n2) Missing literature. \n\n3) Comparing Fish and IDGM might be necessary to confirm that the approximation works.\n\nReview:\n\n1) How is the concept of using gradient inner product different from using kernel methods? Can authors compare and contrast with a popular DG method [1]? \n\n2) Is there any way one can check or compare - how well the linear approximation is working? If given infinite resources, can authors improve their scores further? If authors can show this on at least a few dataset, it will really help to understand the method better. \n\n3) iWildCam is a much tougher dataset to work with where there are pictures of deers in grass (for ex) which is present throughout all the images. Similar vegetation is also present around other wild animals in other pictures. This aspect might have confused this approach significantly as it might think vegetation to be an invariant feature. This is where proposed method doesn’t perform too well. Can authors comment why?\n\n4) Gradient inner products are also related to Neural Tangent Kernels [2]. Can authors comment more on this?\n\n[1] Blanchard, Gilles, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. \"Domain Generalization by Marginal Transfer Learning.\" arXiv preprint arXiv:1711.07910 (2017) J. Mach. Learn. Res. (JMLR) 22 (2021): 2-1.\n\n[2] Jacot, Arthur, Franck Gabriel, and Clément Hongler. \"Neural tangent kernel: Convergence and generalization in neural networks.\" arXiv preprint arXiv:1806.07572 (2018).\n\n", "summary_of_the_paper": "Authors propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Authors also give a computationally efficient optimization for the proposed method and give results on various datasets. ", "summary_of_the_review": "Literature could be improved. For better understanding slow but exact algorithm could be tested on small dataset. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DOlZ_ySlx8q", "reviewer": "Reviewer_Xwsw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. This paper proposes to maximize the similarity of gradients of the classifcation loss for different domains to learn invariant features. This is an interesting way to address the domain generalization problem.\n\n2. This method also provides a effcient way to approximate the inner product between gradients due to the large computation cost. A theoretical proof is also given for this approximation.\n\nCons:\n\n1. The improvment is minor on DomainBed in Table 3. The results of baseline methods seem to be inconsistent with the results by facebook?  \nhttps://github.com/facebookresearch/DomainBed/blob/main/domainbed/results/2020_10_06_7df6f06/results.png  Since this paper is claiming that it is trying to obtain domain-invariant features which is also what  DANN wants to achieve, then why DANN is worse than ERM while the IDGM is better than ERM? It would be better to have an explanation for this observation and full comparison with DANN.\n\n2. It would be better to visualize (e.g., t-sne) the learned representations by the IDGM to see if the features are invariant across domains. \n\n\nUpdate:\n\nI have read the response by authors and other reviews. Part of my concern is addressed. So I raised my score to 6.\nBut I still share the same concern with another reviewer that it is unclear why this method works. If some domain-specific information is useful according to the response, then why the gradient matching helps preserves such information?", "summary_of_the_paper": "This paper proposes to maximize the similarity of the gradients of the classification loss for different domains to learn domain-agnostic features.", "summary_of_the_review": "This paper proposes to maximize the gradient similarity to learn invariant features across domains. But the improvment is minor and some critical explanations to the results are lacking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4aMEkyzQtiZ", "reviewer": "Reviewer_eig7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) The theorem of FISH looks interesting, which can avoid costly second-order computations.\n\n(2) The pitfall of ERM is interesting to explain why ERM fail in the domain generalization.  \n\n(3) This paper is well written and easy to follow.\n\n(4) Experiments on the WILDs and DomainBed demonstrate the advantages of the proposed methods in a practical.\n\nCons:\n \n(1) The motivation for using gradient matching is not clear.  e.g., Why using gradient matching can achieve domain-invariant representations?  \n\n(2) Lack of comparative experiments of direct optimization of IDGM on the DomainBed.\n\n(3) Some sentences are not accurately described. For example: In the related work 5. Gradient alignment: \"While they achieve the best performance on the training set, both IGA and ERM could completely fail when generalizing to unseen domains.\"  Not all ERM algorithms fail [1][2].\n\n(4)The structure of the inner-loop and outer-loop is similar to MAML or Reptile. Need more explanation and the difference of MAML.\n\n\n\nReference:\n\n[1] Gulrajani et al 2020. In search of lost domain generalization. \n\n[2] Cha, Junbum, et al. \"Swad: Domain generalization by seeking flat minima.\" arXiv preprint arXiv:2102.08604 4 (2021).\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper proposed inter-domain gradient matching for domain generalization. They also approximated the proposed model with a simple first-order algorithm to avoid costly second-order computations.  The performance on the WILDs and DomainBed seems better than the ERM algorithm. ", "summary_of_the_review": "As stated in the main reviews, the motivation of using a matching gradient is not particularly impressive, but overall the paper does provide some interesting analysis that may encourage new ways of thinking about domain generalization problems. As such, I think this paper may be of interest to the ICLR community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for domain generalization. The main idea is to encourage higher inner-product between gradients from different domains. Instead of adding an explicit regularizer to encourage this, authors propose an optimization algorithm called Fish which implicitly encourages higher inner-product between gradients of different domains. Authors further show their proposed method is competitive on challenging benchmarks such as WILDS and DomainBed.

Reviewers all found the proposed algorithm novel and expressed that the contributions of the paper in terms of improving domain generalization is significant. A major issue that came up during the discussion period was that we realized that the presented results on WILDS benchmark are misleading. In particular, the following statements in the manuscript are false because on "CivilComments" and "Amazon", Fish utilizes a BERT model (Devlin et al., 2018). However, other methods at WILDS benchmark use DistilBERT (Sanh et al., 2019):

- Section 4.2: "For hyper-parameters including learning rate, batch size, choice of optimizer and model architecture, we follow the exact configuration as reported in the WILDS benchmark. Importantly, we also use the same model selection strategy used in WILDS to ensure a fair comparison."

- Appendix C2: "Results: We compare results to the baselines used in the WILDS benchmark over 3 random seed runs in Table 10. All models are trained using BERT (Devlin et al., 2018)."

Authors explained that the mismatch is because at the time they evaluated their model, an earlier version of WILDS benchmark was available but they later updated other methods' results on a newer version of WILDS benchmark. Of course, I do not think that this explanation makes the misleading statements OK. Authors promised to do the following for the camera-ready version to make sure it is not misleading:

- Using "Worst-U/R Pearson r" as the comparison measure for "PovertyMap"
- Submitting their method to WILDS benchmark making sure everything matches the baselines and then reporting the results on "Amazon" and "CivilComments" datasets.

Therefore, I recommend acceptance and I hope that authors would stick to their promise and update the manuscript to include these changes.
Decision: Accept (Poster)