Loss Function Learning for Domain Generalization by Implicit Gradient
Authors: Boyan Gao
Henry Gouk
Yongxin Yang
Timothy Hospedales
Venue: ICLR 2022 Submitted (2022)
Keywords: meta-learning
loss function learning
Domain Generalisation

Abstract: Generalising robustly to distribution shift is a major challenge that is pervasive across most real-world applications of machine learning. A recent study highlighted that many advanced algorithms proposed to tackle such domain generalisation (DG) fail to outperform a properly tuned empirical risk minimisation (ERM) baseline. We take a different approach, and explore the impact of the ERM loss function on out-of-domain generalisation. In particular, we introduce a novel meta-learning approach to loss function search based on implicit gradient. This enables us to discover a general purpose parametric loss function that provides a drop-in replacement for cross-entropy. Our loss can be used in standard training pipelines to efficiently train robust models using any neural architecture on new datasets. The results show that it clearly surpasses cross-entropy, enables simple ERM to outperform significantly more complicated prior DG methods, and provides state-of-the-art performance across a variety of DG benchmarks. Furthermore, unlike most existing DG approaches, our setup applies to the most practical setting of single-source domain generalisation, on which we show significant improvement.

Under review as a conference paper at ICLR 2022
Loss FUNCTION LEARNING FOR DOMAIN GENERAL- IZATION BY IMPLICIT GRADIENT
Anonymous authors Paper under double-blind review
ABSTRACT
Generalising robustly to distribution shift is major challenge that is pervasive across most real-world applications of machine learning. A recent study highlighted that many advanced algorithms proposed to tackle such domain generalisation (DG) fail outperform a properly tuned empirical risk minimisation (ERM) baseline. We take a different approach, and explore the impact of the ERM loss function on out-of-domain generalisation. In particular, we introduce a novel meta-learming approach to loss function search based implicit gradient. This enables us to discover a general purpose parametric loss function that provides a drop-in replacement for cross-entropy. Our loss can be used in standard training pipelines to efficiently train robust models using any neural architecture on new datasets. The results show that it clearly surpasses cross-entropy, enables simple ERM outperform some more complicated prior DG methods, and provides state-of-the-ard performance across a variety of DG benchmarks. Furthermore, unlike most existing DG approaches, our setup applies to the most practical setting of singlc-source domain generalisation, on which we show significant improvement.
INTRODUCTION
Deep learning highly successful when the training and testing samples meet the i.i.d. assumption. However, this assumption is violated in many practical applications of machine learning from medical imaging to earth observation imaging (Koh et al.] (021). This has led large number of studies to investigate approaches to training models with increased robustness to distribution shift at testing- time, a problem setting known as Domain Generalisation (DG). Despite the volume of research in this area (Zhou et al.) 2021a), a recent careful benchmarking exercise, DomainBed (Gulrajani & Lopez-Paz] (022) showed that simple empirical risk minimisation (ERM) on a combination of training domains is a very strong baseline when properly tuned. State-of-the-art alternatives based on sophisticated architectures, regularisers, and data augmentation schemes failed to reliably beat ERM (Gulrajani& & Lopez-Paz 2021).
Rather than propose an alternative to ERM for DG, we investigate a previously unstudied hyper- parameter of ERM, namely the choice of loss sunction-whicl has been ubiquitously taken to be standard cross-entropy (CE) in prior DG work. Loss function choice has been shown to impact calibration Muukhoti et al. (2020), overfitting (Gonzalez & Miikkulainen 2019). and label-noise robustness (Wang et al.) 2019) in standard supervised learning, so it is intuitive that it would impact robustness domain-shift. However, it has not yet been studied in this context. Our preliminary experiments showed that equipping ERM with some recent robust loss functions in place of CE does lead to improvements in DG performance where sophisticated alternatives have failed (Gulrajani & Lopez-Paz 2021). This raises the question: can one design a loss function specialised for DG?
To answer this question, we define meta-learning algorithm to learn a parametric (white-box) loss function suitable for DG. Our desiderata are: (1) Performing ERM with this loss on source domain should lead to good performance when tested on out-of-domain target data; and (2) It should provide ppdu--and-play drop-in replacement for cross-entropy that, once learned, can be used without further modification or computational expense with any new dataset or model architecture. While there has been growing interest in meta-learning for loss function design (Li et al.) 2019a), they mostly fail to meet these criteria. They learn applied to DG, this would imply replacing simple ERM learning with sophisticated meta-learning
caan--speiii-ratttiie cetccf than re-usable losses.

---

Under review as a conference paper at ICLR 2022
pipelines to train a loss on per-problem basis. In contrast, our discovered loss provides drop in replacement for CE that leads standard training pipelines to produce more robust models.
To train general purpose robust loss function we need a search space that is flexible enough to include interesting new losses, but simple enough to generalise across tasks without overfitting to the problem used for loss learning. We choose 12-dimensional space of fourth order Taylor polynomials. Furthermore, we need a loss that is suitable for all stages of training. This precludes the majority of approaches based on online meta-learning which update the loss and base model iteratively (Li ct al. 20 9a c). and also suffer from short-horizon bias (Wu et al. 2018). Evolutionary methods (Gonzalez & Mikkullanenn 2019) and reinforeementllamming (Li et al. 2019a) could support loss learning in principle, but are too slow to be feasible. Therefore we develop the first -mplicit-rradient based approach to loss learning. This allows us to tractably compute meta-grade veta-gradients of the target recognition performance with respect to the loss used for training in the source domain.
We use a simple DG task (RotatedMNIST) to train our robust loss, termed Implicit Taylor Loss (ITL), to replace CE in ERM. Subsequent experiments show that ERM with ITTL surpasses CE across a range of DG benchmarks, and leads to state of the art performance, despite being much simpler and faster than competitor DG methods. While the majority of existing DG methods require multiple source domains to conduct data augmentation or feature alignment strategies, ITL improves single-source domain generalisation, crucial problem setting which has been minimally studied thus far.
To summarise our contributions: (i) We provide the first study on the significance of supervised loss function choice in DG (ii) We demonstrate the first efficient solution to loss-leamnnn based on meta grardeegasentss computed by the Implicit Function Theorem. (iii) Empirically, we show that our learned ITL loss enhances simple ERM and achieves state of the DG performance across a range of benchmarks, including the challenging single-source DG scenario.
RELATED WORK
Domain Generalisation aims to learn a model using data from one or more source domains, but with the further requirement that it is robust to testing on novel target domain data-without accessing target data during training. DG is now a well studied (Zhou et al.) 202 area with diverse approaches including data augmentation (Shankar et al. training algorithms such as domain alignment objectives (Li et al.
Domain Generalisation:
2018 Zhou et al. (021b), robust
(018b)) and other regularisers (Li 2019c Balaji et al. 2018). Most DG studies have assumed the multi- source setting, which enables new data-augmentationn strategies (Zhou et al. 2021b), and allows ceneraliistioon-rrootingg design features to be tuned by domain-win wise cross-valin cross-validationn validation. In particular, few studies (Li et al.. 2019c Balaji et al.] 2018) have considered meta-learning based DG, where a regulariser applied in a training domain is tuned by meta-grad gerta-gradients from the resulting validation-domant performance. The resulting model is then deployed to the true target domain within the same family. These methods require regulariser meta-learning for each given multi-source DG problem family. In contrast, we propose to learn a simple loss function once, which then provides drop-in replacement for CE in any single- or multi-source DG problem. A recent criticism of the DG literature showed that no method consistently outperformed well tuned ERM baseline on the carefully designed BemainBed benchmark (Gulrajani & Lopez-Paz 2021). Rather than competing with FRM, we simply enhance the ERM loss function and this leads to a clear improvement on DomainBed.
Loss Function learning: Loss Function learning aims to discover new losses that improve model optimisation from various perspectives including conventional generalisation performance, (Gonzalez
& Müikkulainen 2019; Liu et al. 202T), optimisation efficiency (Li et al., 2019a; Gonzalez &
Mükkulainen 2onggWang et al. [2200] Bechtle al. 2020). and noise robustness (Li al. 2019a).
Key dichotomies are in the search space of black box (ncural) (Bechtle et al. 2020 Li et al. 2019c)
vs white-box Chuman-readable (Li et al. 2019a Gonzalez & Miikkulainen 2019 Wang et al. 2020)
losses; whether learned losses are problem specific (Li et al. 2019a| Wang et al. 2020)
reusable
Gonzalez & Miikkulainen] 2019); the meta-optimisation algorithm used evolution (Liu et al. 2021
Gonzalez & Miikkulainen 2019), RL (Li et al. 2019a Wang et al. |020 Bechtle et al. 2020), or
gradient (Li et al.] 2019c); and whether the loss is updated offline (Luu et al.
2021
Gonzalez &
Mikkulainen 2019) (long inner loop, typically intractable), or online (Li et al.
2019a
Wang et al.
2020> (short inner Toop, efficient but suffers from chort-horizon bias (Wu et al.
2018)). No studies
have thus far investigated loss learning for domain-shift robustness. In order to learn reusable
2

---

Under review as a conference paper at ICLR 2022
robust loss white-box loss search space of Taylor polynomials proposed in Gonzalez & Mikkullainen (2020), and offline/long inner loop meta-learning, To make this meta- vetaa-opimisstion tractable, we exploit the Implicit Function Theorem, to efficiently generate accurate hypergradients of the validation domain performance with respect to the training domain loss function parameters. Besides being the first demonstration of loss learning for DG, to knowledge it is also the first. demonstration of any implicit muddeent-based loss learning.
3
METHOD
The need for Domain Generalisation arises when one is using machine learning to build model where the available training data is not representative of the data that will be observed by the model once it has been deployed. In domains, P, from which particular, it is assumed that there is an underlying distribution over
sample several source domain distributions, {p1-) ..... Pu P), to make use of during training. We can construct a training set for each of these source domain distributions by sampling K data points, D.
{(x) (#:j) (a:))) Pi 33=1+ and the union of M/
(5) )
all these sets as the full training set, D(s) = Uh_1 D Empirical Risk Minimisation (ERM) then simply finds the model parameters, 0, that minimise the loss measured on this training set,
K clo((?")
Σ
min 0
(1)
where -(--)) is a loss function (typically cross entropy) measuring how well the predicted labels match the ground truth labels. One can empirically check the resulting model's robustness to domain shift by sampling one or more target domain distributions, {p} ..... ....P P}. from the same distribution over domains that was used to generate the training data. Data can then be sampled for each of these target domains, yielding such as accuracy can then be computed using this data.
dataset ( U D."). Standard evaluation metrics
3.1
LOSSES FOR DG
Our goal is to replace the standard CE loss typically used ERM with a learned loss function. We are motivated by recent work showing that learned losses can enable models to perform better for variety of other problem settings, such as training with label noise (Wang et al. 2019) and improving calibration (Mukhoti et al. 2020). We formulate the task of learning the parameters, w, of loss function, Cu.. as a bilevel optimisation problem. The outer objective is to find the w that maximises the performance of model evaluated on the target domain data, and the inner problem is to train a model to minimise the value of L. measured on the source domain data. The loss parameters optimised using gradient-based methods that take advantage of the implicit function theorem to efficiently compute gradients for the outer optimisation problem. Crucially, once the optimal loss function has been found, new DG problems can be solved via ERM on the Lux loss.
The bilevel optimisation we use to formalise the meta-leaminng process is given by
K
w arg min
1
M(Jo((w)
(x)))) (25))
K
i=
K
s.t. (w(w)= = arg min
L. CL ((J))
(3)
K
2=1
where M is loss function used measure the performance of the model on the target domains, typically chosen to be cross entropy.
Optimising w is challenging due to the need to backpropagate through the long inner loop optimi- sation of 6. Existing approaches for learning loss functions typically resort to slow evolutionary reinforcement learning updates (Li et al. 2019a; Gonzalez & Miikkulainen, 2019 Wang et al. 2020) in the outer loop, or to an online approximation based on alternating steps on w and (Li et al. 2019a Wang al. 2020). The latter approach leads to loss function w" that cannot be transferred
3

---

Under review as a conference paper at ICLR 2022
to tasks, as it suffers from short-horizer bias (Wu et al. 2018). To solve this problem, use the Implicit Function Theorem (IFT) to compute problem to approximate 0 (w).
without truncating the inner optimisation &w
3.2 IMPLICIT GRADIENT
The conceptually simplest way to optimise w is to store all the intermediate iterates generated by the optimiser when training the network in the inner loop, and then backpropagate through all of these weight updates (Maclaurin et al. 2015). This becomes prohibitively expensive in both memory and computation. Instead, after finding H (w) we compute the gradient using the Implicit Function
Theorem (IFT). The implicit gradient computation takes advantage of the fact that acw. O, because a8 we have found locally optimal model parameters for the inner problem. The gradient we want to compute given by
0M
AM 00
(4)
dw
00 Ow
w.0
and the IFT can be used to obtain
00
82
x
(5)
dw
20
1AT
00
O.
|0]x|0|
10|x|u
The inverse the Hessian can be rephrased in terms Neumann series,
02
8 & 00 0gT P.
lim 1-+00
00 067
and approximated by truncating the summation finite number of terms. In practice, can make use of vector-Jacobian products to avoid explicitly constructing the Hessian in the summation. Further details can be found in Lorraine et al. (2020), but provide pseudo-code for computing the implicit gradient in Algorithm
Algorithm IFT-based loss learning for DG.
Algorithm 2 Computing the hyper- gradient of the meta-objective M, with respect to the loss w. The grad(....)) function from PyTorch computes Jacobian-vector prod- uet when called with a non-scalar first argument. Inspired by Lor raine et al. (2020), we use this to efficiently compute the Hessian re- quired for approximating the Neu- D: (Construct mann series.
1: Input: P.
2: Output: wt
3: Init w
4: while not converged or reached max steps do
5: sample P1.--.74 Pr. from P
6: sample D,,... Dr from p1,... Pn.
7: Init H =0 E gn*||a]
8: for all D; do
9: Init 6; (Get random network weights) D (D,, Dr Di, D
source/target splits) splitss
Input: Cw. M, (w, 6"),
0: arg ming (0,, (Train the network)
Output: - 82 Euu
12:
h Hypergradient(C.. M, (w, 0)),
3.4
H Hi, h
for all (w.0-)
do
14: end for
grad 80 020)
0c
h Grad-surgery((1
16:
1yh UUpdtte the loss function)
p --= v end for
17:
end while
3.3 ROBUST GRADIENT ESTIMATION
Algorithm I summarizes the gradient estimation procedure. To obtain high quality gradient estimates in each outer loop iteration, employ wewe--on-ddominn.o strategy. The DG task, P, used for meta-traine training the loss parameters has m domains associated with it. In each iteration of the outer loop, train m networks with the prospective loss (i.e., instantiate different copies of the
4

---

Under review as a conference paper at ICLR 2022
Meta-
Meta-
Train
Source
Inner Loop
test
Source
Inner Loop
Deploy Loss fur
arg min
0' arg min
0"
c
ang min (()
fo
Guitar
Target
Outer Loop
Target
Algorithm schematic. Loss Lu is trained to optimize held-out domain performance on R-MNIST and then deployed on novel datasets.
Figure 1:
inner loop), where each network has different target domain and the remainder of the domains are used train the network. We can then compute gradient for each of the m networks and aggregate them together in order to perform an update to the loss parameters. Rather than using the mean gradient, we found aggregation using gradient surgery (Yu et al. 2020), which reduces the gradient noise caused by different source/target domain splits in the inner loop, to work better in practice.
3.4 TAYLOR POLYNOMIAL REPRESENTATION
The choice of loss function parameterisation a crucial factor our framework. One must balance ability to represent sufficiently broad range of loss functions, with the susceptibility to overfitting the data used to learn the loss. The search space we consider is based on the truncated Taylor polynomials used by the evolutionary optimisation loss learning approach of (Gonzalez & Miikkulanenn 2020). This family of loss functions treats the point around which the Taylor polynomial is centred, and also the value of the derivatives this point, learnable parameters. In this sense, it is variational learning method-though it should be stressed it is not variational Bayesian method. The family of order multivariate Taylor polynomials has the form
((z) = 1 V"((c)"((-c")
where e is a fixed point around which function is being expanded. Because e is fixed, the values of the derivative this point are also fixed. As such, we can replace and V with meta-learnable parameters. This allows to parameterize the learned loss function in terms of the gradients it should have a meta-learnable point. We define our learnable loss
1 Σ yS). C
(9, y)
((ll("))))))" n!
(wo, wi))", (8)
where each V''(( (wo, wil actually be replaced by introducing more meta-parameters w. Please see Appendix A.4 for an expanded definition of this loss function.
3.5 ALGORITHM SUMMARY
Given set of training domains, the loss function search space in Section 3.4 and efficient update strategy in Section 3.2 and Algorithm we are able to train a robust loss function f... We conduct such loss function learning only once using a small dataset, and then evaluate the resulting loss on a variety of larger datasets that are unseen during meta- veta-training. Meta-test: Given the learned loss function Cur, we fix it and use together with ERM algorithm for novel DG tasks. Each target problem is trained from scratch and has not been seen during loss learning. An overview the algorithm, and the learning curve of the meta -taii phase, are given in Figure
Meta-train:
EXPERIMENTS
4.1 DATASET AND IMPLEMENTATION DETAILS.
Meta-train stage: We aim to learn general purpose loss function that can be used diverse DG problems, but first need select dataset for initial loss learning. We chose RotatedMNIST (Ghi-
5

---

Under review as a conference paper at ICLR 2022
Table Resnet18 Cross-domain recognition accuracy (%) on PACS.
Target set
Art
Cartoon
Photo
Sketch
Avg.
(Li Epi-FCR
al.
2019b
82.1
77.0
93.9
73.0
81.5
JiGen Carlucci et 21T9
79,4
75.3
96.0
71.6
80.5
MASF Dou
80.3
77.2
95.0
71.7
81.0
CrussGrad (Shankar cl
2018
79.8
76.8
96.0
70.2
80.7
Entropy Zhao al.
2020
80.7
76.4
96.7
71.8
81.4
L2A-OT Zhou
20,20
83.3
78.2
96.2
73.6
82.8
RSC (reported Huang
2020
83.43
80.31
95.99
80.85
85.15
Mixstyle (rs) (Zhou et al.
202 lb
82.3 + 0.2
79.0 = 0.3
96.3 + 0.3
73.8 + 0.9
82.8
Mixstyle (dl) (Zhou al. 2021b
84.1 + 0.4
78.8 = 0.4
96.1 + 0.3
75.9 0,9
83.7
RSC (our rerun their code)
79.25 0.69
77.63 = 0.50
93.61 + 0.37
78.11 1.40 1.40
81.91
RSC ITL
81.67 +: 0.80
76.56 0.50
95.57 + 0.22
77.05 0.56
82.71
ERM BCE
0.81
70.82 = 0.29
93.11 + 0.78
57.65 + 0.98
73.19
ERM CE
76,9 0.6
76.5 = 0.7
93.3 + 0.1
68.8 + 0.6
78.9
ITL-Net (ERM+ITL)
83.9 + 0.4
78.9 = 0.6
94.8 + 0.2
80.1 + 0.6
84.4
fary et al., 2015 which contains six different domains that are all derived from MNIST (LeCun Cortes 2010) but with different rotations: 0%, 15%, 30%, 45%, 60%, and 75%. The leave- ont strategy for robust implicit gradient estimation, therefore, results in six inner loop instantiations for each outer loop iteration. For efficiency, we use 2-layer MLPs as the base model, which contains 1024-256-10 units from the input layer to the output one with ReLU the activation function. The learning rates in the inner loop and outer loop are both 0.01, a batch size of 32 is used for the inner loop, and the Neumann series used for approximating the inverse Hessian is truncated 15 iterations. The result is a set of 12 parameters that define the learned fourth-order polynomial loss function and an example of learned loss is shown in Appendix A.3 The meta-train compute cost descripted in Appendix A.6
We now evaluate our learned ITL. using ERM, but with our learned loss instead of cross entropy. Models trained with our method are denoted by -TeL-Net. We evaluate the learned loss function on the four common DG benchmarks: VLCS (Fang al. 2013), PACS (Li al.] 201 (a), OfficeHome (Venkateswara et al. 2017), and Terra Incognita (Beery et al. 2018). Two sets experiments are conducted: (i) We evaluate the conventional PACS benchmark, as the most widely used in the DG literature, and enables comparison against the most recent state-of-the-art competitors. (ii) We evaluate all four benchmarks using the recent DomainBed platform, which is designed to enforce fair and consistent hyperparameter tuning across different methods.
Meta-test (deployment) stage:
4.2 RESULTS
PACS: Setup A pre-trained ResNet18 backbone is used throughout, together with the source and target domain split described in (Li et al. 2018a). We train ResNet-18 with ITL on the training split and perform model selection using the validation set. We same set of hyperparameters (learning rate of 0.001, weight decay of 0,00001, 1, and batch size of 32 for each domain) for the baseline ERM with Cross-Entropy (ERM +CB) and Binary Cross-Entropy (BCE) train our model. We compare ITL-Net with the existing sate-of-the-at methods on this benchmark, including RSC (Huang et al.)
2020), data augmentation based L2A-OT (Zhou et al. 20216). Mixstyle (Zhou et al. (020) including random shuffle (rs) and domain label (dl), regulariser based Entropy (Zhao et al. 2020). adversarial
gradient-based CrossGrad (Shankar et al. 2018), meta learning-based MASF DDu et al., 2019) and Epi-FCR (Li et al. 2019b) and self-supervision-basse JiGen (Carlucci et al. 2019).
PACS: Results We first conduct experiments using the classic PACS protocol to facilitate com- parison against many recent competitors that were not evaluated on DomainBed. Table compares our ITL-Net performance Vs state-of-the-an methods. From the results we can see that: (i) Simply swapping the loss in ERM from CE to ITL, leads to significant 5.5% improvement. (ii) Overall our ITL--ee leads to state-of-the-aa performance on this benchmark, surpassing the most recent and sophisticated competitors such as Mixstyle.
DomainBed: Setup We next evaluate ITL using the DomainBed platform, which enforces careful and fair evaluation by ensuring that all competitors use the same hyper-parameter tuning strategy

---

Under review as a conference paper at ICLR 2022
Table 2: DomainBed Cross-domain recognition accuracy (%) with ResNet50 on ColoredMNIIT
VLCS, PACS, Terralncognita, Offe and DomainNet. Bottom: Results of Wilcoxon signed- rank hypothesis test comparing TTL.-Net against competitors.
Models
Dataset
ERM
SagNet
CORAL
CDANN
RSC
ITL-Net
CoLOTedMNNIS
51.5
51.7
51.5
51.7
51.7
52.0
VLCS
77.5
77.8
78.8
77.5
77.1
78.9
PACS
85.5
86.3
86.2
82.6
85.2
86.4
Terralncoginita
46.1
48.6
47.6
45,8
46.6
51.0
OfficeHome
66.5
68.1
68.7
65.8
65.5
69.3
DomainNet
40.9
40.3
41.5
38.3
38.9
41.6
Avg. Rank
4.17
2.67
3.00
5.17
5.00
1.00
p-value (Ho)
Reject (0.016)
Reject (0.016)
Reject (0.016)
Reject (0.016)
Reject (0.016)
Table 3: Cross-domain recognition accuracy (%) on DomainBed-PACS--Renae Comparison with alternative manually-designed robust losses.
Loss
ERM+CE
ERM+FOCAL
ERM+SCE
ERM+GCE
EMR+LS
ERM+ITL
Avg Perf 83.9 + 0.5
84.6 + 0.8
84.2 0.5
83.0 0.2
84.9 + 0.6 86.4 + 0.5
(random search, driven by source domain validation performance), and the same number hyperpa- rameter search iterations. We follow the standard DomainBed protocol and use ResNet-50, with experiments conducted on VLCCS, PACS, OfficeHome, and Terra Incognita.
DomainBed: Results
The results in Table Tableaacmmarr ITL--eet with ERM and some of the most competitive published alternatives: SagNet (Nam et al. 2019), CORAL (Sun & Saenko (016)..
CDANN (Li al. 2018c) and RSC (Huang al. (020) in the original DomainBed paper (Gulrajani Lopez- Paz 2021p. A detailed comparison is given in Appendix 6 The conclusion of the DomainBed study was that existing methods did not reliably beat ERM under this hyperparameter tuming protocol. In contrast, we can see that ITT---ett provides clear improvement on ERM and matches improves on the strongest existing competitor each case, especially on Terralncognita and OfficeHome. To formally compare ITTL-Net with competitors, we perform significance testing using the Wilcoxon signed-rank test, where the p-value is set as 0.025 and the sample size is number the Domain datasets applied. For example, when comparing the performance of ITL-Net and ERM, the null hypothesis (H₂) is that ITL-Net has equal performance to ERM on DomainBed and the alternative hypothesis (H₂) that ITL-Net is statistically significantly better than ERM DomainBed. The results of the hypothesis tests comparing ITL to each competitor are summarized at the bottom of Table 2 and confirm that ITLL-Net outperforms them all.
Single Source DG: Setup Most existing DG methods rely on the availability of multiple source
domains some form: For example synthesise new domains for data augmentation (Zhou et 2021b), or perform feature alignment among training domains (Sun & Saenko 2016). A unique Teature of ITL-Net is that, since is only a small modification to ERM, can be used to learn on single source domain. Although this setting is not well explored in the literature, it is obviously highly practical as multiple source domains are often not available in practice. To explore this setting, we modify the DomainBed benchmark to train on single source at a time and average over each source-xtarget combination, rather than training on the conjunction all sources.
Single Source DG: Results From the results in Table we can see that performance drops across the board compared to multi-source training (Table 2 as expected. However, the state of the art alternatives CORAL and SagNet are no longer competitive compared to ERM as they were in the multi-source case (Table 2 this is expected as they are designed to exploit cues from multiple source domains. In contrast, our ITL-Net maintains clear lead over the conventional ERM with cross entropy baseline in this setting. This is a significant achievement as existing work has not produced algorithms that improve robustness under the single-source setting.
7

---

Under review as a conference paper at ICLR 2022
Table 4: Cross-domain recognition accuracy (%) DomainBed with single source domain. The heading of the table denotes the single source domain, and results average across all target domains.
Source Dataset
VLCCS
PACS
OfficeHome
Terralncognita
ERM
64.08
51.85
53.57
32.13
CORAL
64.07
51.84
53.51
32.13
SagNet
61.78
53.00
51.30
33,93
Mixup
59.01
54.92
52.70
30.80
ITL-Net
62.17
56.54
55.04
35.09
Classifled Entropy
Misclassifled Entropy
1.6
1.4
SCE
L5
1.4
SCE
LS
1.2
Entropy
Entropy
1.0-
0.8
0.6
0.8-
0.4
0.6
0.4-
0.2
0
12
18 24 30 36 42 48 54 60
6
18 Training iteration x100
12
24
30 36 42 48 54 60
Training iteration x100
Figure 2: The evolution of posterior entropy for target domain test samples during training. Left: Correctly classified samples. Right: Misclassified samples.
4.3 FURTHER ANALYSIS
Qualitative comparison and impact on learning To understand our learned loss, we visually compare it Figure 5 with several other loss functions: CE, SCE (Wang et al.) (2019), Label Smoothing (LS) (Pereyra et al. 2017), and FOCAL (Mukhoti et al. 2020). Compared to the standard cross entropy loss, we can see that ITL has softer penalties for severe misclassification, and stronger penalties for moderate misclassification. In addition, it can be noticed that FOCAL has the most similar shape to our ITL. The properties of the FOCAL loss for probability calibration were recently studied in (Mukhoti et al. 2020). Motivated by this, we report the evolution of the target-domain entropy of correctly and incorrectly classified samples during training in Figure leeft) and Figure[]right) respectively. Clearly, a byproduct of ITL compared to CE and SCE is a (desirable) increase in uncertainty for misclassified instances. FOCAL has similar behaviour to ITL in general, especially the entropy of prediction distributions for misclassified instances.
Quantitative comparison to other robust losses We next investigate whether the good perfor- mance of ITL in cross-domain robustness can be easily replicated by applying existing robust loss functions, or whether our meta-learnin pipeline has learned something new in terms of robust model training. SCE (Wang et al. 2019) and GCE (Zhang & Sabuncu 2018) were designed with label-noise robustness mind, while Focal (Mukhoti et al. 2020 Lin et al. 2017) was designed for class imbalance and calibration. Label-smooth (LS) (Pereyra al. (017)) is for improving generalisation and reducing overconfidence. From the results in Table 3 we can see that while some losses improve on CE, ITL leads the clearest improvement. The somewhat similar behaviour of ITL and FOCAL in terms of induced entropy from Figure 2 does not carry over similar cross doos--dominn recognition accuracy. Note that all experiments in Table 3 were run by us, while competitor performance in Table 2 is taken from (Gulrajani & Lopez-Paz 2021).
Loss landscape analysis and Perturbation analysis One of the factors that affect model general- isation is the loss landscape at convergence. (Keskar et al.) [017 Chaudhari et al.] 2019) observed that flatter loss landscapes lead to good generalisation of the learned model. To this end, we compared a ID slice through the loss landscape of ITL-Net with that of ERM on both source domain and target domains. Namely, we perturb the converged paraeters by moving it around though gradient direction which generated by the evigenvector of the Hessian matrix. From Figure held out target domain, ITL-Nets have flatter loss landscapes compared with models trained by CE with respect the source domains. To further analyse the quality of the minimas provided by CE
that for each
8

---

Under review as a conference paper at ICLR 2022
Table 5: Cross-domain recognition accuracy (%) on OfficeHome: Impact of meta-tran seed (#:
standard deviation), and choice of pre-training dataset.
Target sel
Artistic
Clipart
Product
Real World
Avg.
ITL-NET (RotatedMNIST)
64.22 0.84
56.25 + 0.38
77.52 = 0.32
78.12 0.32
09.03 + 0.18
ITL-NET (RoiatedKMNIST)
64.28
55.84 0.29
76.89 1.04
77.96 0.30
68.74 0.35
Held-ou out deraiin: Art
Held-out domain: Clipart
Held-out domain: Product
Held-out domain: Real-World
56
77
Accuracy
62
64
Accuracy
54
75
76
76
77
74
60
ITL-Net
48
ITL-Net
ITL-Net
ITL-Net
58
56 ERM
46
ERM
71
72
ERM
72-
ERM
0.01
0.03
0.05
0.01
0.05
6.1
6.03
0.05
Gaussian noise
Gaussian
noise std
Gaussian
Figure 3: Perturbation analysis on OfficeHome: ITL-Net vs ERM. Multiplicative Gaussian noise with mean and std: 0.01, 0.05, 0.08 is added to network weights.
and ITL, we follow the perturbation analysis routine (Keskar et al. 2017 Zhang et 2018) by adding multiplicative noise to the weights of the converged models. From the results in Figure 3 can be see that ITL-Nets outperform ERM at every noise ratio.
Held-out
Hejd-out domain: Clipart
Held-out
Held-out domain: Clipart
1.2
17.5
17.5
-0.5
8 0.88
0.6
1.0
ERM
ITL-NET
-4.0
-6.0
ITN
FRM
7.5
10.0
0 10.0
7.5
ER
12:5
10.0
15.0
.9.
-9.0
FRM
12.0
14.0
10.0
-8.0
5.0
5.0
0.2
0.0
-0.4 0.0
-1..0
0.4
0.4
0.8
2.5
2.5
0.0
0.8
0.4 0.0 0,4
0.4
0.8
4.0
Perturbation
Perturbation
Perturbation
Perturbation
Figure 4: ID Loss Landscape: ITT.-Net ERM OfficeHome. Left two: Source domain loss landscape. Right two: Target domain loss landscape.
Repeatability analysis Our message thus far that a single loss produced by our pipeline can be re-used as a alug-and-playy modification to improve vanilla ERM+CE wide variety of held-out downstream DG tasks. That said, one might reasonably wonder about the reliability of the loss function learning procedure itself. To investigate this, repeat entire pipeline including the meta-train stage five times. We then evaluate the consistency of the resulting five loss functions on the downstream ColoredMNIST task. Furthermore, evaluate the dependence of our result on the choice of meta-training dataset, we repeat the above experiment on PotatedKMNIST (Clanuwat el al., 2018) to replace the RotatedMNIST used previously. From the results in Table 5 we can see that performance is quite consistent over trials (small standard deviation). It also differs little with choice of pre-training dataset with both options performing well compared to competitors Table 2
Limitations ITL. improves ERM+CE for DG tasks in general, but in some cases, the margin over SOTA is small, since other state-of-the-ar competitors may beat ERM+CE. It would be more inter- esting if ITL is highly complementary to SOTA methods based on other architectural, augmentation, or domain- cammain-alignment improvements but this remains for future work to determine.
CONCLUSION
We provided the first study of the effect of ERM loss functions on Domain Generalisation. We observe that models trained by ERM with existing robust loss functions can improve performance on Domain Generalisation compared with those trained by Cross-Entropy. To discover the best loss for DG, we perform meta-leaming to find re-usable white-box loss function. This is tractably solved using IFT to obtain gradients of the target domain performance with respect to the source domain loss parameters. This also provides the first demonstration of IFT based loss learning in the literature. The results show that a simple modification to the standard ERM pipeline improves both molti-source and single source DG, and even surpasses the despone state of the art models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain general- ization using reta-regularrvatton. NeurIPS, 2018.
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta-learning via learned loss. ICPR, 2020.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018.
Fabio M Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, (11: 8,, 2019.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. In NeurlPS (Workshop), 2018.
Qi Dou, Daniel C. Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NeurlPS, 2019.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learningg On the utilization of multiple datasets and web images for softening bias. In ICCV, 2013.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In ICCV, 2015.
Santiago Gonzalez and Risto Miikkulainen. Improved training speed, accuracy, and data utilization through loss function optimization. arXiv preprint arXiv:/90s. 1528, 2019.
Santiago Gonzalez and Risto Mükkullanen. Optimizing loss functions through multivariate taylor polynomial parameterization, arXiv preprint arXiv:2002.00059, 2020.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. Computer Exion-ECCY 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part I1 16, pp. 124-140. Springer, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR. 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque. Sara M Beery, Jure Leskovec, An- shul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: benchmark of tn-the-wild distribution shifts. In ICML, 2021.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs: Automl for loss function search. ICCV, 2019a.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 201 8a.
Da Li, Jianshu Zhang, Yongxin Yang. Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, 2019b.
10

---

Under review as a conference paper at ICLR 2022
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018c.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for heteroge-
neous domain generalization. In ICML, 2019c.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Peidong Liu, Gengwei Zhang, Bochao Wang, Hang Xu, Xiaodan Liang, Yong Jiang, and Zhenguo Li. Loss function discovery for object detection via convergence-simulaton driven search. In International Conference on Learning Representations, 2021.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by
implicit differentiation. In AISTATS, 2020.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimiza- tion through reversible learning. In ICML, 2015.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip HS Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. NeurIPS, 2020.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap via style-agnostic networks. arXiv preprint arXiv: 1964. 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. ICLR, 2017.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In ICLR, 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-450) Springer, 2016.
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Xiaobo Wang, Shuo Wang, Cheng Chi, Shifeng Zhang, and Tao Mei. Loss function search for face recogmition. JCML, 2020.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In CVPR, 2019.
Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie- Yan Liu. Learning to teach with dynamic loss functions. In NeurlPS, 2018.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurlPS, 2020.
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NeurlPS, 2018.
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generalization via entropy regularization. NewrIPS, 2020.
11

---

Under review as a conference paper at ICLR 2022
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, 2020.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, 2021a.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021b.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 THE DETAILED RESULTS FOR DOMAINBED
In Table of the main paper, summarized performance across held out target domains for the standard multi-source DG problem. In Table we now give the detailed results of ITLL-Net for each target domain.
Table 6: DemainBed Cross-domain recognition accuracy (%) with ResNet50 on ColOredMNISST VLCS, PACS, Incrainc, OfficeHome and DomainNet.
ColoredMNIST
Target set
+90%
+80%
-90%
Avg.
ERM
71.7 +
72.9 0.2
10.0
51.5
SagNet
71.800
73.0 0.2
10.3 0.0
51.7
CORAL
71.6 0.3
73.1 1: 0.1
9,9
51.5
CDANN
72.0 + 0.3
73.0 + 0.2
10.2 0.1
51.7
RSC
1
72.9 1 0.4
10.2 0.2
51.8
ITL-Net
+
73.4 0.1
11.3 0.7
52.0
Target
Caltech
Labelme
Sun
V-Pascal
Avg.
VLCS
ERM
97.7 + 0.4
61.3 + 0.9
73.4
77.3 + 1.3
77.5
SagNet
97.9 + 0.4
64.5 0.5
1.3
77.5 0.5
77.8
CÓRAL
98.3 + 0.1
66.1 1.2
73.4 73.4+0.33 0.3
77.5
78.8
CDANN
97.3 0.3
65.1 1.1+112 1.2
70.7 +
77.1 77.1+1.5
77.5
RSC
97.9 0.1
62.5 0.7
72.3 1.2
75.6
77.1
ITL-Net
98.3 0.4
65.4 0.7
75.1 0.6
76.8 1.2
78.9
Target set
Art
Cartoon
Photo
Sketch
Avg.
PACS
ERM
84.7 0.4
80.8 0.6
97.2
79.3 + 1.0
85.5
SagNet
87.4 + 1.0
80.7 0.6
97.1 0.1
80.0
86.3
CÓRAL
88.3 133+-00 0.2
80.0 0.5
97.5 +
78.8+1.3
86.2
CDANN
84.6 : 1.8
75.5 4 0.9
96.8 4 0.3
13.5
82.6
RSC
85.4
79.1 0.6
0.5
77.7
84.9
ITL-Net
87.1 0.4
83.3 0.6
96.1 4..4
79.3 1 0.6
Target set
L100
L38
L43
L46
Avg.
TerraIncognia
ERM
49.8 4.4
42.1 1.4
56.9 + 1.8
35.7 + 3.9
46.1
SagNet
53.0 + 2.9
43.0 + 2.5
57.9 + 0.6
1.3
48.6
CORAL
51.6 2.4
42.2 4: 1.0
57.0 1.0
39.8
47.6
CDANN
47.0 +7.0
41.3 + 4.8
4.79
+
45.8
RSC
50.2 2 2.2
39.2 1 1.4
56.3 1.4
40.8 1 0.6
46.6
ITL-Net
3.7
46.2 1.8
58.5
40.9 = 1.8
51.0
OfficeHome
Target set
Artistic
Clipart
Product
Real World
Avg.
FRM
61.3 0.7
52.4 0.3
75.8 +
76.6 + 0.3
SagNet
63.4 +: 0.2
54.8 0.4
75.8 0.4
78,3 0.3
68.1
CORAL
65.3 + 0.4
54.4 0.5
76.5 0.1
78.4
68.7
CDANN
61.0 +:0 1.4
50.4 2.4
7.9
76.6 + 0.8
65.8
RSC
60.7 + 1.4
51.4 0.3
1.1
75.1 + 1.3
65.5
ITL-Net
65.6 0.4
55.6 + 0.4
77.5 0.3
78.6 + 0.4
69.3
DomainNet
Target set
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Avg.
FRM
58.1 = 0.3
18.8 - 0.3
46.7 + 0.3
+.3
12.2 + 0.4
59.6 + 0.1
49.8 + 0.4
40.9
SagNet
0.3
19.0=
4
12.7 0.5
58.1 0.5
48.8
40.3
CORAL
59.2-01 - 0.1
0.2
46.6 0.3
1..4
5.2
50.1+
41.5
CDANN
54.6 0.4
1.1
43.7
12.1
0.42
4.59
38.3
RSC
55.0 1.2
18.3 0.5
44.4
0.2
0.7
47.8
38.9
ITL-Net
63.5 0.3
19.4
46.3
0.4
53.2
13.2-0.66 0.6
53.5
41.6
Competitors
ERM Reject (0.016)
SagNet Reject (0.016)
CORAL Reject (0.016)
RSC Reject (0.016)
CDANN Reject (0.016)
Ho va-uul
13

---

Under review as a conference paper at ICLR 2022
A.2 DETAILED RESULTS FOR SINGLE SOURCE DOMAIN EXPERIMENT
In of the main paper, we reported single-source DG results, summarising performance across choice of source domain and over all target domains. In Table 7 we now give the detailed results of ITL.-Net for each choice of source and target domain.
Table 7: DomainBed Single source domain recognition accuracy (%) with ResNet50 on VLCS, PACS, Terraincognita and OfficeHome. Each cell reports the accuracy for set of target domains, and the source domain used for training corresponding to the column. The performance of target domains separated by '/' Average over target domains for given source domain is given the bottom of the cell.
Source
Caltech
Labelme
Sun
V-Pascal
Avg.
VLCS
sel
47.81/55.58/59.22
70.00/57.61/66.99
$2.93/6..2760.330
96.75/63.29/766 29/76.81
ERM
54.37
64.5
58.5
78.95
64.08
77.81/55.8//99.7
70.00/57.61/65.90
$2.93/62.2760.33
96.75/63.29
CORAL
54.37
64.5
58.5
78.95
64.08
48. 18.76/53.14/56.99 14/56.93
67.
67.28/62.05/66.70
06.96/62.88/75
SagNet
52.94
50.85
65.34
78.35
61.87
43.19/39. 85/511.1
0..82/55.18/6000
48.90/61.78/0..01
160.4760.
ITT..-Ntt
44.68
68.54
56.9
78.55
62.17
Source
Art
Cartoon
Photo
Sketch
Avg.
PACS
25.35/16..3/45.411
70.17/86.17/66.022
17/66.02
68.07/20.05/66.22
24.76/36.05/27.23
ERM
69.0
74.12
34,91
29.35
51.85
65.36/96.21/455
70.20/86.15/66 15/66.01
68.03/20.04/16.22
24. 24.70/36.05/77.2
CORAL
68.99
74.12
34.9
29.33
$1.84
66. 66.60093.41/55..4
61. 41.42/79.76664.99
2.04/30.38/25.88
26.17/36 16.17736.86/2593
SagNet
71.87
68.7
41.77
29.65
53.0
66. 66.330944.7/57.22
24.85/86.52/75..1
62.60/45.82/55 8.455.44
17.87/26.45/19.66
ITL-Net
72.75
78.81
53.29
21.32
56.54
Terranncognita
Source
L100
L38
L43
L46
Avg.
set
43.82/60.93/69155
39.97/51.27/54.40
40.96/39.1 1/64.70
58.26/46.53/73.77
ERM
57.97
48.55
48.26
59.51
53.57
43. 43.80/60.90/69.11
10.00/51.12/54.221
40.87/40.1
1137/40.12/4444 12/64.23
58. 73,70//5..0 43/73.62
CORAL
57.96
48.44
48.41
59.25
53.51
40.71/56.95/68. 19
37.95/50.44/53.71
33.99/34.41/59.01
59.54//5.93/744
SagNet
55.28
47.37
42.47
60.06
14.79/55.71/677
14.66/53.62/58.07
0197/39.29/66620
61.68/51..6/76..
ITL-Net
56.04
52.12
48.87
63.15
55.04
OfficHommc
Source
Artistic
Clipart
Product
Real World
Avg.
set
44.75/23.36/21.09
39.50/18.66/20.11
37.45/26.00/39.5
27.67/31.4455600
ERM
29.73
26.09
34.33
38.37
32.13
44.75/23.36/21.00
39.49/18.67/20.11
37.45/26.0139.55
27.77/31.46555
CORAL
29.73
26.09
34.34
38.4
32.14
47.98/22.33/17.88
46.40/23.98/14
34.84/34.34/41.0
33.30/35
41/54.25 40.99
SagNet
29.39
28.42
36.93
33.93
21.63/21.57/21.55
13.72/21.79/30..0
01.71/55.4//66 6.64
ITL-Nel
24.91
35.83
37.61
42.0
35.09
14

---

Under review as a conference paper at ICLR 2022
A.3 THE LEARNED LOSS FUNCTION
Recall that our experimental design trained a single loss function on RotatedMNIST, and then evaluated from different perspectives across all main experiments. The specific parameters of our learned ITL (as visualised in Fig. righht) are given in Table A.3 Then reader can plug-and-play for their own Domain Generalisation problems.
Table 8: The parameters of the learned ITI.
Loss type
parameters
Wi, 4/ ... W11
ITL
-2.0193, -1.2234, 0.1363, 0.1269, -0.4566, -0. 1016, -022545, 1.0971, -0.9203, 0.2368, 0.4795, 0.9975
A.4 THE PARAMETERISATION OF THE LEARNABLE LOSS FUNCTION
We apply fourth order bi-variate Taylor polynomial parameterise the learnable loss function. The terms only contain y; are removed from the polynomial since these do not generate gradients with respect the prediction of the network. The final form, only containing 12 learnable parameters, is given as:
ys) y.) w wu) + w3: wa)2 + w wo) + ws (9: wo)* wo(B: wo) w1) + wT(S: wo) w1)² + wis (9: wn) (y) wi) wb(fi " w1) wio wo) w1)x + wi1 w)) wi
A.5 ROBUST LOSS FUNCITON ILLUSTRATION
Gran
bioe
Dven mhe
Mtonn label-
6.5
62
pndtelgonddaa chopue insiiity
21
52
12 32
wuth endbubd
St Ea
Figure 5: Comparison of loss functions (from left right): CE, SCE, FOCAL and ITL. The range of ITL is normalised between 0 and 1.
A.6 META-TRAIN COMPUTE COST
Due to the efficiency of implicit gradient, training our ITL required using PyTorch (Paszke et al. 2017) only required 8 hours on single V100 GPU to complete 200 gradient descent steps w. While the goals and base learning problems are not directly comparable, this is dramatically faster than alternatives that require entire cluster (Li et al. 2019a), and where even very recent fast methods require about 12 GPU-days (Liu et al. 2021).
A.7 META-TRAINIEE CONVERGENCE
Figure shows the convergence of ITL during meta-training on R-MNIST. The x-axis shows outer loop iterations/loss function updates. The lines graph (i) the inner loop accuracy (mean over all source domains) after model training with the current loss function, and (ii) and the outer loop accuracy (held out domain accuracy). The convergence process is quite smooth.
A.8 FULL LOSS LANDSCAPE PLOTT
Figure 7 shows the landscape for all four domains, of which two were shown in Figure of the main manuscript.
15

---

Under review as a conference paper at ICLR 2022
Learning Curve (meta-train)
0.5
0.4
Inner loop (mean all the source domain)
0.2
Outer loop
(holding out domain)
0.
25
75 100
150
175
Outer loop Iteration
Figure 6: The learning curve for ITL meta-training stage.
1.2
Held-out domain: Art
Held-out domain: Clipatt
Held-out domain: Product
Held-out domain: Woald
0.6
1.0
IT-NET
ERM
17.5
15.0
10.0
-..0
-6.0
ML-
ERM
-7.0
8.0
mL
ERN
6.0
4.00
2..0
0.0
mL
EMM
25.0
20.0
-9.0
-6.0
10.0
2.0
.8.0
-10.0
-0.8
0.4
0.0
0.4
0.8
0.8 -0.4 0.0
044
0.8
6.8 84
0.0
0.4
0.8
-0.8
0.4
0.0
0.4 0.8
Perturbation
Pertur
18 12.5
17.5eld Held-out domain:
ITL-E)
17.5
Held-out domain:
-6.5
12.0
14.0
Held-out
0.8
Product
8.0
Held-out
4.0
ERM
Weal-World
35.0
20.0
30.00
10.0
0.8
20.0
15.0%
-0.5
10.0
80
0.2
-1.0
0.0
0.0
10.00
0.4
/erturbetion
Perturbation
Nerturbation
Perturbation
Figure 7: ID Loss Landscape: ITL--et ERM OfficeHome. Top row: Source domain loss
landscape. Bottom row: Target domain loss landscape.
16


Reviews:
[{"review_id": "qIgvp0K8YU8", "reviewer": "Reviewer_DuTo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The idea is interesting.\n+ Lots of experiments and ablation is done.\n---------------\n- The paper lacks an ablation over multi-label binary sigmoid cross-entropy loss. The proposed loss is actually multi-label loss which regards the multi-class problem as a multi-label problem although the label vectors are always in the form of one-hot vectors. Multi-class cross-entropy is a multi-class loss. So it's more fair to compare the proposed loss with multi-label loss, instead of the multi-class cross-entropy loss, as the multi-label loss might be a good baseline for DG.\n- The performances in DG is not SOTA and the paper lacks important references:\nSelf-Challenging Improves Cross-Domain Generalization (ECCV 2020) provides better performance in PACS while authors did not mention this paper in their paper. I doubt this paper can achieve SOTA performances in all the benchmarks.\n- The writing and clarity can be improved in the following aspects:\na. the texts in fig. 1, 3, 4 are extremely small. It's better to enlarge them\nb. the authors should add more descriptions about how they conducted analysis and experiments. For example, what is \"+90%, +80%, - 90%\" in table 5. \n\n", "summary_of_the_paper": "This paper proposes a loss for domain generalization. The loss is learned by meta-learning on based on RotatedMNIST. The paper evaluate the loss's generalization ability in several other datasets.", "summary_of_the_review": "This paper lacks evidences to show its SOTA performances and lacks important references and ablation study. I suggest authors to improve from the above points.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_kfBzmPIAne", "reviewer": "Reviewer_57HS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and well structured. The idea of meta-learning a loss function based on a given domain generalization task is interesting and (to the best of my knowledge) novel. The experimental setup (especially on DomainBed) appears rigorous (but see my comments below).\n\n### Weaknesses:\n\n- The final loss function used is actually not printed in in the paper---or did I miss it? Given that the whole result section is meant to show improvements of using this loss function, it should be added to the main paper along with information about any implementational details needed for re-implementing it.\n- Table 2 is missing confidence intervals or standard deviations / standard errors. Otherwise the differences (sometimes as small as .1 percentage points) are hard to judge. For example, the original paper reported 41.8% for CORAL on DomainNet, vs. 41.5% reported here, so there seems to be some variantion. As an alternative suggestion, what about showing the average rank of the method across all domains in a dataset in brackets behind the performance?\n- §4.3, Repeatability analysis: ColoredMNIST seems to be a bad task choice for this claim, as all results in the paper essentially fail on the task (which is expected---it was specifically designed to show an advantage of the Invariant Risk Minimization algorithm). I suggest to re-run the experiment in Table 5 on a different dataset.\n- The presentation of the results is generally misleading. At many places, section 4 reads more like an advertisement of the method, rather than a neutral assessment and fair comparision of the results. It is good that the full tables are in the appendix to give a better overview of where the method works, and where it fails. For example, on VLCS, CORAL outperforms the method on 3/4 domains, still the average performance is a bit better --- communicating the results in this form is misleading, and needs to be improved (it is simply conditional on the baseline numbers of the different domains). This is not the only example: E.g. in Table 7, the wrong number is bolded for VLCS.\n- The conclusion does not match the paper story and should be improved. I highly recommend to write an extra paragraph on the limitations of the presented experiments.\n- The paper lacks a good \"Figure 1\" that outlines the approach. I suggest one of the less interesting analysis figures (E.g. the current Figure 1) to the supplement, in case space is needed for such a figure.\n\n### Minor points\n\n- The plots need work: When printed, labels and legend entries are not readable.\n- Tables should not have vertical lines according to the formatting instructions.\n- The Table 2 caption should highlight that these results are for a ResNet-50 model.\n\n### Additional questions\n\n- What informed the choice of 0.025 as the signficance threshold?\n- Table 2: How is it possible the the p-value for all comparisions is exactly 0.016? Could you clarify what was actually compared? Was the test performed on average performances, or by considering the ranking on individual domains? If you check all individual domains in DomainBed, what is the ranking of the considered methods? Looking at Table 6 suggestions that e.g. CORAL is often close in performance (despite being a quite old algorithm), the only clear improvements are observed on TerraIncognita and to some extend on OfficeHome. \n- Abstract: \"enables simple ERM to outperform signficiantly more complicated prior DG methods\" --> I disagree with this statement --- the amount of work needed to come up with the loss function is arguably larger than e.g. the CORAL technique. ", "summary_of_the_paper": "The authors design a scheme for meta-learning loss functions for domain generalization: Based on the RotatedMNIST problem, a new implicit gradient algorithm is used to select a loss function from a chosen parametric family, and then applied to other domain generalization problems. The authors demonstrate improvements over the Mixstyle method from Zhou et al (ICLR'21) on PACS, and improvements over ERM and other methods on DomainBed.", "summary_of_the_review": "The paper proposes a new interesting setting for discovering new loss functions based on a predefined task, and extensively evaluates one \"proof-of-concept\" loss function derived with this technique.\n\nMy main concern is that the authors occassionally oversell their results, along with some issues related to this, e.g. a potential inconsistency in the hypothesis testing done for Table 2 (see above). Realistically, it seems that the method works for a few of the settings, but not all of them---which I think is totally fine, especially in the light of the results in the DomainBed paper, that showed that a lot of methods proposed over the past years actually fail to meaningfully outperform ERM.\n\nThe authors should also critically examine their full results again, adapt the language in the paper to be more realistic about the actual merit of the method and the improvements that can be expected. This should be easy to fix by revising the language in the result and conclusion section a bit, and adding a short discussion of the failure cases.\n\nI assign a score of (5) for now, but would be very happy to increase it --- based on the results already in the paper, I think the paper could become an interesting contribution to ICLR, if presented correctly.\n\n---\n\n**Post-Rebuttal comments:**\n\nThe authors sufficiently addressed my concerns, and I decided to increase my score to (6), as I think that this work might be an interesting contribution in the space of domain generalization, first and foremost for putting forward the idea of learning loss functions based on an auxilary domain generalization task itself, which to the best of my knowledge this paper provides the first proof-of-concept for.\n\nFor the camera ready, I would however urge the authors to carefully re-check the paper again, and tune down claims that are not sufficiently supported by the presented data. I would also consider to remove the statistical analysis (but keep the average ranks--these are informative), cf. our ongoing discussion, I think in the current form it does not add much information to the paper. If the authors include it, the performed test should be well justified in the supplement, which is currently not the case. I would also recommend to run a test applicable to multiple comparisons, and then perform a suitable post-hoc test, instead of the current practice of testing all models with a Wilcoxon test. It should be highlighted that especially for CORAL/SagNet vs. the proposed method, the gains are rather small in some cases.\n\nIf the authors add a bit more critical discussion on their method to the paper, I would hence recommend to accept the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fk9BYuP1mr", "reviewer": "Reviewer_Hx46", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+ The paper is well written and easy to follow, the idea is presented clearly.\n\n+ The idea is novel and interesting; the empirical observation that learned losses transfer to new tasks results in a practical framework.\n\n+ An extensive empirical evaluation is carried out, comparing the proposal against a diverse set of alternative methods and using a number of benchmarks, clearly supporting authors' claims.\n\n+ The proposed loss can be directly combined with other mechanisms to further improve performance; e.g., it could be used along with some invariance-inducing strategy. \n\n\nWeaknesses/questions/suggestions:\n\n- It's possible that the proposal requires assumptions that are not discussed in the manuscript. In particular, empirical evaluation is carried out on the more common cases where data marginals shift, but data-conditional label distributions P(y|x) are fixed across domains; i.e., observing x suffices in order to determine y. It's unclear whether the learned losses would still yield improvements in the more general cases where P(y|x) is domain-dependent, i.e. the same x could have different labels depending on the domain it was observed from; C.f. [1] for an in-depth discussion on the effects of conditional shifts. In particular, if the proposal induces some type of domain-invariance, be it in the feature or prediction level, it would likely damage performance relative to standard ERM in such cases.\n\n- The evaluation/discussion lacks in determining how learned losses improve models' robustness; e.g., are resulting models more domain-invariant in some sense? If not, what is it that makes them more robust than models training with standard ERM?\n\n- The fact that learned losses transfer to new tasks is clearly supported by the reported evaluation. However, it would be interesting to get an idea as to whether one could benefit from training a loss tailored to a particular set of domains. In that case, it would be also interesting to get an idea of the computational cost involved in running Algorithm 1 in a larger-scale task such as PACS.\n\n- In section 3, the text makes it a bit confusing to determine what are the authors' original contributions and what are the bits that were introduced previously and re-used in the proposal.\n\n- Please include in the manuscript a brief description of the \"gradient surgery\" scheme used to aggregate gradients in Algorithm 1. Is there any reason why this particular approach would work better than simply averaging?\n\n- Authors follow the common practice within domain generalization literature and perform evaluations using models pre-trained on ImageNet. However, in my opinion, doing so renders comparisons noisy and it's unclear how much pre-training helps in generalization to new domains. As a suggestion, I recommend repeating a subset of the evaluations without pre-training and determining whether conclusions hold.\n\n- What is the cross-validation criterion used for experiments on PACS?\n\n- Please report the sample size used for the significance test reported in Table 2.\n\nReferences:\n\n[1] Zhao, Han, et al. \"On learning invariant representations for domain adaptation.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "Authors introduce a bi-level optimization procedure aiming to learn parametric loss functions that result in predictors able to better generalize to new data sources, unseen during training. Specifically, an outer optimization loop iteratively updates a parametric loss function to minimize the empirical risk (under a standard loss) on a left-out data source, not presented to the model during the inner optimization loop, which trains the model on a set of domains using the current version of the parametric loss. The proposed approach is computationally expensive, but authors empirically showed that losses learned in low-scale tasks directly transfer to other cases, rendering the proposal practical.", "summary_of_the_review": "The proposal is novel to my knowledge and interesting. It can also be directly combined with other approaches to further improve performance. The execution is robust and covers a number of well-known datasets; comparisons include recent alternative methods. Results include significance tests.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DGzYVuzXJb", "reviewer": "Reviewer_eZ71", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is interesting since the author provides an alternative perspective in domain generalization: searching the proper loss function.  However, this reviewer still found a couple of important concerns (including theory, method, and paper writing) that prevent me from accepting the current version. Based on these, I believe a major revision and rethinking about the cons can significantly improve the paper.\n\nPros:\n\n- This paper proposes a novel direction in domain generalization: searching the proper loss. Although this idea is not completely novel in the related domains such as NAS or AutoML, applying them to the domain generalization seems quite interesting and novel.\n- The empirical studies and analysis are extensive.\n\nCons (see detailed reviews for explanations):\n\n- The main concern lies in the theoretical aspects: why and when it works or fails? The proposed idea currently seems a bit ad-hoc. \n- The paper is not self-contained. Some details are quite difficult to follow or lack motivation.\n\n---------------Detailed reviews\n\n**About theoretical understanding.**\n\nI totally agree with the author’s viewpoint about the problem revealed in domainBed. However, in this paper, the author fails to clearly justify why the proposed parametric loss $w$ can effectively solve this problem. Specifically,\n\n1. Does the bi-level optimization indeed search for the correct parametric loss rather than over-fitting? Note in the domain generalization, the source number is generally quite limited (different from NAS), supposing you only have 3 sources, it is quite difficult to show the bi-level indeed finding the best configuration. (It is more likely overfitting). To address this, the author should provide a clear theoretical analysis to show when the proposed method can work. \n\n2. The selected Taylor Polynomial representation is wired and lacks motivation. Why do we choose this specific loss? What is the convergence behavior in the bi-level optimization if this loss is adopted? Why not MSE loss? Triple loss? This paper lacks a clear motivation in illustrating why it is essential.\n\n3. The experimental settings in the single DG are also wired. Clearly, one source could not generalize to other related targets. Therefore I am a bit doubtful about the proposed method...\n\n**Other technique details**\n\n- Some parts are not self-contained and unclear. For example, \n\n1. Algo 1, Line 15 there is no description of the grad-surgery approach.\n\n2. Algo1, Line 16. h{update...} what does it mean?\n\n3. In algorithm 2, the description seems inconsistent with the sum of the Neumann series. The role of j is missing in the loop. \n\n4. Computing jacobian-vector product in PyTorch seems a bit difficult if we directly adopt PyTorch since PyTorch currently does not support the form (algorithm 2) Jacobian-vector-product(parameter of neural-network,\\theta,v). I am wondering how you addressed this.\n\n- The convergence bound of using Neumann series or Implicit gradient is highly expected to provide. (I think it is feasible under proper assumptions.)\n\n- There may exist a memory complex concern since the $H$ in the algorithm depends on the task number. If n is large, the memory complexity can be quite high (although the implicit function addressed the computational complexity).\n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper proposed a novel viewpoint in domain generalization, i.e, the loss function search. Specifically, the search procedure is decomposed as a bi-level optimization and solved through implicit function. The author later adopted the Neumann series for memory saving. The extensive empirical results validate the correctness of the proposed approach.\n\n====== Final Update after rolling discussions\n\nI would appreciate the authors for the detailed responses and I have read the rebuttal. Unfortunately, the concern of rigor is still not addressed after discussion and I will maintain the same rating. Below are my *final* remarks (I fully understand your high-level idea), I hope the author will carefully rethink the paper and improve it in the future version.\n\n- In the rebuttal and paper, many terms are not mathematically/rigorously defined, which made me rather confused. These terms even made me doubtful about the generalization properties since they should be rigorously defined and discussed. (see Note 1 for the detailed examples)\n\n- The motivation for choosing the specific loss parameter families is not properly addressed in the paper. From the rebuttal, I finally got the logic/motivation but it was too late. I do think a major revision for fully justifying the motivation can greatly improve the paper. (see Note 2 for the detailed discussions)\n\n---------------------\n\nNote 1.  Examples of unclear terms, most of them are not formally formulated/defined. The descriptive words rather than the math formula made the reviewer confused and a bit doubtful about the generalization property since generalization should be rigorously defined.\n\n  - \"Scale of the domain shift is similar enough between meta-train and deployment datasets.\" What is the formal definition of the scale of domain shift?\n\n -  “the cardinality of the classification problem is not too different between meta-train and meta-test.” What is the formal definition of the cardinality of classification?\n\n  - “But please note that something like MNIST->SVHN transfer is not exactly the problem setting we address.” It is not domain generalization (trained on digit A and deployed on digits B)? What is the formal (or mathematic) definition of a single source setting? It is best to use equations rather than descriptive words to depict the problem settings. In fact, the single source is a confusing word (since it is not mathematically defined.)\n\n -  “large search space is expected to lead to meta-overfitting.” What is the formal definition of meta-overfitting? Overfitting w.r.t. which distribution? The author should be cautious to use unclear words since meta-overfitting has not been mathematically defined. \n\n\nNote 2. The motivation of choosing the specific loss parameter.\n\nThanks for the rebuttal. It makes me clear about the motivation. But it requires a major revision of your paper to fully justify it. For example, I would suggest testing the linear combination and showing its results. Also exploring other loss families to fully justify the value of the loss search. In the current version, it is more or less we proposed loss family A (without strong motivation or benefits) and it works in some classification benchmark.  A *comprehensive and systematic* empirical understanding can significantly improve it.  \n\n-------------------------------------------------------\n", "summary_of_the_review": "This paper adopts the idea of NAS or autoML to search for the proper loss in the domain generalization. Overall the idea sounds novel in the domain generalization while the current versions lack clear justifications (in terms of theory). Based on these, I do not recommend accepting the current version but encourage a major revision for the resubmission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the idea of meta-learning the loss function for domain generalization. It's a simple idea, that seems to work reasonably well. Although, as pointed out by the reviewers, the margin is actually quite modest when compared to the strongest baselines (not ERM).

On a positive note, many reviewers agree that the idea was simple, novel, and interesting. The insight that cross-entropy can be improved for domain generalization is interesting. On the other hand, many reviewers pointed out that the, despite some careful empirical work, it's not clear why this idea works. I read the paper myself, and I agree that the paper could use a bit more work before it is ready for publication. Specifically, I agree with Reviewer eZ71, who asked for a clear justification of the proposed idea. The idea seems sensible, but there is some burden on the paper to provide insight, and not simply present an idea.  Here are some specific suggestions that came up during discussion, which could strengthen the paper:
- A more comprehensive discussion of the limitations of this approach.
- It would be good to understand how critical was the specific choice of parametric loss family. Here are some questions that would be good to address: does the parametric family interact with the type of domain shift in the datasets? Why are Taylor polynomials preferable or beneficial for domain generalization compared to, e.g., a linear combination of standard loss functions?
- Is the dataset on which you learn your ITL loss critical? I.e., how critical was the choice of rotated MNIST for learning the ITL loss? Does it generalize to very different and more diverse domain shift tasks, like those in the WILDS benchmark? It would be particularly interesting to see if loss functions meta-trained on distinct datasets learn similar parameters.
- More broadly, evaluation on larger and more diverse domain shift tasks, like those in the WILDS benchmark, would further strengthen the conclusions in the paper.
Decision: Reject