Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models
Authors: Byounggyu Lew
Donghyun Son
Buru Chang
Venue: Submitted to ICLR 2023 (2023)
Keywords: domain generalization
gradient estimation
pre-trained models

Abstract: Domain generalization aims to build generalized models that perform well on unseen domains when only source domains are available for model optimization. Recent studies have demonstrated that large-scale pre-trained models could play an important role in domain generalization by providing their generalization power. However, large-scale pre-trained models are not fully equipped with target task-specific knowledge due to a discrepancy between the pre-training objective and the target task. Although the task-specific knowledge could be learned from source domains by fine-tuning, this hurts the generalization power of the pre-trained models because of gradient bias toward the source domains. To address this issue, we propose a new domain generalization method that estimates unobservable gradients that reduce potential risks in unseen domains, using a large-scale pre-trained model. Our proposed method allows the pre-trained model to learn task-specific knowledge further while preserving its generalization ability with the estimated gradients. Experimental results show that our proposed method outperforms baseline methods on DomainBed, a standard benchmark in domain generalization. We also provide extensive analyses to demonstrate that the estimated unobserved gradients relieve the gradient bias, and the pre-trained model learns the task-specific knowledge without sacrificing its generalization power.

Under review as a conference paper at ICLR 2023
GRADIENT ESTIMATION FOR UNSEEN DOMAIN RISK MINIMIZATION WITH PRE-TRAINED MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Domain generalization aims to build generalized models that perform well on unseen domains when only source domains are available for model optimiza- tion. Recent studies have demonstrated that large-scale pre-trained models could play an important role in domain generalization by providing their generaliza- tion power. However, large-scale pre-trained models not fully equipped with target task-specific knowledge due to a discrepancy between the pre-training ob- jective and the target task. Although the task-specific knowledge could be learned from source domains by fine-tuning, this hurts the generalization power of the pre- trained models because of gradient bias toward the source domains. To address this issue, we propose a new domain generalization method that estimates unob- servable gradients that reduce potential risks in unseen domains, using large- scale pre-trained model. Our proposed method allows the pre-trained model to learn task-specific knowledge further while preserving generalization ability with the estimated gradients. Experimental results show that our proposed method outperforms baseline methods on DOMAINBED, standard benchmark in domain generalization. We also provide extensive analyses to demonstrate that the esti- mated unobserved gradients relieve the gradient bias, and the pre-trained model learns the task-specific knowledge without sacrificing its generalization power.
INTRODUCTION
Many machine learning studies assume that training and test data are independent and identically distributed (i.i.)). However, this i.idd assumption does not always hold in real-world scenarios where distribution shifts between training and test data occur frequently. Thus, traditional machine learn- ing models often show poor performance on unseen domains shifted from source (training) do- mains (Quinonero-Cande) et al., 2008 Torralba & Efros 2011). ) To tackle this problem, domain generalization has attracted much attention recently.
The main goal of domain generalization is to build generalized models that also perform the target task (e.g., classificationn well on unseen domains (e.g.. painted images) when only source domains (e.g., realistic images) accessible during model optimization. Early domain generalization stud- ies (Muandet et al. 2013) Ganin et al. 2016 Li et al., 20185) have focused on learning domain- invariant representations across the source domains. However, Gulrajani & Lopez-Pazi (2021) have recently shown that simple empirical risk minimization (ERM) [Vapnik] 1999) outperforms the pre- vious methods DOMAINBED, benchmark for domain generalization, with pre-trained ResNet- 50 (He et al. 2016). Moreover, Yu et al. (2021) provide empirical evidence that large-scale pre- trained models could play an important role domain generalization by providing their generaliza- tion power.
Motivated by this, several studies have begun to leverage the generalization power of large-scale pre-trained models. [Cha et al. (2022) employ pre-trained model for regularization, considering as an approximation of the oracle model on any domain, and Li et al. (022) utilize a frozen pre-trained model as feature extractor. These studies have proven the usefulness of pre-trained models in domain generalization. However, the pre-trained models used in those studies cannot leam task-specific knowledge further since they are frozen during model optimization preserve their generalization ability. To learn the task specific knowledge, one can choose fine-tuning that updates all the parameters of pre-trained models by optimizing the models on the source domains.

---

Under review as a conference paper at ICLR 2023
Source Domains
Parameter Space
PACS
VLCS
FRM
Our Melhoo
ERM
Our Melhod
0.4
Painting
of Contlica ia of Total Ierrattnns
0.3
0.2
Cartoon
RasNet
GLIP
SWAG
RosNet
CUP
SWAG
OfficeHomme
Terraincognite
Sketch
gu
MurMedood
Methoo
0.4
Unseen Domains
biased gradient unobserved gradlent 8: model parameters
0.3
0.2
Photo
0.1
ResNet
CLIP
SWAG
(a) Biased gradicnts of naive fine-tuning
(b) Gradient conflict between g and gu
Figure 1: (a): model optimization is inftuenced by the gradient biased toward the source domains, neglecting the unobservable gradient gu that could minimize risks in the unseen domains. (b): Gradient "conflicts" (Yu et al., [2200] Mansilla et al. (021) between and go (i.e., g g. < 0) constantly occur throughout the whole fine-tuning iterations due the gradient bias. Our proposed method reduces the number of gradient conflicts by adding the estimated unobservable gradient ga the biased gradient g. This observation indicates that the gradient bias is relieved with the estimated gradient during model optimization. The more details are described in 3.4
However, Kumar et al. (022)) demonstrate that fine-tuning distorts generalized representations of the pre-trained models. Namely, fine-tuning hurts the generalization ability of pre-trained models.
In this paper, interpret the above issue in terms of gradient bias during model optimization. As shown in Figure Ia] the gradient of naive fine- tinettunig is biased toward the source domains because it is computed by only the source domains, disregarding unseen domains. Although this biased gradient reduces empirical risks in the source domains with the learning of task-specific knowledge, probably increases risks in the unseen domains. We argue that the gradient bias would be relieved if gradients that lower the risks in the unseen domains are observable.
To this end, propose new domain generalization method, called GESTUR, which estimates the unobservable gradients with a large-scale pre-trained model. GESTUR consists of two key components: a task expert (TE) and a generalization expert (GE). Based on ERM where gradients tend to be biased to the source domains, TE learns task-specific knowledge from source domains directly transfer the knowledge GE. Meanwhile, GE learns the task-specific knowledge from TE indirectly via exponential moving average (EMA) while preserving the generalization ability of a large-scale pre-trained model. Still, the gradient bias of TE might impair the generalization ability of GE. To mitigate this, GE is utilized to estimate the unobservable gradient that minimizes risks unseen domains for TE based on the assumption that large-scale pre-trained models could act as loose approximation of the oracle model of unseen domains ($ ( As shown in Figure [b] the biased gradient of TE is relieved by simply adding the estimated unobservable gradient to the biased gradient, improving domain generalization performance ($ 3). Extensive experiments and analyses demonstrate that GESTUR outperforms baseline methods by learning the task-specific knowledge appropriately from source domains while preserving the generalization ability of large-scale pre- trained models.
Contributions: (1) We propose a simple yet effective domain generalization method that learns task task-specific knowledge while preserving the generalization ability large-scale pre-trained mod- els. Our proposed method estimates the unobservable gradients that reduce potential risks in unseen domains to relieve the gradient bias toward source domains, based on the two experts, TE and GE. (2) We conduct extensive experiments to show the effectiveness of our proposed method in domain generalization. By providing careful analyses, we demonstrate that the unobservable gradients could be estimated with large-scale pre- trained model, and it relieves the gradient bias. We also demon- strate that our proposed method learns task-specific knowledge without sacrificing the generalization ability of the large-scale pre-trained model.
2

---

Under review as a conference paper at ICLR 2023
2
METHODOLOGY
2.1 PRELIMINARIES
Problem formulation. Let D, and P, be sets of source domains and unseen domains, respec- tively. Each domain D contains the total number of nip data samples, { 8r) DD where each data sample (zy, y/) consists of an input 2; and its target label yr- The np data samples are i.dd over some probability distribution. The main goal of domain generalization is to build model 0 that performs well on the unseen domains Du when the source domains D, are only available:
min ED-DD (a [e(( 0);
where (((x, y); 0) is the loss function defined for the model O on the data sample (1r, y). Note that this study focuses on solving classification tasks. Hence, we denote the model in detail as = {0); 0*) consisting of its feature extractor e and classifier 6
Motivation. With success in many downstream tasks, has become a convention to initialize the feature extractor of with a large-scale pre-trained model. Although pre-trained models provide better feature representations than randomly initialized parameters, they do not fully equip task- specific knowledge yet. is because there is discrepancy between the pre-training objective and the target task. For example, CLIP (Radford et al.] (02T) is pre-trained to match web-crawled image- caption pairs, whereas the target task is to classify data into seven classes (e.g., horse and dog), in the case of PACS (Li et al. 2017). Therefore, many studies have adopted fine-tuning that updates all the parameters of the feature extractor of to learn the task- task-specific knowledge by optimizing the model on source domains D,. However, Kumar et al. (2022) observe that fine-tuning impairs generalization ability of pre-trained models during the learning of task-specific knowledge.
We try to interpret this issue at the gradient level. Based ERM (Vapnik 1999), the gradient g of fine-tuning is computed for the model A on the source domains D., as follows:
g V VoB(o.))-s((((z y): 0)),
(2)
where B is a mini-batch sampled from the source domains D,. The gradient g is influenced by only the source domains D, because the unseen domains D., are not accessible. Namely, the gradient is biased toward the source domains. We presume that this gradient bias degrades generalization performance in the unseen domains.
2.2 GESTUR: GRDIENT ESTIMATION FOR UNSEEN DOMAIN RISK MINIMIZATION WITH MODELS
We hypothesize that the gradient bias mentioned above could be relieved if the unobservable gra- dient g., minimizing risks in the unseen domains is computable. To achieve this, we borrow the assumption of Cha et al. (2022) that large-scale pre-trained models are the approximation of the oracle model 6* which is optimally generalized for any domain D. Since the unobservable gradi- ent gu cannot be computed from the unseen domains D, directly, we consider the direction from the current model 0 the oracle model 0 as the unobservable gradient ga. However, the oracle model is inaccessible in practice. Hence, we estimate the unobservable gradient using a large-scale pre-trained model the approximation of the oracle model.
Note that we aim to estimate the unobservable gradient gu for the unseen domains D., to alleviate the gradient bias, so the above assumption needs to be more elaborate due the following rea- sons. First, we intend to design the unobservable gradient for the unseen domains only rather than any domain. Second, pre-trained models do not have task-specific knowledge yet, as described in & $2.1 Therefore, we slightly modify the assumption as follows: pre-trained models are the loose approximation of the oracle model 0% of the unseen domains Da, and they could get closer to the oracle model by learning task-specific knowledge. Based on this assumption, we propose simple yet effective domain generalization method, GESTUR, which estimates the unobservable gradient ga for unseen domain risk minimization with large-scale pre-trained model.
Task expert and generalization expert. GESTUR consists of two classification models: task expert (TE, Ore) and generalization expert (GE, OCE), which are complementary each other.
3

---

Under review as a conference paper at ICLR 2023
Their feature extractors are both initialized with a large-scale pre-trained model ou respectively. TE learns task-specific knowledge from the source domains D, directly to transfer the knowledge to GE. Mcanwhile, GE also learns task-specific knowledge from TE via EMA, but it preserves the generalization ability of the pre-trained model deliberately. Here, the gradient bias of TE might hurt the generalization ability of GE because the knowledge of 'E is injected into GF. To relieve the gradient bias, GE is used to estimate the unobservable gradient gu. as the loose approximation of the oracle model 0% for the unseen domains D.. Our proposed GESTUR summarized in Algorithm |
Gradient estimation. Using Equa- tion [ the gradient g for TE is computed as g = (([[,,))) v): fn)) while learning task-specific knowledge. The gradient is biased toward the source domains D.. A gradient that minimizes risks in the unseen domains could relieve the gradient bias, but unobservable. When we have access to the oracle model 0% of the unseen domains D... we can direct the current model to head to the oracle model instead empirically cal- culating the unobservable gradient from the unseen domains. Hence, we treat the direction from the current model ÔTE the oracle model A" as the unobservable gradient gu:
Algorithm GESTUR
1: Input:
task expert Oru, generalization expert duz, gradient scale factor A, and moving average cocfficient m.
2: Init: initialize the feature extractors 6 and 0/ with
pre-trained model On and randomly initialize the classi- fiers 0f: and 0G.-
3: Output: the updated generalization expert Ocs
4: for sampled mini-batch B from the source domains D, do
5:
g Vol(rr,)) ((2, y); 8n)]]
6:
g!! 0/2 f
B!
7:
E - Allg 112
1g.l2
8:
g (g' + gg)/2
9:
update of with g' and update Oie with g"
10:
update Ocx móGe + (1 m)0TE
11:
end for
gu = 6 PIE.
(3)
In fact, it is infeasible to access the oracle model. Thus, we estimate the unobservable gradient using GE that approximates the oracle model loosely, as follows:
Su
OCE θTE-
(4)
This estimated gradient ga is used to relieve the gradient bias during the parameter optimization.
Parameter optimization. We want to emphasize again that GESTUR leverages the generalization power of large-scale pre-trained models relieve the gradient bias which distorts the generalized feature representations of the feature extractor A1, Hence, we limit the scope of usage of the esti- mated unobservable gradient g. only to the feature extractor 05, not the classifier 8
For TE, the estimated gradient é! for the feature extractor of is added to the biased gradient g' for the same feature extractor, follows:
g -/= +
(5)
where À is gradient scale factor that controls the influence of the normalized g!- The feature extractor of is updated with the gradient g' adjusted by g! On the other hand, the classifier fig of TE is updated with its original gradient g".
As our assumption, GE can get closer to the oracle model 0* by learning task-specific knowledge. However, the generalization ability of GE decreases when we optimize GE on the source domains D, directly to learn the task-specific knowledge. Therefore, we inject the learned tpec-spe knowledge of TE into GE delicately via EMA:
OCF mÔGe (1 m))TE,
(6)
where m is the moving average coefficient. By encouraging the parameters of GE to change slowly, EMA is helpful in preserving the generalization ability of GE. Since the goal of domain generaliza- tion is to build a model that minimizes the risk the unseen domains, we choose GE, designed to approximate the oracle model of the domains, our final model 0.
4

---

Under review as a conference paper at ICLR 2023
Table 1: Evaluation results (%) on the five datasets with the three different pre-trained models.
Method
PACS
VLCS
OfficeHome
TerraInc
DomainNet
Avg.
Using ResNet-50 pre-truined on ImageNet.
ERM
84.2 10.1
77.3 10.1
67.6 10.2
47.8
10.6
44.0
10.1
64.2
SagNet
86.3 10.2
77.8 10.5
68.1 10.1
48.6
11.0
40.3
10.1
64.2
SelfReg
85.6
77.8 10.9
67.9 10.7
47.0
42.8
64.2
CORAL
86.2 10.3
78.8 10.6
68.7 10.3
47.6 11.0
41.5 10.1
64.5
mDSDI
86.2 #0.2
79.0 +0.3
69.2 #0.4
48.1 #14
42.8 #0.1
65.1
GVRT
85.1 #0.3
79.0 #0.2
70.1 #0.1
48.0 #1.4
44.1 #0.1
65,2
MIRO
85,4 #0.4
79.0
70.5 #0.4
51.4
44.3
65,9
SMA
87.5
78.2 10.2
70.6 20.1
40.5
66.5
SWAD
88.1 10.1
79.1 10.1
70.6 20.2
50.0 10.3
46.5 10.1
66.9
GESTUR
88.0 #0.2
80.1 #0.2
71.1 20.0
51.3 10.2
46.3 10.1
67.4
Using ViT-B/1h with CLIP.
ERM
83.4 10.5
75.9
11.3
66.4 10.5
10.8
44.4 10.6
61.1
SWAD
91.3 #0.1
79.4
+0.4
76.9 #0.1
45.4 #0.5
51.7 20.8
68.9
MIRO
95.6 #0.8
82.2 #0.3
82.5 #0.1
54.3 #0.4
54.0 #0.3
73.7
GESTUR
96.0 =0.0
82.8 #0.1
84.2 #0.1
55.7
#0.2
58.9 #0.1
75.5
Using RegNerY- 16GF with SWAG.
ERM
89.6 10.4
78.6 10.3
71.9 10.6
51.4 11.8
48.5 10.6
68.0
SWAD
94.7 10.2
79.7 10.2
80.0 10.1
57.9 10.7
53.6 10.6
73.2
MIRO
97.4
79.9 #0.6
80.4
58.9 +13
53.8 #0.1
74.1
SMA
95.5 +0.0
80.7 #0.1
82.0 #0.0
+0.0
60.0 +0.0
75.6
GESTUR
96.9 #0.1
83.5 #0.1
83.1 +0.0
61.1 #0.4
60.1 +0.0
76.9
EXPERIMENTS
3.1
EXPERIMENTAL SETUP
Datasets. We conduct experiments using five popular domain generalization benchmark datasets: PACS et al. (017) (4 domains & classes), VLCS (Fang al. 2013) (4 domains & 5 classes),
OfficeHome (Venkateswara et al. 2017) (4 domains & 65 classes), TerraIncognita (Beery et al.,
2018 (4 domains & T0 classes), and DomainNet (Peng al. 2019) (6 domains & 345 classes).
Pre-trained models. GESTUR heavily relies on pre-trained models. Therefore, we employ three pre-trained models of different sizes verify that the proposed method performs well with various pre-trained models generally: ResNet-50 (He et al.) (0066) pre-trained on ImageNet (Deng et al.) 2009) (RN50), ViT-B/16 (Dosovitskiy et al. (2021) with CLIP (Radford et al. 2021 (CLIP), and RegNetY-16GF (Radosavovic et al. 2020) with SWAG (Singh et al. 2022 (SWAG).
Evaluation protocol. We adopt the experimental protocol of DOMAINBED, which enforces fair and realistic evaluations (e.g., same model selection criterion) across competitors. We divide the data from each domain into 80% and 20% splits and follow training-domae validation set strategy for the model selection and the hyperparameter search in every experiment. We also repeat every experiment three times to reduce the randomness in dataset splits and parameter initialization, sim- ilar to Gulrajani Lopez-Paz (20T). and report the mean and standard error the experimental results.
Implementation details. Our implementation is built on the codebase of Cha et al. (0022). We use Adam optimizer (Kingma & Ba 2015) for parameter optimization. GESTUR has two hyperpa- rameters, the gradient scale factor (X) and the moving average coefficient (m)) In every experiment, search the optimal À from {0.01, 0.05, 0.1. 0.5) } and fix as 0.999. Other hyperparameters such as learning rate, weight decay, and dropout are searched in the same way as Cha et al. (2022). We explain more details of implementation in Appendix A
5

---

Under review as a conference paper at ICLR 2023
Table Evaluation results (%) on the four datasets with the three different pre-trained models. We separate the cases where GESTUR uses TE and GE as the final model, respectively.
Method
PACS
VLCC
OfficeHome
TerraInc
Avg.
Using ResNet-50 pre-trained on ImageNet.
ERM
84.2 #0.1
77.3 #0.1
67.6 10.2
47.8 10.6
69.2
GESTUR w/ TE
84.9 #0.1
79.2 #0.5
66.3 #0.2
45.6 +1.3
69.0
GESTUR GE
88.0 =0.2
80.1 =0.2
71.1 +0.0
51.3 #0.2
72.6
Using ViT-B/16 with CLIP.
ERM
83.4 10.5
75.9 11.3
66.4 10.5
35.3 10.8
65.3
GESTUR w/ TE
90.7 10.9
82.4 10.4
76.9 10.5
50.4 10.2
75.1
GESTUR GE
96.0 10.1
82.8 10.1
84.2 10.1
55.7 1:0.2
79.7
Using RegNetY- 16GF with SWAG.
ERM
89.6 20.4
78.6 20.3
71.9 10.6
51.4 118
72.9
GESTUR w/ TE
94.8 20.5
82,5 #0.4
77.7 40.2
54.7 +2.0
77.4
GESTUR w/ GE
96.9 20.1
83.5 #0.1
83.1 40.0
61.1
40.4
81.2
Baselines. We exhaustively compare our proposed method with various baseline methods in the experiment. For simplicity, we report only the experimental results of baseline methods that show the higher performance than ERM (Vapnik 1999). the simplest bascline method. We describe the baseline methods and report the full version of the results Appendix B.1
3.2
MAIN RESULTS
Results on RN50. The first part of Table shows the experimental results where RN50 is used to initialize the feature extractor. GESTUR achieves the best performance for all the datasets except DomainNet. In detail, the proposed method outperforms ERM by average of 3.2%p. Further- more, our proposed method improves the runner-up by: 1.0%p in VLCS, 0,5%p in OfficeHome, and 1.3%p in Terralncognita. Especially, the proposed method outperforms the state-of-the-arr method (SWAD (Cha et al. )02I) by an average of 0.5%p.
Results on CLIP and SWAG. In the second and third parts of Table [ we show the experimental results where the larger pre-trained models, CLIP and SWAG, are used initialize the feature extrac- tor, respectively. In summary, GESTUR achieves the best performance in all the datasets. In detail, the proposed method outperforms MIRO that also leverages generalization power large-scale pre- trained models by 1.8%p and 2.8%p CLIP and SWAG, respectively. From this, verify that the proposed method successfully leverages the generalization ability of pre- pre-trained models compared to other baseline methods. Interestingly, observe that the performance gap between the proposed method and ERM increases as the size of the pre-trained model increases.
3.3 COMPARISON BETWEEN THE TASK EXPERT AND THE GENERALIZATION EXPERT
Setup. GESTUR consists of two essential components: the task expert (TE) and the generalization expert (GE). In this paper, we use GE as the final model based on the assumption that GE is set as the approximation of the oracle model of unseen domains. Nevertheless, TE also designed preserve the generalization ability of pre-trained models since it also considers the estimated unobservable gradient in every update to relieve its gradient bias. Therefore, we compare the performance of ERM, GESTUR w/ GE, and its variant GESTUR w/ TE based on the hyperparameters. searched in 3.2
to show that they preserve the generalization ability of large-scale scale pre-trale traitee models.
As shown in Table 2, GESTUR w/ GE achieves the best performance in all experiments. Also, GESTUR w/ TE outperforms ERM by averages of 9.8%p and 4.5%p when using CLIP and SWAG, respectively. The performance of GESTUR w/ TE higher when the larger pre-
Results.
pre-trained trained mod- 3.2 These observations demonstrate that GESTUR w/
els are given, similar the observation in

---

Under review as a conference paper at ICLR 2023
TE could preserve the generalization ability of the pre-trained models with the estimated unobserv- able gradient, i.e. the gradient bias of TE is relieved. Moreover, GESTUR w/ GE shows higher performance than GESTUR w/ TE, which indicates that EMA ensures the model preserves gener- alization ability during the learning of task-specific knowledge stable. From these, we reaffirm the justification for our choice of GE as the final model.
3.4 COMPARISON WITH ERM TERMS OF GRADIENT BIAS
Setup. As described in $ we suspect that
Table The percentage (%) of gradient conflicts between and gu to the whole training iterations.
the gradient bias degrades the domain gen- eralization performance. We further conduct analysis to check how much gradient bias oc- curs during the fine-tuning and how much gradient bias is alleviated by our proposed method. To quantify the gradient bias, we borrow the concept of gradient conflict (Yu al., 2020; MMnsiila et al. 202T): there a conflict between two gradients
Method
PACS
VLCS
OH
TI
Ayg.
Using ResNet-50 pre-trained ImageNet.
ERM
28.6
37.3
20.3
35.4
30.4
GESTUR
26.2
29.8
21.0
30.7
26.9
Using ViT B/16 with CLIP.
and g g: 8) < O. For every iteration, we first sample two mini-batches from both source domains and an unseen domain, respectively. We then compute losses of the mini-batches, and calculate gradients g and gu from the losses, respectively. Finally, number iterations where the gradient conflict (g ga < 0) occurs, for ERM and GESTUR. Here, update the model using only the gradient g since unseen domains are inaccessible in practice.
ERM GESTUR
35.3
43.1
33,4
42.6
38.6
28.7
37.4
25.0
30.8
30.5
Using RegNetY-I6CH with SWAG.
ERM GESTUR 24.5
31.7
39.7
30.0
37.5
34.7
34.8
16.5
23.7
24.9
count the
Results. As shown in Table 3 GESTUR reduces the gradient conflicts of ERM by around 1 11.5%, 21%, and 28.2% for the pre-trained models, respectively. From this, we verify that our proposed method relieves gradient bias by estimating unobservable gradients with the pre-trained model. We observe that gradient conflicts occur more often GESTUR than ERM on only the experimental setup (OfficeHome w/ RN50), which is consistent with the performance in Table 2 where ERM out- performs GESTUR w/ TE. This observation indicates that the domain generalization performance is affected by the gradient bias represented as the gradient conflicts in this analysis. Additional analysis on the similarity of the true and estimated unobservable gradients provided in Appendix
3.5 TASK-SPECIFIC KNOWLEDGE LEARNED BY THE GENERALIZATION EXPERT
Table Linear probing performance (%) with the two different pre-trained feature extractors: frozen pre-trained model &g and the feature extractor ff of GE.
Model
PACS
VLCS
OfficeHome
TerraInc
Avg.
Using VIT-B/16 with CLIP.
fruzen GE
98.5 10.1
88.5 10.2
89.3 10.1
83.4 10.2
89.9
98.7 +0.1
90.0 +0.6
89.4 =0.3
88.3 +0.1
91.6
Using RegNetY-16GF with SWAG.
frozen GE
98.9 +0.1
87.1 +0.2
89.6 =0.0
89.3 +0.0
91.2
98.7 40.1
88.8 40.2
90.1
40.2
90.2 10.1
92.0
Setup. We conduct additional experiments to show that the feature extractor of of GE learns task- specific knowledge successfully. Linear probing that updates parameters of only the classifier while freezing those of the feature extractor is common practice for assessing representation quality. We assume that the more task-specific knowledge the feature extractor learns, the better linear probing performance it exhibits in unseen domains targeting the same task. In detail, we first train GE on source domains and then evaluate linear probing performance on an unseen domain with the trained feature extractor of GE. We compare it with the case that frozen pre-trained model is used as the
7

---

Under review as a conference paper at ICLR 2023
feature extractor. For linear probing, simply train a logistic regression classifier the output feature representations of each feature extractor using the unseen domain only. Note that, this analysis, use CLIP and SWAG which pre-trained with objectives significantly different from the target task to demonstrate the effectiveness of the newly learned task-specific knowledge clearly.
Results. As shown in Table GE outperforms frozen all benchmark datasets except the one case where the two models reach the near 99% performance, This shows that GE is learning task-specinf knowledge further during training. which makes a better approximation of the oracle model. The result supports our claim that pre-trained models are not fully equipped with target task-specific knowledge, and injecting the knowledge further increases performance.
3.6
RELATIONSHIP BETWEEN AND THE SIZE OF THE PRE-TRAINED MODEL
Table 5: Evaluation results (%) on PACS with the three different pre-trained models varying X.
X
Dataset (size)
Pre-training
Architecture
0.01
0.05
0.1
0.5
ImageNet (1.3M)
ERM
ResNet-50
88.0 #002
86.0 #0.2
82.1 #0.2
73.4 #0.4
CLIP (400M)
CLIP
ViT-B/16
94.8 #0.2
96.0 +0.0
96.2 #0.1
96.0 =0.0
Instagram (3.6B)
SWAG
RegNecY-16GF
96.3 1...2
96.9 1.0.1
97.6 1:0.1
97.9 11011
Setup. Our proposed GESTUR controls the scale of the estimated unobservable gradients that reduce risks in unseen domains using the gradient scale factor A. To verify the effect of the scale factor, observe the performance change varying the scale factor.
Results. In Table 5 RN50 achieves the best performance with À - 0.01. On the other hand, the larger pre-trained models, CLIP and SWAG achieve the best performance with the relatively larger = 0.1 and A = 0.5, respectively. We summarize more results on other datasets (i.e. VLCS, OfficeHome, and Terralncognita) in Appendix C.T and they show the similar pattern as in PACS.
Intuitively, the larger pre-trained models act better approximation of the oracle model than the small one because they are likely to encounter various domains from the huge web-crawled datasets during pre-training. They help to estimate unobservable gradients more accurately. The larger gradient scale factor, gradients g TE more affected by the estimated unobservable gradients g.. while optimizing the model source domains. From this, we can conclude that the larger scale factor improves the generalization performance when larger pre-trained models are given.
RELATED WORK
4.1
DOMATN GENERALIZATION
Domain alignment. Domain alignment is to learn domain-invariin feature representations by removing domain-specific knowledge in the representations. Adversarial training is widely adopted to learn domain invariant features through min-max game between feature extractor and domain discriminator (Ganin et al.] 2016 Li et al.) 2018c; Matsuura & Harada 2020 Zhu et al.) 2022).
On the other hand, several studies (Muandet al., 2013 Sun Saenko 2016 Li et al. 2018b)
aim to minimize feature divergence across source domains. Recently, contrastive learning- algorithms (Kim et al.) [021] Yao al., (002)) have been proposed minimize distances between feature representations of samples in the same class, regardless their domains.
Data augmentation. Many studies have employed data augmentation techniques to improve do- main generalization performance. For example, Gulrajani & Lopez-Paz (2021) apply simple data augmentation techniques as a default setup in DOMAINBED and some studies (Wang et al.) 2020; Xu al. 2020; Yan et al., 2020) utilize Mixup (Zhang et al. 2017). Recently, few works (Zhoul et al. [221 Nam et al. 2021]] Kang et al.) 2022) focus on image style, based on the idea that domain gap closely related to image style. On the other side, some works single domain generalization
8

---

Under review as a conference paper at ICLR 2023
introduce adversarial data augmentation (Volpi et al. 2018 Fan et al. 2021 Qiao et al. 2020 to generate hard samples adversarially while assuring their reliabliity.
Gradient-based. Recently, several studies utilize gradients to build generalized models, especially by aligning gradients from different domains. Mansilla et al. (2021) exploit gradient agreement for gradient surgery, based on the hypothesis that conflicting gradients contain domain-specific infor- mation. Shi et al. (2022) propose a training method that maximizes inner product between source domain gradients
match optimization paths across domains. Similarly, |aae al.) (2022) try to match domain-level Hessian to align loss landscapes across domains. As another line of work, Huang et al. (2020) introduce the self-challenging algorithm that iteratively masks dominant fea- tures, which are selected by the scale of the gradients.
Meta-learning-basel Since simulating domain shift by dividing source domains into meta- meta-train and meta-test domains was first introduced in MLDG (Li et al. 2018a), several approaches have been proposed similar setting. For example, Balaji al. (2018) propose learn a regularizer for classifier weights and [Zhang et al.] (2021a) bring the idea of Reptile (Nichol et al. 2018) to MLDG to further increase performance with a multi-view framework. On the other hand, Zhang al. (2021b) employ meta-lcarning to adaptively predict model parameters from batch of inputs.
Others. Some of the works bring concepts of causality (Lv et al. 2022), optimize the worst-case performance (Sagawa et al.] 2019 Krueger et al. ). utilize text labels (Min et al.) (2022), or average model weights from different epochs (Cha et al. 20211] Arpit et al. 2022).
Our work differs from aforementioned approaches in that mainly concentrate effectively utilizing large-scale pre-trained models.
4.2 DOMAIN GENERALIZATION WITH PRE-TRAINED MODELS
Recently, Gulrajani& & Lopez-Paz [2021) empirically show that simple ERM (Vapnik 1999) outper- forms most of early methods with pre-trained ResNet-50 (He et al. 2016). Yu et al. (2021) show that using large-scale models pre-trained massive datasets improves out-of-disti performance. Kumar et al. (2022) find that fine-tuning distorts pre-trained features and propose the linear-probing then fine-tuning to mitigate the feature distortion. Wortsman al. (2022) find that linearly interpo- lating the zero-shot and fine-tuned parameters of a pre-trained model improves performance in both source and unseen domains. Although GESTUR's s EMA (Equation looks similar to their inter- polation, GESTUR updates the pre-trained model to inject task-specific knowledge. et al. (2022) propose a method efficiently leverage a pool of large-scale pre-trained models through specialty- aware ensemble learning. Chaa et al.] (2022) propose MIRO, regularization method that targets to minimize mutual information with pre-trained models which approximate the oracle model. this work, we share similar motivation with MIRO in that we initially approximate the oracle model with a large-scale pre- pre-trained model. However, we iteratively inject task-specific knowledge into the approximation of the oracle model, resulting in better approximation.
5 CONCLUSION AND FUTURE WORK
In this paper, we propose a new domain generalization method that learns task-specific knowledge while preserving the generalization ability of large-scale pre-trained models. We point out that gra- dient bias toward source domains hurts the generalization ability of pre-trained models during fine- tuning. To alleviate the gradient bias, our proposed method estimates unobservable gradients that minimize risk in unseen domains based on two key components: a task expert and generalization expert. Experimental results on DOMAINBED show that our proposed method outperforms baseline methods in domain generalization. Through extensive analyses, also demonstrate that the esti- mated unobservable gradients effectively reduce gradient bias, thereby helping to learn task specific knowledge without hurting the generalization power of large-scale pre-trained models.
Although we verify the effectiveness of our proposed method, it heavily relies on the capability of pre-trained models. When unseen domains that pre-traine traitrained models did not encounter are given (e.g. ResNet trained on ImageNet does not see medical images), the pre- pre-trained models might not act as an approximation of the oracle model of the domains. We will address this issue in future work.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide the source code for reproduction in the supplementary materials. See Appendix A for the hyperparameters used for the experiments.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02899, 2019.
Devansh Arpit, Huan Wang. Yingho Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Alice H. Oh, Alekh Agar- wal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /ttpht:s:///peneeeeee http://wwwweeeeeemmmmmmmmamaa
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- eralization using meta-regullarization. Advances in neural information processing systems, 31, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1): 46-100, 2021.
Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specifte features to en- hance domain generalization. Advances in Neural Information Processing Systems, 34:21189- 21201, 2021.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. arXiv preprint arXiv:2203. 10789, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 200.CCVVR.2009.5066555
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openzeview. wwtp:/eeteotteeeee
FicTTdNNTT
Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou. Adversari- ally adaptive normalization for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8208-8217, 2021.
Chen Fang. Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1657-1664, 2013.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversane training of neural net- works. The journal of machine learning research, 17(1):206-2000 2016.
Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021. URthttps: opwnreview..c. net forum? 1d=10dxeeDDa
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In European Conference on Computer Vision, pp. 124-140. Springer, 2020.
Juwon Kang, Sohyun Lee, Namyup Kim, and Suha Kwak. Style neophile: Constantly seeking novel styles for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1700-7140, 2022.
Dachee Kim, Youngjun Yoo. Seunghyun Park. Jinkyu Kim, and Jackoo Lee. Selfreg: Self- supervised contrastive regularization for domain generalization. In Proceedings the the IEEE/CVF International Conference on Computer Vision, pp. 9619-9628 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapo- lation (rex). In International Conference on Machine Learning, pp. 5815-5826, PMLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-iistributtin In International Con- ference on Learning Representations, 2022. URL
/wt/fffte nwtp:orrnttfff/t forum?
net
Td==UYnF2xSsJJ
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 32, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5400-5409, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings the European Conference on Computer Vision (ECCV), pp. 624-639, 2018c.
Ziyue Li, Kan Ren, Xinyang Jiang, Bo Li, Haipeng Zhang, and Dongsheng Li. Domain generaliza- tion using pretrained models without fine-tuning. arXiv preprint arXiv:2203.nl 046000 2022.
Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu. Causality inspired representation learning for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8046-8056, 2022.
Lucas Mansilla, Rodrigo Echeveste, Diego H Milone, and Enzo Ferrante. Domain generalization via gradient surgery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6630-6634 2021.
Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture of multiple latent domains. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11749- 11756, 2020.
Sconwoo Min, Nokyung Park, Siwon Kim, Seunghyun Park, and Jinkyu Kim. Grounding visual representations with texts for domain generalization. arXiv preprint arXiv:2207.1028: 2022.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
11

---

Under review as a conference paper at ICLR 2023
Hyeonseob Nam, HyumJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8690-8699 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-leamingg algorithms. arXiv preprint arXiv:1803. 02999, 2018.
Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556 12565, 2020.
Joaquin Quinonero-Candee
Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp.
PMLR, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEESCVF conference on computer vision and paltern recognition, pp. 10428-10436, 2020.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning, pp. 18347- 18377. PMLR, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint
2019.
Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In International Conference Learning Representations, 2022. http:tttttp/ppeeeeeeeeee
net/ wwwwoooowwiiwwwoooooeeee Hmo
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the ЛЕEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 804 814, June 2022.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pp. 443-450. Springer, 2016
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1521-1528. IEEE, 2011.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-5027, 2017.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018.
Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 3622-3626. IEEE, 2020.
12

---

Under review as a conference paper at ICLR 2023
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7959-7971, June 2022.
Minghao Xu, Jian Zhang. Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 6502-6509, 2020.
Shen Yan, Huan Song. Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001 100677. 2020.
Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang, Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7097-7107, 2022.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances Neural Information Processing Systems, 33:5824 0185824--836, 2020.
Yaodong Yu, Heinrich Jiang, Dara Bahri, Hossein Mobahi. Seungyeon Kim, Ankit Singh Rawat, Andreas Veit, and Yi Ma. An empirical study of pre-trained vision models on out-of-distribution generalization. In NeurlPS 202/ Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: (etp:111000913. 2017.
Jian Zhang, Lei Qi, Yinghuan Shi, and Yang Gao. domain generalization, 2021a
Mvdg: unified multi-view framework for
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Infor- mation Processing Systems, 34:23664-2361 2021b.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021. URL
Wei Zhu, Le Lu, Jing Xiao, Mei Han, Jiebo Luo, and Adam P Harrison. Localized adversarial domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2022.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
A IMPLEMENTATION DETAILS
Hyperparameter search strategy. Similar Cha al. 2022), the hyperparameter tuning strat- egy differs depending on which pre-trained model is used. In the experiments of this work, we use three different pre-trained models: ResNet-50 (He al. 2016) pre-trained on ImageNet (Deng al.] 2009) (RN50), ViT-B/16 (Dosovitskiy et al. 2021) with CLIP (Radford et al. 2021 (CLIP), and
RegG (Radosavovic et al. 2020) with SWAG (Singh et al. 2022 (SWAG).
Table 6: Hyperparameters used for RN50 in the experiments.
Hyperparameter
PACS
VLCS
OfficeHome
TerraInc
DomainNet
0.01
0.05
0.01
0.01
0.01
Learning rate
Se-5
5e-5
5e-5
5e-5
5e-5
Weight decay
0.0
lc-4
lc-6
0.0
le-4
Dropout
0.0
0.5
0.1
two-stage hyperparameter search strategy is used for experiments with RN50. Here, the batch size and the moving average coefficient (m) are fixed as 32 and 0.999 in the entire search procedure, respectively. In the first stage, we search the gradient scale factor (A) from {0.01, 0.05, 0.1, 0.5). In this stage, we fix the learning rate as 5e-5 and do not use weight decay and dropout (i.e. weight decay and dropout are equal to 0). In the second stage, we fix À as the one searched in the first stage. Then, search the learning rate from (1c-5, 3e-5, 5e-5], weight decay from {0, 1c-6, le-4), and dropout from {0, 0.1, 0.5). We provide the hyperparameters we use for RN50 in Table
Table 7: À used for CLIP and SWAG in the experiments.
Pre-trained Model
PACS
VLCS
OfficeHome
TerraInc
DomainNet
CLIP SWAG
0.05
0.1
0.05
0.05
0.05
0.05
0.1
0.05
0.05
0.05
Unlike the experiments with RN50, apply single-stage hyperparameter search strategy CLIP and SWAG. Here, we only search À from {0.01, 0.05, 0.1, 0.5} with hyperparameters such as the batch size, learning rate, weight decay, and dropout fixed. In particular, fix the learning rate, weight decay, and dropout as the first stage of the hyperparameter search of RN50. For the batch size, fix the batch size 32 except for the experiments with DomainNet (Peng al. 2019) where the batch size is fixed as 24 for the experiments with CLIP. For SWAG, we fix the batch size as 16 for all experiments. In Table 7 we show what we use for CLIP and SWAG. Similar to Cha eet al.) (2022), we fix the number of iterations as 15,000 for DomainNet and 5,000 for the others regardless pre-trained models.
14

---

Under review as a conference paper at ICLR 2023
B ADDITIONAL RESULTS
B. MAIN RESULTS
Table 8: Domain generalization accuracy (%) on the five domain generalization benchmark datasets with the three different pre-trained models. We mark *, i. and for the results from Gulrajani & Lopez-Paz 2022I). Cha et al. (202I) and Cha al. (2022) respectively. We use the reported numbers from each paper for Fish, Fishr, SelfReg, mDSDI, GVRT, and SMA.
Method
PACS
VLCS
OfficeHome
TerraInc
DomainNet
Avg.
Using ResNet-S0 pre-trained on ImageNet.
MMD
84.7 20.5
77.5 #0.9
66.3 10.1
42.2 11.
23.4 49.5
58.8
MixStyle"
85.2 #0.3
77.9 #0.5
60.4 10.3
44.0 +0.7
34,0 #0.1
60.3
GroupDRO
84.4
76.7 20.6
66.0
41.1
33.3
60.7
IRM
83.5 #0.8
78.5 20.5
64.3 #2.2
47.6.0..
33.9 42.8
61.6
ARM
85.1 #0.4
77.6 #0.3
64.8 #0.3
45.5 #0.3
35.5 #0.2
61.7
VREx
84.9 10.6
78.3 10.2
66.4 10.6
46.4 10.6
33.6 12.9
61.9
CDANN
82.6
77.5
41.3
45.8 11.6
38.3
62.0
DANN"
83.6 10.4
78.6 10.4
65.9 10.6
10.5
38.3 10.1
62.6
RSC
=0.2
77.1 #0.5
65.5 #0.9
41.6
38,9 +0.5
62.7
MTL*
80.5
77.2 #0.4
66.4
41.2
40.6 #0.1
62.9
Mixup"
84.6
77.4 #0.6
68.1
+0.8
39.2
63.4
MLDG
14.9
77.2 20.4
66.8
47.7
41.2
63.6
Fish
85.5 10.3
77.8 10.3
68.6 10.4
41.3
42.7 40.2
63.9
Fishr
85.5 10.4
77.8 10.1
67.8 10.1
47.4
41.7 40.0
64.0
ERM
84.2 10.1
77.3
67.6 10.2
47.8 10.6
44.0 10.1
64.2
SagNet"
86.3 10.2
77.8 10.5
10.1
11.0
40.3 40.1
SelfReg
85.6 #0.4
77.8 #0.9
67.9 10.7
47.0 +0.3
42.8 +0.0
64.2
CORAL
86.2 20.3
78.8
#0.6
68.7 20.3
47.6 +1.0
41.5 #0.1
64.5
mDSDI
86.2 20.2
79.0
20.3
69.2 20.4
41.
42.8 #0.1
65.1
GVRT
85.1 20.3
79.0
#0.2
70.1
48.0
44.1 10.1
65.2
MIRO
85.4 20.4
79.0 20.0
70.5 #0.4
50.4
44.3 #0.2
65.9
SMA
87.5 20.2
78.2
20.2
70.6 20.1
50.3 10.5
46.0
66.5
SWAD
88.1 #0.1
79.1
10.1
70.6 10.2
50,0 +0.3
46.5 #0.1
66.9
GESTUR
88.0 20.2
80.1
#0.2
71.1 10.0
51.3 +0.2
46.3 #0.1
67.4
Using ViT-B/16 with CLIP.
ERM
83.4 10.5
75.9 11.
66.4 +0.5
35.3 +0.8
44.4 40.6
61.1
SWAD
91.3 #0.1
79.4 20.4
76.9 10.1
45.4 10.5
51.7 +0.8
68.9
MIRO
95.6 =0.8
82.2 20.3
82.5 10.1
54.3 40.4
54.0 +0.3
73.7
GESTUR
96.0 #0.0
82.8 #0.1
84.2 #0.1
55.7 +0.2
58.9 +0.1
75.5
Using RegNetY- 1-GGF with SWAG.
ERM=
89.6 10.4
78.6 #0.3
71.9 +0.6
51.4 +1.
48.5 40.6
68.0
SWAD
94.7 10.2
79.7 20.2
+0.1
57.9 10.7
53.6 10.6
73.2
MIRO
97.4 #0.2
79.9 20.6
80.4 +0.2
58.9 +1.3
53.8 #0.1
74.1
SMA
95.5 20.0
80.7 20.1
82.0 10.0
59.7 10.0
60.0 10.0
75.6
GESTUR
96.9
83.5 10.1
83.1
61.1
60.1 10.0
76.9
In 3.2 only compare baselines superior to ERM /apnnk 1999) Here, we provide the entire results of the main experiment Table
with GESTUR for simplicity.
Baselines. the main experiment, we compare GESTUR against a number of baselines: MMD (Li et al. (2018)), MixStyle (Zhou al.) (021), GroupDRO (Sagawa et al.) 2019), IRM Ar- jovsky et al. | (019), ARM (Zhang et al.) (0221)) VREx (Krueger et al. 2021). CDANN (Li et al. 2018c), DANN (Ganin al. (016). RSC (Huang al. (020), MTL (Blanchard
al. 2021
Mixup (Wang et al. 2020 Xu et al. 2020 Yan et al. 2020), MLDG (Li et 20188))))) Fish (Shi al. 2022), Fishr (Rame al. 2022)
ERM (Vapnik 1999. Sag (Nam et al. 2021). Self-
15

---

Under review as a conference paper at ICLR 2023
Reg
Kim et al. 2021 CORAL (Sun & Saenko 2016).
mDSDI (Bui al., 2021). GVRT (Min
et a.. 2022). MIRO (Cha et al. 2022). SWAD (Cha et al.
2021). and SMA (Arpit al. 2022)
B.2
APPLICABILITY OF SWAD (CHA ET AL. 2021 TO GESTUR
Table 9: Evaluation results (%) of combination of SWAD and GESTUR the four datasets with the three different pre-trained models.
Method
PACS
VLCS
OfficeHome
TerraInc
Avg.
Using ResNet-50 pre-trained on ImageNet.
GESTUR
88.0 10.2
80.1 10.2
71.1 10.0
51.3 10.2
72.6 51.2 40.2
GESTUR + SWAD
88.3 10.1
80.1 #0.1
71.0 10.0
72.7
Using ViT-B/16 with CLIP.
GESTUR
96.0 40.0
82.8 #0.1
84.2 #0.1
55.7 10.2
79.7
GESTUR + SWAD
95.9 #0.0
82.8 #0.1
84.3 #0.0
55.3 #0.6
79.6
Using RegNetY-16GF with SWAG.
GESTUR
96.9 #0.1
83.5 #0.1
83.1 #0.0
61.1 +0.4
81.2
GESTUR + SWAD
96.8 10.0
83.0 10.1
83.4 10.1
60.6 10.8
81.0
Setup. The recent study (Cha et al. 2022) has observed that SWAD (Cha et al. 2021) that seeks the flat minima a good optimizer for domain generalization, improving the generalization performance of several baselines by applying it to the baselines as a optimizer. Motivated by this observation, we evaluate the performance of our GESTUR applied with SWAD optimizer verify whether GESTUR and SWAD are orthogonal directions to each other.
Results. Table shows that SWAD does not improve the performance of GESTUR. We conjecture that it is because EMA used transfer the knowledge of TE GE has similar effect as SWAD to find a flat minima by averaging the model's weights.
B.3 COMPARISON WITH CLIP-BASED BASELINES
Table 10: Evaluation results (%) on the four datasets with CLIP. Here, we compare GESTUR with CLIP-based baclines, CILP Zero-shot and WiSE-FT Wortsman et al. 2022).
Method
PACS
VLCS
OfficeHome
TerraInc
Avg.
CLIP Zero-shot
96.8 #0.0
81.7 10.3
83.0 40.3
31.3 40.2
73.2
WiSE-FT (a 0.5)
94.5 10.0
83.9 10.3
83.9 40.2
47.5 11.2
77.5
GESTUR
96.0 =0.0
82.8 +0.1
84.2 40.1
55.7
40.2
79.7
Setup. CLIP (Radford et al. 2021 pre-trained on the huge web-crawled image-caption pair dataset and has been widely adopted various computer vision tasks due its generalization abil- ity. CLIP-based methods could be strong baselines in domain generalization because the text content they used in pre-training could act as a robust anchor the domain shift of images. Therefore, conduct additional experiments using CLIP- based methods, CLIP Zhoo and WiSE-FT (Worts-) man et al. 2022). The CLiP-based methods require text-based queries to output text-based repre- sentations of target classes. Following the previous study, we obtain the 80 text-based queries from official r of CLIP and compute the final text-based representation of each target class
neeri ng_
for_ mageeet.t................ ipynb
16

---

Under review as a conference paper at ICLR 2023
by averaging the text-based representations of the queries. Finally, the model predictions are puted with the text-based representations and the representations of input images. For WiSE-FT, an ensemble of the fine-tuned and zero-shot models, we set the balance factor a 0.5 following its original paper since target unseen domains are inaccessible the domain generalization setting.
Results. Table 10 shows the evaluation results where GESTUR achieves the best averaged per- formance. In detail. GESTUR outperforms CLIP Zero-shot on VLCS, OfficeHome, and Terralne, and shows comparable performance on PACS. Likewise, GESTUR achieves better performance on PACS, OfficeHome, and Terralnc than WiSE-FT and comparable performance on VI.CS.
Interestingly, the CLIP-basee methods exhibit severe performance degradation on Terralnc. We conjecture that their performance is sensitive to pre-defined text-based queries. For example, the query "a sketch "a plastic { and "a {} in video game" are not helpful for TerraInc, which is composed of animal images taken from the wild. These observations indicate that the CLIP-based methods require hard prompt engineering for each target dataset. Moreover, the ELiP-based methods solely depend on CLIP, which cannot be extended to other architecture or learning methods trained on only visual modality, such ResNet with ImageNet and RegNet with SWAG. Considering these, our GESTUR achieves a meaningful performance.
{)" is helpful for the "Sketch" domain of PACS. On the other hand, the queries
17

---

Under review as a conference paper at ICLR 2023
C FURTHER ANALYSIS
C.1 RELATIONSHIP BETWEEN À AND THE TYPES OF THE PRE-TRAINED MODEL
Table 11: Evaluation results (%) VLCS with the three different pre-trained models varying X.
X
Dataset (size)
Pre-training
Architecture
0.01
0.05
0.1
0.5
ImageNet (1.3M)
ERM
ResNet-50
78.9 +0.3
80.1 #0.2
80.0 +0.1
77.6 =0.1
CLIP (400M)
CLIP
VIT-B/16
81.3 +0.4
82.7 #0.1
82.8 #0.1
82.1 #0.3
Instagram (3.68)
SWAG
RegNecY-16GF
81.7 40.0
82.7 40,2
83.5 #0.1
82.4 10.2
Table 12: Evaluation results (%) on OfficeHome with the three different pre-trained models varying X.
X
Dataset (size)
Pre-training
Architecture
0.01
0.05
0.1
0.5
ImageNet (1.3M)
ERM
ResNet-50
71.1 =:000
71.1 #0.1
70.4 :0.2
68.9 =0.1
CLIP (400M)
CLIP
VIT-B/16
82.5 #0.2
84.2 #0.1
84.4 #0.0
84.7 #0.0
Instagram (3.6B)
SWAG
RegNetY-16GF
81.5 +0.2
83.1 +0.0
83.5 +0.0
81.1 #0.1
Table 13: Evaluation results (%) Terralncognita with the three different pre-trained models vary- ing
À
Dataset (size)
Pre-training
Architecture
0.01
0.05
0.1
0.5
ImageNet (1.3M)
ERM
ResNet-50
51.3 +0.2
50.0 +0.4
45.5 +0.2
31.2 #0.1
CLIP (400M)
CLIP
ViT-B/16
51.3 +0.2
55.7 40.2
+4.3
42.3 #0.9
Instagram (3,68)
SWAG
RegNetY-16GF
57.6 40.9
61.1 10.4
62.1 10.3
54.9 10.1
In $ 3.6 we analyze the relationship between and the size of the pre-trained model. However, only present the results from PACS (Li al. 2017) in Table simplicity, Here, we provide the additional results from VLCS (Fang et al. 2013), OfficeHome /Venkateswara et al. 2017). and
TerraIncognita (Beery et al. 2018) Table Table T2 and Table 13 respectively.
C.2 PERFORMANCE ON SOURCE DOMAINS
Table 14: Evaluation results (%) on the four datasets with RN50. Here, we average the performances in the source domains, not the performance in the unseen target domain.
Method
PACS
VLCS
OfficeHome
TerraInc
Avg.
ERM GESTUR w/ TE
97.4 40.2
86.7 +0.1
82.9 40.3
92.2 #0.1
89.8
97.1
86.9 40.2
81.7 40.2
89.4 10.1
88.8
GESTUR w/ GE
98.2 #0.1
87.4 40.2
84.8 40.3
91.4 #0.1
90.5
Setup. Domain generalization aims to improve the generalization performance on unseen domains
shifted from source domains. Thus, domain generalization studies often do not consider situations where the source domains and the target domains are similar. To verify whether estimated unob- servable gradients useful when the unseen domains similar the source domains, report
18

---

Under review as a conference paper at ICLR 2023
the performance the saining-domaiie validation set, simulating the situations when the testing domains are exactly the same as the training domains.
Results. The evaluation results are summarized in Table 14 GESTUR w/ TE shows worse per- formance than ERM, indicating that the estimated unobservable gradients act as noisy gradients. Namely, gradients biased toward the source domains are more helpful than estimated unobservable gradients when the source domains and the target domains are similar. Nonetheless, GESTUR w/ GE performs better than ERM, demonstrating that our two-expert architecture robust to various situations even when source domains and unseen domains are similar or not.
C.3 SIMILARITY BETWEEN TRUE UNOBSERVABLE GRADIENTS gu AND ESTIMATED UNOBSERVABLE GRADIENTS g. OF GESTUR
Setup. In this paper, argue that gradient bias is major culprit in degrading domain general- ization performance (Figure Tab and our proposed method relieves the gradient bias by estimating unobservable gradients. To support this argument, we reported the number of iterations where gra- dient conflicts exist in Figure Tb and Table 3 To examine whether the estimated unobservable gradients ga are similar the true unobservable gradients gu. we add the analysis calculating the cosine similarity of the true and estimated unobservable gradients. Note that the true unobservable gradients are computed by cross-entropy loss using true labels of unseen domain datasets D.. On the other hand, the estimated unobservable gradients are just computed as the parameter difference between GE and (OGE- - OTE).
Results. Figure 2 shows that estimated gradients display positive similarity scores with the true gradients. This trend demonstrates that the estimated gradients reduce the number of gradient conflicts, leading models reduce the risks of unseen domains without accessing unseen domain data.
19

---

Under review as a conference paper at ICLR 2023
Art Paining
Cartoon
Photo
Sketch
Restict 50 (Imageket)
0.05
ResNit 50 lima
Restiet (imageNet)
ResNet- (ImageNit)
NeGF (SWWG)
ReggNet (SWAC)
RegNecY-15GF (SWAG)
RegNecY-16GF (SMAG)
cosine similami
6.63-
8.02
0.01
0.01
0.01
0.91
5.00
0.00
0.03
0.09
-0.03
0.00
0.03
-3.01
1000
1000
iterations
(a) PACS
Caltech101
LabelMe
SUN09
VOC2007
2.00
0.08
Roskitt 50 (InageMeti
ResNitt 50 lina pcNett
ResNet 50 (maggNet)
RosNet 50 (ImageNict)
3.05
VIT-8/16 (CLIP)
0.05
VIT-8/26
0.05
VIT-/116
3.55
cosine simlarity
0.04
(SWNG)
0.02
Re (SWAS)
0.04
RogNeeY (SWAG))
0.04
RogMety 16GF (SWAG)
5.09
0.03
0.01
2.00
0.00
0.02
0.00
-0.01
-0.01
-0.01
-0.01
1000
2000
3000
5000
1000
1000
itef terations
VLCS
Clipart
Product
Real World
2.04
0.04
0.04
5.04
Reshet-SC (Imagehet)
Restlee-S0 50 Time gellet
Restien-0 (mageNeet
Restatt-S (Imagekit)
VIT-8/16 (CLIP)
VIT-8/266 (CLIP)
VIT-8/16 (CUP)
VIE-8/15 (CLIP)
Cosine similarity
2.09
NetY-16GF
0
ReggNtt-16GF (SWAG)
RogNetV1s5F (SWAG)
RegNNeYY-edFF (SWAG)
0.01
0.01
2.00
0.00
0.00
5.00
-5.01
2002
2002
2000
4000
5200
1000
2000
2000
4000
5000
1000
2202
3000
4000
5000
1000
2020
3002
4002
5000
iterations
(c) OfficeHome
L100
L38
L43
L46
5.04
0.04
0.04
0.54
Reskict 50 (ImageNeti
Restiet 50 Hima geNeti
Restiee 50 lmageNeti
Restltt 50 (imageNet)
VIT--U16 (CLIP)
VIT-8/16 (CLI))
VIT-0/16 (CUP)
VIT-8/16 ICLIP)
cosine simllarity
NeGY-16GF (SWWAG)
Rerghet1-16GF (SWAG))
RegNeeY-15GF (SWAG)
RegNNecY-16GF (SWAG)
3.02
6.01
0.01
0.01
0.01
3.00
0.00
0.00
9.00
1002
2002
3003
4000
5200
1050
2030
2000
4000
5000
1000
2003
3000
4000
50.0
30000
7000
3002
4003
5000
Iterations
(d) Temrllne
Figure 2: Cosine similarity between the true unobservable gradients gu and the estimated unobserv- able gradients g.
20


Reviews:
[{"review_id": "d5bo5V7aSz", "reviewer": "Reviewer_e3A6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of domain generalization. The authors propose to reduce the gradient conflict between the gradient biased toward the source domains and the unobservable gradient that could minimize risks in unseen domains. Specifically, the authors leverage the large-scale pre-trained model as a loose approximation of the oracle model for the unseen domains and propose a simple yet effective method called GESTUR. In GESTUR, a fast-update model called task expert (TE) is directly optimized on source domains with calibrated gradient. A slow-update model called generalization expert (GE) is EMA updated by TE and provides the approximated unobservable gradient for calibration. Empirical results on five popular domain generalization benchmarks show that GESTUR outperforms other methods.", "summary_of_the_review": "This paper proposes a simple yet effective method for domain generalization. Extensive experiments well justify the advantage of the proposed method. My main concern is how to select the sensitive hyperparameter $\\lambda$ without accessing unseen target domain data.\n\n***********Post-rebuttal**************\nAfter reading all the discussions and reviews, I share the same concern on the novelty of this submission as Reviewer seAG and Reviewer TqRb. Also, I also agree with Reviewer ENaU that the feasibility and reliability of using large-scale pre-trained models as a loose approximate of unseen domains are not provable, limiting the proposed method. Therefore, I decided to decrease my rating to 5.", "strength_and_weaknesses": "### Pros\n1. The proposed method GESTUR is simple and well-motivated.\n\n2. GESTUR is competitive on various domain generalization benchmarks, verified by empirical results.\n\n3. The analysis and ablation are comprehensive, well justifying the proposed method.\n\n### Cons\n1. The hyperparameter $\\lambda$ is very sensitive to domain generalization tasks, as verified by Table 5 and B.2 in the appendix. I am still confused about how to choose $\\lambda$? What does \"while those of the unseen domain are used for hyperparameter search.\" in Section 3.1 mean? DomainBed provides three model/hyperparameter selection protocols, which protocol does this paper adopt, source validation set, oracle test-domain validation set, or leave-one-domain-out cross-validation?\n\n2. In MIRO paper, MIRO can combine with SWAD to further improve the generalization performance, average accuracy of 68.1 for the ResNet-50 model. How about GESTUR in this paper? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X_Sn1avF1UD", "reviewer": "Reviewer_ENaU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets the domain generalization task. Based on the observation that fine-tuning may introduce gradient bias and hurt generalization ability, the paper estimates unobservable gradients that reduce potential risks in unseen domains. The main requirement is that there is a pre-trained model. Experimental results show that the proposed method outperforms baseline methods on DomainBed.\n", "summary_of_the_review": "The overall quality of the paper is good, and the idea is interesting. The two main concerns of the paper are:\n1. Theoretical or empirical analysis should be provided to support the motivation of the paper;\n2. How to design a fair comparison with other methods since a pre-trained model is included.\n", "strength_and_weaknesses": "This paper proposes to estimate the unobservable gradients from a pre-trained model, which helps the model generalization across various domains. Results on various configurations show the ability of the proposed method. \n\nThere are some concerns with the paper:\n1. The main idea of the paper is to reduce the influence from the source domain gradients and make the model generalize better. However, since we do not know which is the target domain, does it requires some kind of prior knowledge encoded in the pre-trained model? For example, if the target domain is very similar to (or distant from) the source domain, the pre-trained model will produce the same unobservable gradients, but not both of them help the generalization.\n2. Do the experiments in Figure 1 depend on the relationship between the source and target domain? Could the authors show the similarity between the true gradient to the target domain and the gradient estimated by the pre-trained model?\n3. Based on the results in Table 3.6 and supplementary, the value of $\\lambda$ depends on the size of the pre-trained model. Does it relate to the similarity between the source and target domains? Or the diversity of the dataset of the pre-trained model?\n4. Please consider more comparison methods in the experiments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T61GK5N2GPG", "reviewer": "Reviewer_TqRb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores domain generalization by estimating gradients of unobserved domains using large-scale pretrained models. The method learns task-specific knowledge for the pretrained model while preserving its generalization ability with the estimated gradients by EMA. Results on several benchmarks on DomainBed show the effectiveness of the proposed method.", "summary_of_the_review": "The paper proposes to fine-tune the large-scale pretrained model for domain generalization by learning task-specific knowledge from source domains while keeping the generalization ability of the pretrained models. But the motivation and justification behind the method are not clear and well supported by experiments. The technical novelty of the method seems weak to me. Therefore, I think it is below the acceptance threshold. ", "strength_and_weaknesses": "[+] The paper propose to use EMA to simultaneously learn the task-specific knowledge and preserving the generalization ability of large-scale pretrained model for domain generalization, which is interesting.\n\n[+] The ablation studies and comparisons on several domain generalization datasets  support the effectiveness of the method.\n\n[-] The paper uses pretrained models as the loose approximation of the oracle model of the unseen domains and and estimates the unobserved gradient of unseen domains. But it is not clear why the pretrained models is able to approximate the oracle model. For example, how can an ImageNet-pretrained ResNet can be the oracle model for unseen domains like “sketch”? More explanations and discussions are needed for this motivation.\n\n[-] The reviewer is also concerned about the novelty of the proposed method. The estimated gradient in eq. 4 looks more like a regularization that protects the model from changing too much compared with the originally pretrained parameters, which may hurt its generalization ability. Eq. 6 also seems to be doing the same thing. However, this kind of techniques have already been explored in some continue learning methods to deal with catastrophic forgetting.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MibCPyDD5f", "reviewer": "Reviewer_seAG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple method that leverages the generalization ability of large pre-trained models for domain generalization. The key idea is to use large pre-trained models to estimate the unobserved gradient that minimizes the risks in the target domain, mitigating the gradient biased towards to the source domain during the fine-tuning stage. Experimental results show that the approach is promising despite its simplicity.", "summary_of_the_review": "Overall, the motivation and the method design is interesting. However, the paper currently lacks proper comparisons to related work, making the contribution less strong as it stands.", "strength_and_weaknesses": "Strengths:\n- The proposed method is simple but shows promising results\n- The key idea is well-motivated: large pre-trained models have strong generalization ability and fine-tuning impairs this characteristic; thus, one can do fine-tuning more conservatively to retain the models generalization ability.\n\nWeaknesses:\n- One main concern I have about the paper is that the proposed method is very related to a recent work [1], that proposed to linearly interpolate a fine-tuned model’s weight with the un-tuned model’s weight. The only difference lies in that [1] does the interpolation “after” the model is fine-tuned, while the proposed method in this paper does the interpolation “during” fine-tuning (in the optimization steps). [1] is only very briefly mentioned in the related work section. However, I believe properly comparing to [1] in the paper on both the method design as well as experimental results is needed to make the contribution of this work stronger.\n- In the experiments, the evaluation metric only measures the model performance on the unseen domain. However, it’s generally also important to see the model performance on the source domain. It will give a more complete picture on the pros/cons of different methods. For example, maybe the simple ERM can achieve best performance on source domain by trading-off only a bit of its performance on target domain; while methods designed to optimize for target domain may trade-off more performance on source domain.\n- I appreciate that the authors have conducted the analysis in Sec 3.4 to validate the original hypothesis of biased gradient. However, more details on the experimental setup are needed. Specifically, how are the gradients on the unseen domain calculated? Is it by using the actual $(x, y)$ pairs in the unseen domain?\n- Another relevant baseline to consider in the experiments is the zero-shot model baseline. Particularly, with models like CLIP, one can directly do zero-shot predictions on the unseen domain. Will such a model without any fine-tuning achieve even better performance compared to the proposed method?\n- Paper presentation can be improved. There are many repetitive texts in the Introduction section and the Method section. For example, the paragraph on “Task expert and generalization expert” in the Method section is repeated in the Introduction. Authors could use the space to provide more insights to the method design.\n\n[1] Robust fine-tuning of zero-shot models. Wortsman et al. 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject