Domain Generalisation via Domain Adaptation: An Adversarial Fourier Amplitude Approach
Authors: Minyoung Kim
Da Li
Timothy Hospedales
Venue: ICLR 2023 poster (2023)
Keywords: Domain generalisation
Domain adaptation
Fourier analysis

Abstract: We tackle the domain generalisation (DG) problem by posing it as a domain adaptation (DA) task where we adversarially synthesise the worst-case `target' domain and adapt a model to that worst-case domain, thereby improving the model’s robustness. To synthesise data that is challenging yet semantics-preserving, we generate Fourier amplitude images and combine them with source domain phase images, exploiting the widely believed conjecture from signal processing that amplitude spectra mainly determines image style, while phase data mainly captures image semantics. To synthesise a worst-case domain for adaptation, we train the classifier and the amplitude generator adversarially. Specifically, we exploit the maximum classifier discrepancy (MCD) principle from DA that relates the target domain performance to the discrepancy of classifiers in the model hypothesis space. By Bayesian hypothesis modeling, we express the model hypothesis space effectively as a posterior distribution over classifiers given the source domains, making adversarial MCD minimisation feasible. On the DomainBed benchmark including the large-scale DomainNet dataset, the proposed approach yields significantly improved domain generalisation performance over the state-of-the-art.

Published as a conference paper at ICLR 2023
DOMAIN GENERALISATION VIA DOMAIN ADAPTATION: AN ADVERSARIAL FOURIER AMPLITUDE APPROACH
Minyoung Kim', Da Li & Timothy M. Hospedales 1. Samsung Al Center Cambridge, UK [mikim21, dali. academi c) @0mmagmmlll.... com
2 University of Edinburgh, UK hospedaeeeeedddcco uk
ABSTRACT
We tackle the domain generalisation (DG) problem by posing as a domain adap- tation (DA) task where we adversarially synthesise the worst-case 'target' domain and adapt a model to that worst-case domain, thereby improving the model's robust- ness. To synthesise data that is challenging yet semantics preserring,..... Fourier amplitude images and combine them with source domain phase images, exploiting the widely believed conjecture from signal processing that amplitude spectra mainly determines image style, while phase data mainly captures image semantics. To synthesise a worst-case domain for adaptation, we train the classifier and the amplitude generator adversarially. Specifically, we exploit the maximum classifier discrepancy (MCD) principle from DA that relates the target domain performance to the discrepancy of classifiers in the model hypothesis space. By Bayesian hypothesis modeling, we express the model hypothesis space effectively as posterior distribution over classifiers given the source domains, making ad- versarial MCD minimisation feasible, On the DomainBed benchmark including the large-scale DomainNet dataset, the proposed approach yields significantly improved domain generalisation performance over the state-of-the tfath-rrr...
generate
INTRODUCTION
Contemporary machine learning models perform well when training and testing data are identically distributed. However, in practice it is often impossible to obtain an unbiased sample of real-world data for training, and therefore distribution-shiftt inevitably exists between training and deployment. Performance can degrade dramatically under such domain shift (Koh et al. 22021), and this is often the cause of poor performance of real-world deployments (Geirhos et al. 2020). This important issue has motivated a large amount of research into the topic of domain generalisation (DG) (Zhou et al. 2021a), which addresses training models with increased robustness to distribution shift. These DG approaches span diverse set of strategies including architectural innovations (Chatooaahyyy et al. 2020), novel regularisation (Balaji et al.) 2018). alignment (Sun & Saenko 2016) and learning (Li al. 2019) objectives, and data augmentation (Zhou et al., 2021b) to make available training data
more representative of potential testing data. However, the problem remains essentially unsolved, especially as measured by recent carefully designed benchmarks (Gulrajani & Lopez-Pazz Paz 2021).
Our approach is related to existing lines of work on data-augmentation solutions to DG (Zhou et al. 2021b; Shankar et al. 2018), which synthesise more data for model training; and alignment-based approaches to Domain Adaptation (Sun & Saenko, 2016 Saito et al.] 2018) that adapt a source model to an unlabeled target set but cannot address the DG problem where the target set unavailable. We improve on both by providing a unified framework for stronger data synthesis and domain alignment.
Our framework combines two key innovations: Bayesian approach to maximum classifier dis- crepancy, and a Fourier analysis approach to data augmentation. We start from the perspective of maximum classifier discrepancy (MCD) from domain adaptation (Ben- function of discrepancy between multiple
et al. 2007 2010 Saito
et al. 2018). This bounds the target-domain error source-domain domain classifiers. is not obvious how to apply MCD to the DG problem where we have no access to target-domain data. A key insight is that MCD provides a principled objective that can maximise in order to synthesise a worst-case target domain, and also minimise in order to train a model that is adapted to that worst-case domain. Specifically, we take Bayesian approach that learns

---

Published as a conference paper at ICLR 2023
Amp.
y-fforse
Phose
N(O.1)
G2()
F()
Post-mixup
(
Generated target domain
(-
Amp
Source domain
y-House
y=House
-ELBO Loss
SMCD Loss
P 4
y=Dog
y=
Feature Extractor Qu()
FF
Params 2
Fwd pass
Back prop
Figure 1: Overall training flow of the proposed approach (AGFA). We generate target-domain data by
synthesizing Fourier amplitude images trained adversarially. See main text Sec. for details.
distribution over source-domain classifiers, with which we can compute MCD. This simplifies the model by eliminating the need for adversarial classifier training in previous applications of MCD (Saito et al. 2018), which leaves free adversarially train the worst-case target domain. To enable challenging worst-case augmentations to be generated without the risk of altering image semantics, our augmentation strategy operates the Fourier amplitude domain. It synthesises amplitude images, which can be combined with phase images from source-domain data to produce images that are substantially different in style (amplitude), while retaining the original semantics (phase). Our overall strategy termed Adversarial Generation of Fourier Amplitude (AGFA) is illustrated in Fig. [
In summary, we make the following main contributions: (1) We provide novel and principled perspective on DG by drawing upon the MCD principle from DA. (2) We provide AGFA, an effective algorithm for DG based on variational Bayesian learning of the classifier and Fourier-based synthesis of the worst-case domain for robust learning. (3) Our empirical results show clear improvement on previous state-of-the-ee on the rigorous DomainBed benchmark.
PROBLEM SETUP AND BACKGROUND
We follow the standard setup for the Domain Generalisation (DG) problem. As training data, we are given labeled data S = (xr, (xr v) ~ D;,
.,....,,. where X and y E ) (I,..., C). Although the source domain S consists of different domains { J with domain labels available, we simply take their union without using the originating domain labels. This is because in practice the number of domains (N) is typically small, and it is rarely possible to estimate meaningful population distribution for empirical from a few different domains. What distinguishes DG from the dosely-related (unsupervised) Domain Adaptation (DA), is that the target domain (7) on which model's prediction performance measured is unknown for DG, whereas in DA the input data æ from the target domain are revealed (without class labels y). Below we briefly summarise the MCD principle and Ben-David's ' theorem, one of the key theorems in DA, as we exploit them to tackle DG.
Ben-David's theorem and MCD principle in DA. In unsupervised DA, Ben- Dev-D'sid's theorem (Ben) David et al. 2010; 2007) provides an upper bound for the target-domain generalisation error of model (hypothesis). We focus the tighter bound version, which states that for any classifier h in the hypothesis space H {h[h: X + )). the following holds (without the sampling error term):
er (h) < es(h)
sup b.h'HH
(ds(,, h²) dr (h, h") + (HT,,
(1)
where (h) == B + y)] is the error rate of h(-) on the source domain S, ds R²) = Bews([(h(()) + H (x)) denotes the discrepancy between two classifiers h and h' on S (similarly for er(h) and (r, k')), and SH; 7) := minkeH (h) er(h). Thus we can provably reduce the target domain generalisation error by simultaneously minimizing the three terms in the upper bound" namely source-domain error eg (h), classifier discrepancy, and minimal source-target emror.
Previous approaches (Saito et al. 2018 Kim al., 2019) aim to minimise the upper bound, and one reasonable strategy is to constrain the hypothesis space H. in such way that it contains only those
'Soe recent work such (Vedantam al. 2021 of the bound in certain scenarios.
however, empirically studied potential risk of looseness
2

---

Published as a conference paper at ICLR 2023
h's with small (h)). Within this source- hypothesis space (denoted by His), the terms es(h) and ds(h, h²) in the bound are expected to be close to O for all h, h' E Hiss and the bound of ①T effectively reduces to what is called the Maximum Classifier Discrepancy (MCD) loss,
MCD(H T) ==
sup
(dr(hh)))
sup h
REZT
[T(b(x)) ( + (((x)))
K
This suggests the MCD learning principle: we need to minimise both the error on S (so as to form the source-confined hypothesis space H(s) and the MCD loss on T. Note however that the last term e* is not considered in (Saito al. 2018 Kim al., 2019) mainly due to the difficulty of estimating the target domain error. We will incorporate E" in our DG algorithm as described in the next section.
We conclude the section by briefly reviewing how the MCD learning principle was exploited in previous works. In (Saito et al., 2018) they explicitly introduce two classifier networks h(z) = g(((x=) and h'(x)= = g' ()))). where the classification heads g. g' and the feature extractor are cooperatively updated to minimise the error on S (thus implicitly obtaining H(s)), they are updated adversarially to maximise (minimise) the MCD loss on T with respect to and g (6. respectively). In (Kim al. 2019), they build Gaussian process (GP) classifier on the feature space ((x)), in which His is attained by GP posterior inference. Minimisation of the MCD term is then accomplished by the maximum margin learning which essentially enforces minimal overlap between the two largest posterior modes. Note that (Saito et al. 2018)'s 's strategy requires adversarial optimisation, and hence is less suitable for our DG algorithm which will require adversarial generator learning: Having two adversarial learning components would make the training difficult since we need to find two nested equilibrium (saddle) points. We instead adopt the Bayesian hypothesis modeling approach of (Kim
al. 2019). In the next section, we describe our approach in greater detail.
3 ADVERSARIAL GENERATION OF FOURIER AMPLITUDE (AGFA)
Defining and optimising a hypothesis space. Our DG approach aims minimise the MCD loss, MCD(H)s: 7) defined in ( The first challenge is that the target domain data T' is not available in DG. Before we address it, we clarify the optimisation problem (i.e., what the MCD loss optimised for?) and how the hypothesis spaces (H and H(s)) are represented. The MCD loss is a function of hypothesis space H (or s) not function of individual classifier h in it. Hence, minimising the MCD loss amounts to choosing the best hypothesis space H. To this end, we need to parametrise the hypothesis space (so as to frame as a continuous optimisation), and our choice is the Buyesian linear classifier with deterministic feature extractor.
We consider the conventional neural-network feed-forward classifier modeling: we have the feature extractor network (x(z) E Rdd (with the weight parameters 0) followed by the linear classification head W = [W]..... , wo] (C-way classification, each Wj- E IRd), where the class prediction is done by the softmax likelihood:
Ply j,,, W) & e"] j=1...,C..
So each configuration (0, W) specifies particular classifier h. To parametrise the hypothesis space H (3 h),, ideally we can consider a parametric family of distributions over (0, W). Each distribution (O W) specified by the parameter B corresponds to a particular hypothesis space H. and each sample (0, W) ~ (0 W) corresponds to a particular classifier h E 'H. Although this is conceptually simple, to have a tractable model in practice, we define 0 to be deterministic parameters and only to be stochastic. A reasonable choice for P(W), without any prior knowledge, is the standard Gaussian, P(W)- = NI==NN(uu 0, I).
Now, we can represent a hypothesis space as 7 = [P(y A, W) W ~ P(W)). Thus H is parametrised by 0, and with fixed (H fixed), each sample W from P(W) instantiates a classifier h E H. The main benefit of this Bayesian hypothesis space modeling is that we can induce the source-con cource-confied hypothesis space His (i.c., the set of classifiers that perform well on the source domain) in a principled manner by the posterior,
((W|S,0) x P(W)-
II (x.y)~
P(y|æ, 6, W)
The posterior places most of its probability density on those samples (classifiers) W that attain high likelihood scores S (under given 0) while being smooth due to the prior. To ensure that the source
3

---

Published as a conference paper at ICLR 2023
domain S is indeed explained well by the model, we further impose high data likelihood on S as constraints for 8,
0 E O where θs {0) log P(S]0) 2 Len),
(5)
where Len is the (constant) threshold that guarantees sufficient fidelity of the model to explaining S. Then it is reasonable represent His by the support of P(WIS, 0) for 0 E Os, postulating that His exclusively contains smooth classifiers that perform well on S. Formally, the source-confined hypothesis space can be parametrised as:
(0)( = (P(y)z 8, W W ~ P(W|S, 0)} for E 0s,
(6)
where we use the notation His (0) emphasise its dependency on 9. Intuitively, the hypothesis space His is identified by choosing the feature space (i.e., choosing A E ④s). and individual classifiers h E His are realised by the Bayesian posterior samples W ~ P(W SS, 0) (inferred on the chosen feature space). Since the posterior P(W|S. [WWS,,0 8) in (6) and the marginal likelihood log P(S]0) in (5) do not admit closed forms in general, adopt the variational inference technique to approximate them. We defer the detailed derivations (Sec. 3.11) for now, and return to the MCD minimisation problem since we have defined the hypothesis space representation.
Optimising a worst-case target domain. learning principle since the target domain T is unknown during the training stage. Our key idea is to consider the worst-case scenario where the target domain T maximises the MCD loss. This naturally forms minimax-type optimisation,
For the DG problem, we cannot directly apply the MCD
min max (OD()) EE9g T'
To solve the saddle-poing point optimisation (7) we adopt the adversarial learning strategy with a generator network (Goodiellow al. 2014). The generator for T' has synthesise samples x of T'' that need to satisfy three conditions: (C1) The generated samples maximally baffle the classifiers in H4s to have least consensus prediction (for inner maximisation); (C2) T still retains the same semantic class information the source domain S (for the definition of DG); and (C3) The generated samples in T need to be distinguishable along their classes
Paramaterising domains. To meet these conditions. we generate target domain images using Fourier frequency spectra. We specifically build generator network that synthesises amplitude images in the Fourier frequency domain. The synthesised amplitude images are then combined with the phase images sampled from the source domain S to construct new samples x E T by inverse Fourier transform. This is motivated by signal processing where it is widely believed that the frequency phase spectra capture the semantic information of signals, while the amplitudes take charge of non-semantic (c.g., style) aspects of the signals (Oppenheim & Lim 1981). Denoting the amplitude generator network as (e) with parameters y and random noise input e ~ NO I). our target sampler (x, v) T are generated as follows:
1. (IS)
(Sample an image and its class label from S)
2. AsLP3 = F(xs)
(Fourier transform to have amplitude and phase for xz)
3. G,((), ~~ A((0
(Generate an amplitude image from G')
4. x F (ALPs) == ys
(Construct target data with the synthesised A)
Here, F(.) is the 2D Fourier transform, F(u, u) = F(x) F(x) = ʃʃ æ(h, w)e =(hu-(()+ dhdw, and A/P stands for the polar representation of the Fourier frequency responses (complex numbers) for the amplitude image A and the phase image P. That is, ALP = e- = A (cos P + iss P) with i == V-1, where all operations are clement/pixel-win wise. Note that we set = ys in step 4 since the original phase (semantic) information Ps retained in the synthesised
Algorithm summary. Finally the worst-case target MCD learning can be solved by adversarial learning, which can be implemented an alternating optimisation:
(Fix v) min (CD(H( BEOs
(8)
(Fix 0) max MCD(H,,(0):T(v)))
(9)
Thii condition naturally originates from the solvability of the DG problem.
4

---

Published as a conference paper at ICLR 2023
We used T(v) to emphasise functional dependency of target images on the generator parameters Note that although the MCD loss in DA can be computed without the target domain labels (recall the definition (2) in DG case the class labels for the generated target data are available, as induced from the phase Ps (i.e., y = Vs in step 4). Thus we can modify the MCD loss by incorporating the target class labels. In the following we provide concrete derivations using the variational posterior inference, and propose a modified MCD loss that takes into account the induced target class labels.
3.1 CONCRETE DERIVATIONS USING VARIATIONAL INFERENCE
Source-confined hypothesis space by variational inference. The posterior P(W|S, 0) does not admit closed form, and we approximate P(W|S,6) 0) by the Gaussian variational density.
C Qx(W) = 11 N(wymmy
(10)
where ;= (mj, V=Jj=1 constitutes the variational parameters. To enforce Qx(W) P(W S, we optimise the evidence lower bound (ELBO),
(L: = Σ
Bex P(y (, W. KL((X(W>IP((W)) (11)
(2.8)~uS
which is the lower bound of the marginal data likelihood log P(S () (Appendix A.3 for derivations). Hence maximising ELBO(), 0; S) with respect to À tightens the posterior approximation Qx(W) R
P(W|S, 0). while maximising it with respect 0 leads high data likelihood log 0). The latter
has the very effect of imposing the constraints 0 E O in (8) since one can transform constrained optimisation into a regularised (Lagrangian) form equivalently (Boyd & Vandenberghe 2004
Optimising the MCD loss. The next thing is to minimise the MCD loss, MCD(H (f): T) with
the current target domain T generated by the generator network G.- That is, solving (8). We follow the maximum margin learning strategy from (Kim et al. 2019). where the idea is to enforce the prediction consistency for different classifiers (i.e., posterior samples) W ~ QW) on x T by separating the highest class score from the second highest by large margin. To understand the idea, let j be the model's predicted class label for x ~ T. or equivalently let j" have the highest class score j = arg max, w. Φ as per (3). (We drop the subscript in Φe for simplicity in notation.) We jet be the second most probable class, i. arg maxj w. ((x)). Our model's class prediction would change if w 6(x) < w. o(x) for some W ~ Qx(W), which leads to discrepancy of classifiers. To avoid such overtaking, we need to ensure that the (plausible) minimal value of w. ((x) is greater than the (plausible) maximal value of w o(x). Since the score (logit) [x) == w3 (zx) is Gaussian under (W),,, namely
(x) (z) where p,(x) m3 (x)), of(x) d(x) ( Vjd(z),
(12)
the prediction consistency achieved by enforcing: 11-- - (x) > (2)(2)) + ag,, y1 (x)) where we can choose = 1.96 for 2.5% rare one-sided chance. By introducing slack variables {(x) 2 0,
My (x) a (x) 2 1 max (1z(2) + (z) -(x).
(13)
Satisfying the constraints amounts to fulfilling the desideratum of MCD minimisation, essentially imposing prediction consistency of classifiers. Note that we add the constant 1 in the right hand side of (13) for the normalisation purpose to prevent the scale of μ and a from being arbitrary small. The constraints in
13) can be translated into the following MCD loss (as a function of 0):
MCD := Ez-T (1 (1+ (mg(2)) + az)())) T (1z(x) ao ao, xx)(
(14)
where T is the operator that selects the top-k element, and (a)+ max (0. a).
Modified MCD loss. The above MCD loss does not utilise the target domain class labels y ys that are induced from the phase information Ps (Recall the target domain data generation steps 1 4 above). To incorporate the supervised data ((x, v)} E T in the generated target domain, modify
5

---

Published as a conference paper at ICLR 2023
A,00 (966
P2(R3
Mr00 1.966,
8,00
Logit score f/(0)
--,600 1960
-..L 1969
Rg 1.960;(t)
1260
Ps
-, 1965,(())
Mg633
4g
Olass 1
Class
Class 3
Class 1
Class Class 3
Class
Class 2
Class
Figure 2: Illustration of the SMCD loss on three different hypothesis spaces His shown in three panels. For C = 3-way classification case, each panel shows the class logit scores (Gaussian random) [z( ~ N (
for = 1, 2, 3, at some input æ E T. We assume that the true (induced) class label y = 2. (Left) Since the mean logit for class 2. (12(z) is the maximum among others, the prediction is marginally correct (from softmax). Beyond that, the logit of the worst plausible hypothesis for class 2, (2) ( is greater than that of the runner-up class 1, N1 (x) + 1.9601(x) (I:) by some positive margin (green arrow), meaning there is little chance of prediction overtaking (so, consistent); equivalently. the SMCD loss is small. (Middle) Prediction is marginally correct, but prediction overtaking is plausible, indicated by the negative margin (red arrow); the SMCD loss is large. (Right) Incorrect marginal prediction (to class 1) with more severe negative margin (red arrow); the SMCD loss is even larger.
the MCD loss as follows: First, instead of separating the margin between the two largest logit scores as in the MCD, we maximise the margin between the logit for the given class y and the largest logit among the classes other than That is, we replace the constraints (13) with the following:
My(x) - - ( 2 + max (1)(2) + ao, (x)(x)) - E(x),
(15)
2f4
where y is the class label (induced from the phase information) for the generated instance r. See Fig. for illustration of the idea. Consequently, our new MCD loss (coined supervised MCD or SMCD for short) is defined as follows:
- ao, SMCD T) == (( (1+ max (1)(2) + az)(x)) - (1)(#))oo((#))) ao,( 3f4
(16)
Here the variational parameters À is treated as constant since the only role of À is to maximise the ELBO. It should be noted that (16) essentially aims at maximising the logit for the given class y (the last term), or equivalently, classification error minimisation on T,, and the same time minimising the logit for the runner-up class (the middle max term). Surprisingly, the former amounts to minimising the minimal source-target error term e* (H; S, 7) in the generalisation bound ( which we have left out so far. That is, e (H; S, S) minKER (h) + er)hh) = min er (h), and the last term of the SMCD loss leads to 0 that makes (r(h) small for all h E (0). Moreover, minimising the logit for the runner-up class (the middle max term of the SMCD) has the effect of margin maximisation.
Algorithm summary. Our AGFA algorithm can be understood as MCD-based DA with adversarial amplitude generated target domain. It entails the following alternating optimisation (1 > 0 is the trade-off hyperparameter for SMCD):
1. minx, -ELBO(A, S) + 7/SMCD(0; (model learning + VI; v fixed) 2. max, SMCD(0;TT)
(adversarial generator learning; 0, A fixed)
Our algorithm is summarised in Alg. (in Appendix) and illustrated schematically in Fig. [ At test time, we can apply the classifier (3) with the learned 0 and any sample W Qx to target domain inputs to predict class labels. In our experiments, we take the posterior means Wj = mj instead of sampling from (x))..
3.2 FURTHER CONSIDERATIONS
Post-synthesis mixup of generated amplitude images. In our adversarial learning, the ampli- tude generator network G2 synthesises target domain image samples that have highly challenging amplitude spectra to the current model. Although we retain the phase information from source domains, unconstrained amplitude images can potentially alter the semantic content destructively (e.g., a constant zero amplitude image would zero out the image content), rendering it impossible to classify. To this end, instead of using the generator's output A (e( directly, we combine it with

---

Published as a conference paper at ICLR 2023
the source domain amplitude image corresponding to the phase image by simple mixup. That is, by letting Ag be the amplitude spectra corresponding to the phase Ps, we alter A as:
- AA (1 As where À ~ Uniform(O, cv).
(17)
This post-synthesis mixup can address our desideratum C3 that we discussed before, that is, the generated samples for the target domain need to be distinguishable by class to solve the DG problem. Post-synthesis mixup, ensures synthesised amplitude images lie closer to the amplitude manifold of the source data, ensuring the model can solve the classification problem.
Dense model averaging (SWAD). We found that the DG training becomes more stable and the target-domain test performance becomes more consistent when we use the dense model averaging strategy SWAD (Cha et al., 2021). We adopt the SWAD model averaging for the variational and model parameters (A, 0) while the generator network is not averaged.
Amplitude image structures. From the definition of the Fourier transform, the frequency domain function should be even-conjugate, i.c., F(-u, -u) = F(u, v), for the real-valued images. This implies that amplitude images are symmetric. Conversely, if the amplitude images symmetric, inverse Fourier transform returns real-valued signals. Thus when generating amplitude images, we only generate the non-redundant part (frequencies) of the amplitude images. Also, the amplitude should be non-negative. We keep these constraints in mind when designing the generator network.
4 RELATED WORK
MCD. Several studies have used the MCD principle for domain adaptation, to align a source model to unlabeled target data (Saito et al. 2018 Kim et al. 2019 Lu al.) 2020). We uniquely exploit the MCD principle for the DG problem, in the absence of target data, by using MCD to synthesise worst-case target domain data, as well as to adapt the model to that synthesised domain.
Augmentation approaches to DG. Several DG approaches have been proposed based on data augmentation. Existing approaches either define augmentation heuristics (Zhou al.) 2022b Xu et al. 2021), or exploit domain adversarial learning - i.e., confusing domain classifier (Saanaa ctt al. 2018: Zhou et al., 2020). Our adversarial learning is based on the much stronger (S)MCD principle that confuses a category classifier. This provides much harder examples for robust learning, while our Fourier amplitude synthesis ensures the examples are actually recognisable.
Alignment approaches to DG. Several approaches to DG are based aligning between multiple source domains (Sun & Saenko 2016 Ganin et al. 2016 Li et al. 2018c b). under the assumption that common feature across all source domains will be good for a held out target domain. Differently,
we use the MCD principle to robustify our source trained model by aligning it with the synthesised worst-case target domain.
5 EXPERIMENTS
We test our approach the DomainBed benchmark (Gulrajani & Lopez-Paz 2021), including PACS (Li et al. 2017), VLCS (Fang et al.. 2013), OfficeHome (Venkateswara et al. 2017). Ter-
raIncognita (Beery al. 2018). and DomainNet (Peng et al. 2019). For each dataset, we adopt the standard mave-oonedomamn out source/target domain splits. The overall training/test protocols are similar to (Gulrajani & Lopez-Paz] 2021] Cha et al. 2021). We use the ResNet-50 (He et al. 2016) as our feature extractor backbone, which is initialised by the pretrained weights on ImageNet (Degg jet al. 2009). For the generator network, we found that a linear model performed the best for the noise dimension 100. Our model is trained by the Adam optimiser (Kingma & Ba 2015) machines with single Tesla V100 GPUs. The hyperparameters introduced in our model (e.g.. SMCD trade-off 77) and the general ones (c.g., learning rate, SWAD regime hyperparameters. maximum numbers of iterations) are chosen by grid search on the validation set according to the DomainBed protocol (Gulrajani & Lopez-Paz 2021). For instance, 7) 0.1 for all datasets. The implementation details including chosen hyperparameters can be found in Appendix A.
5.1
MAIN RESULTS
The test accuracies averaged over target domains are summarised in Table [ where the results for individual target domains reported in Appendix A.2 The proposed approach performs the best
7

---

Published as a conference paper at ICLR 2023
Table Average accuracies on DomainBed datasets. Note: indicates that the results are excerpted from the published papers or (Gulrajani & Lopez-Pat Paz. 2021). Our own runs are reported without
t. Note that FACT (Xu al. 2021) adopted slightly different data/domain split protocol from DomainBed's, explaining discrepancy on PACS.
Algorithm
PACS
VLCS
OfficcHome
Terrinn.
DomainNet
Avg.
FRM et 2021
84.2
T7-3
67.6
47.8
44.0
64.2
IRM (Arjovsyy al..
2019
83.5
78.6
33.9
GroupDRO Suruwa
2020
84.4
76.7
66.0
43.2
33.3
60.7
1-ixup
Xa al 2020
Yan
2020
Wang al. 20206
84.6
77.4
68,1
17.9
39.2
63.1
MLDG
Jet al.
Na
84.9
77.2
66.8
47.8
41.2
63.6
CORAL
Sun & Socoko 2016
86.2
78.8
68.7
47.7
41.5
64.5
MMD
S0
84.7
77.5
66.4
42.2
23.4
58.8
DANN Ganin al.
83.7
78.6
65.9
46.7
38.3
62.6
82.6
77.5
38.3
62.0
MTL
Blanchurd
2021
84.6
77.2
66.4
45.6
40.6
62.9
SugNeL
Nam al. 2021
86.3
77.8
68.1
18.6
10.3
64.22
ARM
Zhang et a. M020
85.1
77.6
64.8
45.5
35.5
61.77
VRES
(Krugger al 2020
81.9
78.3
66.4
16.4
33.6
61.9
RSC (Hunng et al 2020
85.2
77.1
65.5
16.6
38.9
62.7
Mixstyle /hoou al. 2021
85.2
77.9
60.4
44.0
34.0
60.3
FACT
FACT
Xu et
2012
86.4
76.6
66.6
45.4
42.6
63.5
Amp-Mixup Xu
2021
84.7
75.9
64.0
46.8
42.0
62.7
SWAD (Cha et al.
2021
88.1
79.1
70.6
50.0
46.5
66.9
FACT-SWAD
88.1
77.7
70.6
51.0
16.7
66.8
Amp-Mixup SWAD
88.1
78.2
70.3
51.2
46.4
66.8
(Propssed AGFA
89.8
79.6
71.6
62.4
47.1
68.0
for all datasets among the competitors, and the difference from the second best model (SWAD) is significant (about 1.1% margin). We particularly contrast with two recent approaches: SWAD (Cha
(et al.
202 I that adopts the dense model averaging with the simple ERM loss and FACT (Xu et al.
2021) that uses the Fourier amplitude mixup as means of data augmentation with additional student-teacher regularisation.
First, SWAD (Cha al. 2021 is the second best model in Table implying that the simple ERM loss combined with the dense model averaging that seeks for flat minima is quite effective, also observed previously (Gulrajani & Paz 2021). FACT (Xu et al., (021) utilises the Fourier amplitude spectra similar to our approach. but their main focus is data augmentation, producing more training images by amplitude mixup of source domain images. FACT also adopted the so-called teacher co-regularisation which forces the orders of the class prediction logits to be consistent between teacher and student models on the amplitude-mixup data. To disentangle the impact of these two components in FACT, we ran model called Amp-Mixup that is simply FACT without teacher co-regularisation. The teacher co-regularisation yields further improvement in the average accuracy (FACT > Amp-Mixup in the last column of TT, verifying the claim in (Xu al. 2021) although FACT is slightly worse than Amp-Mixup on VLCS and TerraIncognita.
We also modified FACT and Amp-Mixup models by incorporating the SWAD model averaging (FACT+SWAD and Amp-Mixup+ SSAD in the table). Clearly they perform even better in combi- nation with SWAD. Since Amp-Mixup+ SWAD can be seen as dropping the teacher regularisation and adopting the SWAD (regularisation) strategy instead, we can say that SWAD is more effective regularisation than student-teacher. Nevertheless, despite the utilisation of amplitude-mine tation, it appears that FACT and Amp-Mixup have little improvement over the ERM loss even when the SWAD strategy is used. This signifies the effect of the adversarial Fourier-based target domain generation in our approach which exhibits significant improvement over ERM and SWAD.
augmen-
5.2 FURTHER ANALYSIS
Sensitivity to 77 (SMCD strength). We analyze sensitivity of the target domain generalisation performance to the SMCD off hyperparameter y,. We run our algorithm with different values of 7. The results are shown in Fig. 3 Note that 17 = 0 ignores the SMCD loss term (thus generator has no influence on the model training), which corresponds to the ERM approach. The test accuracy of the proposed approach remains significantly better than ERM/SWAD for all those 7) with moderate variations around the best value. See Appendix[AA. for the results on individual target domains.
Sensitivity to (post-synthesis mixup strength). We mix up the generated amplitude images and the source domain images as in (I7) to make the adversarial target domain classification task solvable. The task becomes easier for small (less impact of the generated amplitudes). and vice versa. Note
8

---

Published as a conference paper at ICLR 2023
PACS
Officelome
Nopeuel
[ 00.0
72.3
00.4
76.
3.00
041
02
SMC2
Figure 3: Sensitivity (SMCD trade-off) on PACS and OfficeHome.
PACS
officeHome
SIND
$0.00
21.2
60.75
(3.3
25.25
68.38
0,2 of paterrup
Strwugh alpan-mous)
Figure 4: Sensitivity to a (post-mixup strength) on PACS and OfficeHome.
Table 2: Ablation study: 1) unsupervised MCD (instead of SMCD), 2) without post- mosttmmxup, 3) without SWAD, and 4) pixel-based target image generation (instead of amplitude generation).
Art
Cartoon
Photo
Skeich
Average
Unsupervised MCD
88.94 + 0.23
83.83 + 0.19
97.27 = 0.10
81.77 + 0.36
87.95
Without post-mixup
88.90 + 0.16
81.80 + 0.17
97.43 0.14
80.86 +: 0.31
87.25
Without SWAD
84.20 + 0.68
81.56 1 0.55
94.83 + 0.12
79.28 +: 0.94
81.97
Pixel Pixel-baed generation
88.85 1: 0.15
83.62 1 0.26
97.23 1 0.15
82.10 1: 0.63
87.95
(Proposed) AGFA
89.80 = 0.34
85.16 + 0.65
97.59
+ 0.27
84.67
0.82
89.30
that a 0 ignores generated amplitude images completely in post-mixup, and the training becomes close to ERM learning where the only difference is that we utilise more basic augmentation (e.g., flip, rotation, color jittering). As shown in Fig. 4 the target test performance is not very sensitive around the best selected hyperparameters. See also ablation study results on the impact of post-mixup below.
Impact of SMCD (ys. unsupervised MCD). We verify the positive effect of the proposed super- vised MCD loss (SMCD in (T6) that exploits the induced target domain class labels, compared to the conventional (unsupervised) MCD loss (14) without using the target class labels. The result in Table our claim that exploiting target class labels induced from the phase information is quite effective, improving the target generalisation performance.
Impact of post-synthesis mixup. We argued that our post-synthesis mixup of the generated amplitude images makes the class prediction task easier for the generated target domain, for the solvability of the DG problem. To verify this, we compare two models, with and without the post-mixup strategy in T The model trained with post-mixup performs better.
Impact of SWAD. We adopted the SWAD model averaging scheme (Cha et al. 2021) for improving generalisation performance. We verify the impact of the SWAD as in Table where the model without SWAD has lower target test accuracy sigmifying the importance of the SWAD model averaging.
Impact of amplitude generation. The amplitude image generation in our adversarial MCD learning allows us to separate the phase and amplitude images and exploit the class labels induced by the phase information. However, one may be curious about how the model would work if we instead generate full images without phase/amplitude separation in an adversarial way. That is, we adopt pixel-based adversarial image generator, and in turn replace our SMCD by the conventional MCD loss (since there are no class labels inducible in this strategy). We consider two generator architectures: linear (from 100-dim input noise to full image pixels) and nonlinear (a fully connected network with one hidden layer of 100 units), where the former slightly performs better. Table shows that this pixel-based target image generation underperforms our amplitude generation.
6 CONCLUSION
We tried to address the domain generalisation problem from the perspective of maximum classifier discrepancy: Improving robustness by synthesising worst-case target domain for learning, and training the model to be robust to that domain with the (S)MCD objective. To provide an approxima- tion to style and content separation for synthesis, the worst-case domain is synthesised in Fourier amplitude space. Our results provide a clear improvement on the state-of of-the-arts on the challenging Domain Bed benchmark suite.

---

Published as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 00779,229,,. 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- cralization using meta-regullarizttoon In Advances in Neural Information Processing Systems, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. European Conference on Computer Vision, 2018.
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Percira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151-175, 2010.
Shai Ren-David, John Blitzer, Koby Crammer, and Fermando Pereira. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems, 2007.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. Journal of Machine Learning Research, 22(2):1555,
2021.
Boyd and I.. Vandenberghe. Convex Optimization. Cambridge: Cambridge University Press, 2004.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: Domain Generalization by Seeking Flat Minima. Advances in Neural Information Processing Systems (NeurIPS). 2021.
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specifieity and invariance for in and out of domain generalization. European Conference on Computer Vision, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, 2009.
Qi Dou, Daniel C Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In Advances in Neural Information Processing Systems, 2019.
Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. International Conference Computer Vision, 2013.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarin training of neural networks. Journal of Machine Learning Research, 17(1):1-35, 2016.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 (211))665-677 673,6566636666 2020.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley. S. Ozair. A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information P Processing Systems, 2014.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. International Conference on Learning Representations, 2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEF/CVF Conference on Computer Vision and Puttern Recognition, 2016.
Zeyi Huang. Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross domain generalization. European Conference Computer Vision, 2020.
10

---

Published as a conference paper at ICLR 2023
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Uncertainty in Artificial Intelligence, 2018.
Minyoung Kim, Pritish Sahu, Behnam Gholami, and Vladimir Pavlovic. Unsupervised Visual Domain Adaptation: A Deep Max-Margin Gaussian Process Approach. Computer Vision and Puttern Recognition, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. International Conference on Learning Representations, 2015.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A bench- mark of in-the-wild distribution shifts. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 5637-5664. PMLR, 2021.
David Krueger, Ethan Caballero, Joern- Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (REx). arXiv preprint wrXiv:2003.00688, 2020.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. International Conference on Computer Vision, 2017.
Da Li, Yongxin Yang. Yi-Zhe Song. and Timothy Hospedales. Learning to generalize: Meta learning for domain generalization. AAAI Conference Artificial Intelligence, 2018a.
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M. Hospedales. Episodic training for domain generalization. In ICCV, 2019.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. IEEE Conference on Computer Vision and Pattern Recognition, 2018b.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. AAAI Conference on Artificial Intelligence, 2018c
Zhihe Lu, Yongxin Yang. Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9111-9120, 2020.
Hyconseob Nam. HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Oren Nuriel, Sagie Benaim, and Lior Wolf. Permuted AdaIN: Reducing the bias towards global statis- tics in image classification. IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Alan Oppenheim and Jae S Lim. The importance of phase in signals. Proceedings of the IEEE, 69 (5):529-544) 1981.
Xingehao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. International Conference on Computer Vision, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. International Conference on Learning Representations, 2020.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. Computer Vision and Pattern Recognition, 2018.
Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. European Conference on Computer Vision, 2020.
11

---

Published as a conference paper at ICLR 2023
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross cross-gradiem gradient training. In ICLR, 2018.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. European Conference on Computer Vision, 2016.
Ramakrishna Vedantam, David Lopez-Paz, and David J. Schwab. An Empirical Investigation of Domain Generalization with Empirical Risk Minimizers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Computer Vision and Pattern Recognition, 2017.
Shujun Wang, Lequan Yu, Caizi Li, Chi- Wing Fu, and Pheng-Am Heng. Learning from extrinsic and intrinsic supervisions for domain generalization. European Conference on Computer Vision, 2020a.
Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. IEFE International Conference on Acoustics, Speech and Signal Processing, 2020b.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang. Qi Tian, and Wenjun Zhang
Adversarial domain adaptation with domain mixup. AAAI Conference Artificial Intelligence, 2020.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A Fourier-based Framework for Domain Generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint vrXiv:2001.00677 2020.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond Empirical Risk Minimization. International Conference on Learning Representations, 2018.
Marvin Zhang, Henrik Marklund, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group shift. arXiv preprint arX1v:2007.02931, 2020.
Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, and Lei Zhang. Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization. IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generalization via entropy regularization. In Advances in Neural Information Processing Systems, 2020.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversaria image generation for domain generalisation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv preprint arXiv:2103.02550 2021a.
Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. International Conference on Learning Representations, 2021b.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
The Appendix consists of the following contents:
Implementation Details (Sec. A.I
Full Results (Sec. A.2)
Derivation of ELBO Variational Inference (Sec. A.3
Additional Experimental Results (Sec. A.4)
A.1 IMPLEMENTATION DETAILS
We adopt the ResNet50 (He et al., 2016) architecture (removing the final classification layer) as the feature extractor network. For the amplitude generator network, we have tested several fully- connected network architectures with different numbers of hidden layers and hidden units, and the simple linear network peformed the best. The input noise dimension for the generator is chosen as 100. The covariance matrices of the variational parameters are restricted to be diagonal. The number of MC samples from (W) in the ELBO optimisation is chosen as 50.
The optimisation hyperparameters are chosen by the same strategy as (Cha et al. 2021). where we employ the Adam optimiser (Kingma Ba 2015) with learning rate x 10 and no dropout, weight decay used. The batch size was 32 (for each training domain) in ERM/SWAD (Cha et al. 2021). but we halved generation. The standard basic data augmentation is also applied the input images. Following the suggestion from (Cha et al. 2021). we run our model up to 5000 iterations for all datasets except for DomainNet. But the algorithm may stop earlier before the maximum iterations if SWAD termination condition is met (See Sec. A. below). Since DomainNet is a large-scale dataset, and it is required to have et al. 2021), they used I 15000 iterations which roughly corresponds to 3 to 10 data epochs. In our model, since we halved the number of input images in the batch, in order to have the same training epochs as (Cha et al. (021), we increase it up to 30000 iterations for DomainNet. The details of the SWAD implementation follows in the next section.
our model since the remaining half are constructed by the adversarial target
larger number of iterations go through the entire data at least several times. In (Cha
A.1.1 SWAD MODEL. AVERAGING
We adopt the SWAD model averaging strategy (Cha et al. 2021 to have more robust model that is less affected by overfitting. We apply the SWAD to the feature extractor network parameters @ and the variational parameters X, but not the adversarial generator network. Since SWAD is important component in our model, we provide more details here.
SWAD is motivated from stochastic weight averaging (SWA) (Izmailov et al. 2018), however, unlike SWA's model averaging for every epoch, SWAD takes dense model averaging for every (batch) iteration. A key component of the SWAD algorithm to determine the model averaging regime, the interval of iterations for which the model averaging is performed. This regime is aimed to avoid overfitting, and known as overfit-aware model averaging. The regime is specified by the start and end iteration numbers, l,, and l,, respectively, and we take model averaging for iterations / E [4x, 4e), that is,
AD = to to
0¹ ASW to
X
(18)
to
+
1=t.
where H¹ and X' are the model parameters after iteration f. Here (DsW ASWAD) are the final model parameters returned by the training algorithm.
Now we describe how the regime is determined. Ideally, we expect the intermediate models during the interval [(a, Le] to be overfit-free, having high generalisation performance. To this end, we evaluate the model on the validation set (held out from the source domain training data), and denote the validation loss of the (-th model by 1e Then the start iteration of the regime, is determined by the first I where the validation loss is not improved for the next N, iterations (c.g., N, = 3). That is,
ts min(t N, 11/--1
val
1 Lual ... val
#1NN++1
(19)
13

---

Published as a conference paper at ICLR 2023
Algorithm 1 AGFA Algorithm with SWAD Model Averaging.
Input: Source data S, SMCD trade-oft 7), post-mixup a, and learning rate 7, and SWAD hyperparameters N,, N..
Initialise: (feature extractor), À (variational parameters), and (generator).
(flag)
-ee--Reiime----eee +- FALSE, (iteration) + 0.
Repeat:
0. Sample minibatch SB {(x;, ) from S.
1. Prepare {A45. ),)))= by Fourier transform A A = =(
=
2. Generate amplitude images A = Gu(ei), & ~ (N0, for = ,
3. Post-mixup: - + XA + (1 - A) X ~ Uniform(..a).
4. Construct a target batch TB = {(r)
= =F vp =y¹
5. Evaluate Lmodel == ELBO(A, 0; SB) 7
6. Update the model and variational parameters: (A,0) + (2,0) -VV (1.03Cmodel.
7. Evaluate yen = SMCD TB).
8. Update the generator network: f VVlgenn
9. (SWAD procedure)
t + 1, (X', 0*) f- ().
If SWWA-Reeime--nteee
== FALSE:
N,+N minogr=M Loal:: If vol
tg + N₂ + 1, Tvat t N, Ev=0 EN--i n'-
WWAD-Regime Entered t- True.
Else:
Ifr Twat < <inngs b/rl: le e 1- No. Return AD L. 1--4 Ete,, 0t, ASWAD 1--1+4 Eist, N..
Once we find &,, we compute the (average) starting validation loss,
N1 eat
Leat
(20)
N.
which is used as a reference when we decide the end iteration te. As we enter the regime, we start model averaging every iteration. To determine when to stop, inspect the validation losses see if the model starts overfitting. Specifically, if the validation losses are consecutively greater than w by some margin, we regard as overfit signal. That is,
te min(t NN Ceats toa ...... No Not1 > leath.
(21)
where and N₂ user-driven hyperparameters (e.g., 1.3, N. 6).
The pseudo code of our AGFA algorithm with the SWAD strategy is summarised in Alg. There three hyperparameters in SWAD. (N.. N₂,, r). and following (Cha al. (02T), use N,
= 6, = 1.3 for all datasets in DomainBed except r = 1.2 for VLCS. One technical issue is that evaluating the validation loss every iteration is computationally demanding. Similarly (Cha et al. 202I), we compute the validation loss at every V-th iterations (c.g., V = 50 for VLCS, V 500 for DomainNet, and V 100 for the rest) although the model averaging is still performed every iteration. Accordingly, the equations (19). 20).. and (21) need to be changed where essentially all iteration numbers in those equations should be changed to multiples of V. The model averaging in
Alg. Dis implemented by the running (online) average and the use of (FIFO) queue data structures
similarly as (Cha et al. 2021), which does not incur significant extra computational overhead.
A.2 FULL RESULTS
The full results (test errors individual target domains) DomainBed datasets summarised in Table (PACS), Table (VLCS), Table (OfficeHome), Table 6 (Terralncognita) and Table 7 (DomainNet).
We also show the full results of the sensitivity analysis Table (the SMCD loss trade-off 17) and Table 9 (the post-synthesis mixup strength o.). Moreover, visualise in Fig. the ablation study
14

---

Published as a conference paper at ICLR 2023
Table 3: Average accuracies PACS. Note: indicates that the results are excerpted from the published papers or (Gulrajani & Lopez-Par 2021). Our own runs are reported without 1, FACT (Xu al. 2021) adopted slightly different data/domain split from DomainBed's explaining discrepancy.
Algorithm
Avg
MASF Doa al 2019
82.9
80.5
95.0
T2.3
82.7
DMG
Chattonahyyay et
2020
82.6
78.1
94.5
78.3
83.4
MetaRes
Balaji et 2018
87.2
79.2
97.6
70.3
83.6
Zhap
79.3
85.8
EISNot
Wang
2020a
86.6
81.5
97.1
T8.1
85.8
DSON Seo el
2000
87.0
80.6
96.0
82.9
86.6
ERM
Cha all
85.7 2 0.6
77.1 0.8
97.4 1.:00.. U.4
76.6 0.7
84.2
IRM (Arjovsky ol. 2019
84.8 1: 1.3
76.4 1.1
96.7 1: 0.6
76.1 d: 1.0
83.5
GraupDRO Sapawa
83.5
79.1 =
78.3 2.0
2020 1-Mixup Xu
Wang 20206
0.5
78.9
MLDG
Ka
85.5
80.1 1.7
97.4 1.3 0.3
76.6
84.9
CORAL Sun & Sacako, 2016
88.3
80.
97.5
78.8
86.2
MMD al 201361
86.1
79.4 0.9
00.60.22
76.5 0.5
84.7
DANN Ganin et 2016
8..4
77.4= 0.8
73.5 2.3
83.7
CDANN LL al. 80
11
75.5=09
96.8
73.5
82.6
$7.5
77.1
96.4
77.3
SagNe
Nam 2021
1.04
0.
ARM
Zhang
2020
86.8
76.8 0.5
97.4 1 0.3
79.3
85.1
VREs
Keuceer
1020
1.
79.
96.9
77.7 1.7
84.9
RSC
Huung et al 2020
0.84
79.7 1.8
97.6 2 0.3
78.2 1.2
85.2
Mixstyle: (hoo et ml
2021b
86.
79.0 1.4
96.6 0.1
78.5 2.3
85.2
FACT
Xu al. 2021
0.5
81.8
96.8 0.1
81.5
88.2
87.8 0.2
80.5
96.2 0.2
81.2
Amp-Mixup Xu
84.7
81.0=11
SWAD Cha
8.
82.5
FACT-SWAL
89.6 0.8
82.5
96.6 0.2
83.8
88.1
Amp-Mixup--SWW SWAD
88.7 0.1
8..2
96.4 + 0.1
84.1 + 0.5
88.1
(Proposed) AGFA
89.8 0.3
85.2 0.6
97.6 1: 0.3
84.7 1: 0.8
89.3
results for the four different modeling choices: 1) Impact of SMCD (vs. conventional unsupervised MCD), 2) Impact of post-synthesis mixup, 3) Impact of SWAD, and 4) Impact of amplitude generation (vs. pixel-based image generation). For the pixel-based image generation, we consider two generator architectures: linear (from 100-dim input noise to full image pixels) and nonlinear (a fully connected network with one hidden layer of 100 units).
Visualisation of generated adversarial images. We visualise in Fig, 6 some synthesised amplitude images and constructed target domain images from the learned model the PACS dataset. Although the generated amplitude images visually look like random noise, they appear to have the effect of attenuating high frequency spectra (shown darker pixels in the fifth column) when combined with the source domain amplitude images by post-mixup. The constructed images from the generated amplitude images alone without post-mixup (sixth column) look lot like edge detection maps. whereas the post-mixup constructed ones (seventh column) remain visually similar to the original source domain images, promoting DG solvability.
A.3 DERIVATION OF ELBO IN VARIATIONAL INFERENCE
We derive the evidence lower bound (ELBO) in [11 in the main paper. To enforce Qx(W) s P(W|S, 8), we minimise their KL divergence,
(x(W) P(W|S,))
(Qx(w)
log
(22)
FoN(W) log QA(W)P(S10) P(S P)W(
(23)
log P(S EQx(w) log P(S 0) BQx(w) log Qx(W)
(24)
P(W)
log P(S - EQ,(w) log PISIW, 0) (L((I(I))))))))
(25)
log P(S Σ EQ,(w) log Ply2,, W. + KL((X(W)))(W)).
(26)
(#.8)~S
15

---

Published as a conference paper at ICLR 2023
Table 4: Average accuracies VLCSS. The same interpretation Table 3
Algorithm
Avg
ERM
Cha al. 2021
98.0 0.3
61.7 = 1.2
71.4 1.2
73.2 t 1.6
T7.3
IRM
(Arjovsly
2019
98.6 0.1
64.9 0.9
73.4 1 0.8
17.3 1 0.9
78.6
GroupDRO Sapawa et
2020
0.3
63.1=99
09.5 09.5
0.77
76.7
1-Mixup
Xa
Wang 2020b
0..
618=10
0.8
MLDG
N0
9.24
0.72
1..0
75.3
CORAL Sun
Saenko,
2016
98.3
1..1
1.3
77.5
78.8
MMD
La et ml
Nb
97.7
64.0= 1.1
72.8 0.2
75.3 3.3
77.5
DANN Gunin et al 2016
99.0 0.3
65.1=14
0.3
1..6
78.6
CDANN
eL
8c
97.1 0..
70.7 0.8
77.1
77.5
MTL
lanchard et 2021
97.8 0.4
613=033
0.7
75.3
T7.2
SagNet
6.5=0.5
ARM
2020
0.2
63.6=
1.2
7.6
VREs
KKuueger
2020
0..
614=14
74.1
1.
78.3
RSC (Hunng et al 2020
0.11
0..5=077
72.3
12.3111 1.2
75.6 0.8
77.1
Mixstyle Zhou al. 2021b
98.6
64.5=111
72.6 7..
76.7 76.71:11 1.7
77.9
TACT al 2021
97.6
0..
69.2 0.8
1.7
76.6
Amp-Mixup
9.77
65.6=0.3
7.9
0..1
98.8 0.1
63.3=03
0.5
79.2 0.6
79.1
FACT-SWAI
98.4 0.1
6.3
0.5
7.40
Amp-Mixup SWAD
98.7 0.1
63.9
73.5 0.2
76.7.1.02
78.2
(Proosed) AGFA
64.5
0.3
78.9.1 0.6
79.5
Table 5: Average accuracies OfficeHome. The same interpretation Table 3
Algorithm
C
L
S
V
ERM Cha
2021
63.1
51.9=0.4
77.2
78.1 0.2
67.6
Arroosky
2019
72.1
74.0
GroupDRC DRO Suruwa
2020
0.7
1.0
0.7
0.7
66.0
1-Mixup
2020
Yan
2020
Wang al., 20206
0.8
5.6
0.3
0.2
68.1
MLDG
al. Sa
0.9
0.6
7
77.5 1..44...44
66.8
CORAL Sun & Socako,
2016
0..
5..1005
7..5
78.4
68.7
MMD
ct
2018b1
0.2
0.3
74.3
77.4
66.4
DANN Ganin 2016
1.3
63.0=0.3
7..6
0..
65.9
CDANN
1.
2..4
0.9
0.6
MTL.
Bllnchard
1121
0.7
0.6
1.4
0..
66.4
SagNei
Nam nl. 2021
0.2
54.8
2.4
7.3
68.1
ARM
Zhang a)
2020
0..9
=.5
74.1
75.2
64.8
VREx
KKuuege et 2020
60.7
53.0
75.3 0.1
0.5
66.4
RSC
Huang et al 2020
60.7
5.4=0.3
74.8
75.1
65.5
Mixstyle
68.2
0.1
51.9004
7..
0..
FACT
Xu Ku
2012
0.1
0.1
7.20
7..2
66.6
Amp-Mixup Xu
2021
57.1 0.3
01.9=0.1
7.3
7..4
64.0
SWAD Cha al. 2021
66.1 0.4
5..7=0.4
7.4
80.2
70.6
FACT-SWAD
0.2
5..3=002
7..
0.1
70.6
A SWAD SWAD
65.9 0.2
57.9 57.9
77.8 0.22
79.7 0.11
70.3
(Proposed) AGFA
67.5 2: 0.3
58.1
0.1
80.7
0.1
71.5
Since KL divergence is non-negative, re-arranging 26
yields:
log P(S
Σ [log W. W) -LL(X(W)))((W))
(27)
(Z.9)~
and the right hand side constitutes the ELBO.
A.4 ADDITIONAL EXPERIMENTAL. RESULTS
A.4.1 RESULTS ON RESNET- 18 BACKBONE
To test our approach on backbone networks other than ResNet-50, we run experiments with the 18 backbone the PACS dataset. The results summarised in Table I0] Compared the recent approaches MixStyle (Zhou et al.) (021b) and EFDMix (Zhang et al. 2022). our approach AGFA again shows higher performance even with the smaller ResNet- backbone.
16

---

Published as a conference paper at ICLR 2023
Table 6: Average accuracies on Terralncognita. The interpretation Table
Algorithm
1.3
Avg
ERM
Cha et at 2021
54.3 + 0.4
42.5 = 0.7
55.6
0.3
38.8 2.5
47.8
IRM
(Arjovsly et
2019
54.6
39.8
56.2
39.6 0.8
47.6
GroupDRO Sapawa et
2020
11.2 11.2
38.6
56.7
36.1
13.2
1-Mixup
Xa
Wang 20206
MLDG
N0
3.0
1.1 42.2
0.3
2.2
CORAL Sun
Saenkon
2016
51.6 2.4
57.0
39.8 2.8
47.7
MMD
La s ml
Nb
41.9 3.0
34.8 1.0
57.0 17.0011 1.9
35.2
42.2
DANN Gunin et al
2016
51.1 3.5
40.6 0.6
57.4 0.5
37.7 1.8
46.7
CDANN
eL
8c
47.0 1.9
41.3
54.9 1.7
39.8 2.3
45.8
MTL
lanchard et 2021
19.3
39.6
53.6
37.8 0.8
15.6
SagNet
2.5
ARM
2030
0.7
0.8
18.7
VREs
KKuueger
2020
4..
41.7
0.8
38.7 3.1
46.4
RSC (Hunng et al 2020
50.2 2.2
39.2 1.4
56.3 1.4
40.8 + 0.6
46.6
Mixstyle Zhou al. 2021b
64.3 44.3 1.1
34.1=1.1
56.9 56.9 1.1
31.7 1
44.0
TACT al 2021
$2.4
42.3 1.0
55.5 1.3
0.9
45.4
Amp-Mixup
0..8
38.9
5.2
4..8
$5.4 0.0
44.9
09.7
0.2
FACT-SWAI
1.1
0.3
Amp-Mixup SWAD
56.6 0.6
46.3
60.2
41.8.10.44
51.2
(Proosed) AGPA
61.0 0.3
46.2
60.3
42.3 0.9
52.4
Table 7: Average accuracies on DomainNet. The same interpretation as Table
Algorithm
R
Avg
DMG
Chattopsdivay
2020
MetaRar: (Halaj al. 2018
50.2
11.5
64.6
50.1
43.6
ERM Cla ot
2021
63.0 0.2
21.2 + 0.2
50.1 = 0.4
13.9
63.7 13.7-10 0.2
52.0 0.5
44.0
IRM
(Arovvky
2019
2.8
15.0 1: 1.5
38.3
4.3
10.9
0.5
48.2 5.2
42.3 3.1
33.9
GrauDDOO Naguwa et
2020
47.2 0.5
17.5
0.4
33.8
0.5
9.3 0.3
51.6 0.4
40.1 0.6
33.3
1-Mixup (Citation before)
55.7 0.3 0.3
18.3 0.5
44.3 0.5
12.5 0.4
55.8 0.3
48.2
39.2
Li
201
2016
59.2
0.2
MMD
2013b]
32.1 13.3
11.0
26.8 11.3
8.7 2.1
$2.7 13.8
28.9 11.9
23.4
DANN
Ganin
2016
53.1 + 0.2
18.3 0.1
44.2 0.7
11.8
56.5
46.8 0.6
38.3
CDANN
Li al 20
$4.6 0.4
17.3 0.1
43.7 1: 0.9
12.1 0.7 0.7
56.2
45.9 0.5
38.3
MTI.
(Alnnchard al
2021
57.9
18.5
46.0 0.1
12.5 0.1
59.5
49.2 0.1
40.6
SagNet Nam al,
5..7
0.2
45.3 0.3
12.7 0.5
58.1
48.6
40.3
Zaan
49.7
9.1
Kruceer
2020
49.6
3.0
RSC Huung al 2020
1.2
0..
44.1 0.6
12.2 0.2
0.7
0.9
38.9
Mixstyle Zhou
2021b
0..9
0.2
37.0
12.3 0.1
4.3
43.4
34.0
FACT Xu al 2021
0.3
19.1
0.4
13.9 0.3
6..5
51.0 0.7
42.6
Amp-Mixup Xt
2021
6.1
19.0 0.2
47.2 0.4
12.9 0.8
5..5
51.0
42.0
SWAD Cha et 2021
66.0
0.3
53.5
16.1 0.2
6..8
55.5
46.5
FACT-SWAR
0.
0.2
0..7
16.3 0.1
6..0
5..9
46.7
Amp-Mixup-SWAN
6..6
5..6
(Proposed)
5..0
0.1
A.4.2 RESULTS ON COLOURED-MNIS' AND ROTATED-MNIST
Although relatively smaller and easier datasets in the DomainBed benchmark, we also test our method on the Coloured-MNIST and RotASTd-MNIST datasets. Following the experimental protocols including the four-layer ConvNet backbone as in (Gulrajani & Lopez-Paz) (021), the test accuracies are reported II (Colored-d -NNIST) and Table (RoIated-MNISTT As shown, all approaches including ours perform equally well on these datasets.
A.4.3 RESULTS ON SINGLE-SOURGE GENERALISATION
We have focused predominantly the most popular empirical study. Another reasonable experimental setting is single source generalisation setting: training only source domain and testing on the rest domains. Our single source domain results
DG setting in
17

---

Published as a conference paper at ICLR 2023
Table 8: Sensitivity analysis on the SMCD loss trade off v on PACS and OfficeHome.
(a) PACS
Cartoon
Photo
Sketch
Average
0.19
1: 0.30
88,10
80.24
97.17 0.07
+: 0.37
88.78
88,80
0.17
0.80
88.38
0.36
0.22
4 0.57
88.05
(h) OfficeHome
Clipart
Product
Real
Average
0.0
66.09 = 0.28
57.72 + 0.34
78.47 + 0.16
80.19 + 0.11
70.62
0.01
66.86 + 0.17
58.43
+ 0.34
78.53
1 0.09
80.51 1: 0.31
71.08
0.05
66.95 1: 0.09
58.58 1 0.32
78.96 1: 0.28
80.46 1: 0.24
71.23
0.1
67.46 0.28
58.45 1 0.13
79.27
1 0.07
80.70 1: 0.11
71.47
0.15
66. = 0.33
58.3
0.22
78.59 + 0.33
80.51 + 0.09
70.97
0.2
66.05 = 0.03
58. 18.10-0 0.11
78.69
+ 0.18
80.45
1 0.06
70.82
0.25
66.15 + 0.17
58.29 1: 0.36
78.16 1: 0.04
79.87 1: 0.21
70.62
Table 9: Sensitivity analysis the post-mixup trade off on PACS and Ofmi
(a) PACS
Art
Cartoon
Photo
Skerch
Average
0.0
89.29 1 0.37
83.55 = 0.20
97.11 i 0.07
82.17
U.91
88.03
0.2
89.23 1 0.22
83.80
0.21
97.25
0.05
82.59
0.84
88.22
0.4
89.37 1 0.17
83.93
0.07
97.27
1 0.10
83.37 1 0.43
88.49
0.6
89.
+ 0.18
$1.30 0.16
97.31
0.13
83.71
0.60
88.69
0.8
89.47 + 0.61
84.39 0.18
97.41
0.15
83.68
+ U.15
88.74
0.9
89.66
1 0.23
85.04
0.28
97.60
1 0.13
84.27
0.40
89.14
1.0
89.80
1 0.34
85. 16 1 0.65
97.59
1 0.27
84.67
0.82
89.30
OfficeHomme
Clipart
Product
Real
Average
65.99 0.17
57.72
0.16
78.36
0.08
80.22 0.09
70.57
0.15
78.72 0.15
80.36
71.05
79.27
0.07
0.24
58.50
67.04
1
71.10
06.15
0.20
78.53
80.22
70.80
66.56
70.79
66.47 0.30
70.72
18

---

Published as a conference paper at ICLR 2023
Supervised (Proposed)
post-mixup (Proposed)
100-
Unsupervised MCD
100-
Without post-mikup
Target accuracy (%)
Target Target accuracy (%)
90-
90-
A
Target 5
Avg
Aug
Target
W/ (Proposed)
Amp. (Proposed)
(FC-Net)
Without SWAD
100
Pixel (Linear)
Target accuracy (%)
Target accuracy
90
90
80
P
$
Avg
A
Avg
Target domain
Target
Ablation study of four different modeling choices: SMCD, post-mixup, SWAD, and amplitude generation (instead of pixel-based target image generation).
Figure
1) Original
2) Phase
3) Amplitude
4| Amp-gen
5) Amp nntuuo
6) Con-gen
Conn-postmmuuu
Figure 6: Visualisation of the generated amplitude and constructed images. The columns are (from left right): 1) original image, 2) phase and 3) amplitude spectra after Fourier transform, 4) generated amplitude image, 5) post-mixup of 3 and 4, 6) constructed image from phase in 2) and generated amplitude image in 4) (by inverse Fourier transform), and 7) constructed image from phase and the post-mixup amplitude 5.
on the PACS benchmark are shown in Table 13 for (a) ResNet-18 and (b) ResNet-50 backbones. The results indicate that improvement of the proposed AGEA over the existing DG methods is even more pronounced: averaged accuracies higher than the best prior method EFDMIX (Zhang et al. (022) by about 10% for ResNet- 18 and by about 7% for ResNet-50.
19

---

Published as a conference paper at ICLR 2023
Table 10: Average accuracies PACS with ResNet-18 backbone. Results ERM, Mixup (Zhang et al.
2018), MixStyle (Zhou et al. 202Ib), and EFDMix (Zhang et al. 2022 are excerpted
from
Zhang et al. 2022).
Algorithm
Cartoon
Painting
Sketch
Avg
ERM
77.0 + 0.6
75.9 + 0.6
96.0 + 0.1
69.2 = 0.6
79.5
Mixup
76.8 = 0.7
74.9 0.7
95.8 0.3
66.6 0.7
78.5
MixStyle
83.1 = 0.8
78.6 + 0.9
95.9 : 0.4
74.2 = 2.7
82.9
EFDMix
83.9 + 0.4
79.4 + 0.7
96.8 + 0.4
75.0 0.7
83.9
(Proposed) AGFA
84.5 + 0.6
78.5 0.5
95.7 0.1
80.9 + 0.2
84.9
Table 11: Average accuracies on Colored-MNIS: with the four-layer ConvNet backbone. Results competing methods are excerpted from (Gulrajani & Lopez-Paz 2021).
Algorithm
0.1
0.2
0.9
Avg
ERM
72.7 0.2
73.2 0.3
10.0 0.0
52.0
IRM
72.0
7.02
10.1 0.2
51.8
DRO
0.3
0.3
10.0 0.0
51.9
Mixup
0.2
10.0 0.3
51.9
MLDG
0.4
7.3
10.0
51.6
CORAL
0.8
0.2
1.1
51.7
MMD
0.2
0.2
10.5 0.2
51.8
ADA
72.0
0.5
10.0 0.2
51.5
CondADA
0.3
73.2
10.4 0.3
51.9
(Proposed) AGFA
72.6
73.8 0.1
10.5 = 0.1
52.3
Table 12: Average accuracies on Rotated-MNIST with the four-layer ConvNet backbone. Results on competing methods excerpted from (Gulrajani & Lopez-Pau 202I).
Algorithm
Avg
ERM
95.6 + 0.1
99.0 + 0.1
98.9 + 0.0
99.1 + 0.1
99.0 - 0.0
96.7 + 0.2
98.1
IRM
95.9 0.2
98.9 0.0
99.0 0.0
98.8 0.1
98.9
95.5 0.3
97.8
DRO
95.9 16.9:.01 0.1
98.9 + 0.0
99.0
99.0
99.0 0.0
0..9
98.1
Mixup
96.1 0.2
99.1 0.0
98.9
9.0
9..
9.1
98.1
MLDG
98.9 0.2
9.0
9..1
0.0
96.0
98.0
CORAL
95.7 0.2
99.0 0.0
99.1 +.1
9.0
0.0
96.7
98.1
MMD
0.1
9.0
98.9
9.11
0.0
9.1
DANN
98.9 0.3
98.9
9.1
9.0
9.5
97.9
C-DANN
96.0 - 0.5
98.8 + 0.0
99.0 0.1
99.1
0.1
9.3
98.0
(Proposed) AGFA
98.9 +
99.0 +
0.0
- 0.0
96.4
98.0
A.4.4 COMPARISON WITH PIXEL-BASED TARGET IMAGE GENERATION
Our Fourier-based target image generation is effective for preserving semantic class information from the source domains, thanks to the phase/amplitude separation. To see if non-Fourier based generation also has similar property, we visualise adversarial target images generated by purely pixel-based manner without phase/amplitude separation. For the linear pixel-based generator model (from 100-dim input noise to full image pixels), which performed slightly better than nonlinear ones in test accuracy, we show some examples in Fig. 7 Whereas the pixel-based generation is visually uninformative and looks like pure random noise, our Fourier-based generation contains salient object edge information that is closely related to class semanties.
20

---

Published as a conference paper at ICLR 2023
Table 13: Single source domain generalisation results on PACS with ResNet-18 and (b) ResNet-50 backbones. Each column shows test accuracies averaged over the rest three target domains. Results
on ERM. MixStyle (Zhou et al. 2021b). and EFDMix (Zhang et al. 2022) are excerpted from (Zhang
et al.
2022).
(a) ResNet-18
Algorithm
Art
Cartoon
Painting
Sketch
Avg
ERM
58.6 2.4
66.4 + 0.7
34.0 +: 1.8
27.5 + 4.3
46.6
MixStyle
61.9 + 2.2
71.5 + 0.8
41.2 + 1.8
32.2 + 4.1
51.7
EFDMix
63.2 + 2.3
73.9 + 0.7
42.5 + 1.8
38.1 3.7
54.4
(Proposed) AGFA
74.2 1.1
77.5 + 0.6
48.5 = 2.6
58.3 0.9
64.6
(b) ResNet-50
Algorithm
Art
Cartoon
Painting
Sketch
Avg
ERM
63.5 + 1.3
69.2 + 1.6
38.0 0.9
31.4 1.5
50.5
MixStyle
73.2 = 1.1
74.8 + 1.1
46.0 = 2.0
40.6 = 2.0
58.6
EFDMix
75.3 0.9
77.4
0.9
44.2
61.2
(Proposed) AGFA
79.8 +.9 0.9
81.7 + 0.6
48.6 + 0.5
64.6 + 1.1
68.7
1J Original
2) Pixel-based
3) fourier-baser (Ours)
Figure 7: Comparison between pixel-based and our Fourier-based generated target images. Whereas the pixel-based generation is visually uninformative and looks like pure random noise, our Fourier- based generation contains salient object edge information that is closely related class semantics.
21


Reviews:
[{"review_id": "jrXyq4ofmfy", "reviewer": "Reviewer_5QDe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new domain generalization (DG) method by synthesizing the virtual target domain samples in Fourier domain and exploiting the maximum classifier discrepancy (MCD) principle in source domain and generated target domain. It also gives a modified MCD loss, which is suitable for DG task. The experiments show the effectiveness of the proposed method.", "summary_of_the_review": "The paper proposes a novel and well-motived method for domain generalization. However, there are lacks of some experiments and more discussion about the theory and the relation between SMCD and other losses. If you solve the problems, I am willing to improve my rating.", "strength_and_weaknesses": "Strength\n(1)\tThe proposed method is novel and well-motived. \n(2)\tThis paper gives a new perspective on DG by finding the worst-case ib an adversarial way and posing it as a domain adaptation (DA) task. \n(3)\tEmpirical results indicate that the proposed method greatly outperforms the other methods.\nWeakness:\n(1)\tEven the generated target domain is the \"worst case \", it is not as same as the true target domains. It would be better to discuss this in theoretical analysis.\n(2)\tWhy do not directly minimize e^*(\\mathcal{H} ; S, T) by one supervisor loss if you have known the label of generated target domain.\n(3)\tThe results of CMNIST and RMNIST in the DomainBad benchmark are not provided in this paper. \n(4)\tIn this paper, there is no comparison with other methods [3,4] in different backbone networks, such as resnet18 in PACS.\n(5)\tOnly the result of the leave-one-domain-out setting is provided. Can you give the result in single source generalization setting as EFDMix[4]?\n[1] Minyoung Kim, Pritish Sahu, Behnam Gholami, and Vladimir Pavlovic. Unsupervised Visual Domain Adaptation: A Deep Max-Margin Gaussian Process Approach. CVPR2019.\n[2] Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang. Stochastic classifiers for unsupervised domain adaptation. CVPR2020.\n[3] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. ICLR2021.\n[4] Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia and Lei Zhang. Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization. CVPR2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "MhCJqru8ii", "reviewer": "Reviewer_pXLL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles domain generalisation by synthesizing the hardest domain, and then using domain adaptation techniques. Domains are parametrized in fourier space using the amplitudes which the phases are taken from the original images. The hardest target domains synthesized by adversarial training. The classifier is learned via MCD, which is modified here as the target labels are known. Results seem better than mixup in fourier space.", "summary_of_the_review": "This is a well written paper, which tells a nice story and presents strong results. The novelty is a little limited and the domain synthesis is raises some doubts. I am borderline on this paper and waiting for the rebuttal and other reviews.", "strength_and_weaknesses": "Strengths:\n* The paper is clearly written and a pleasure to read.\n* The main ideas are intuitive\n* Augmenting MCD with the true labels is probably novel (as is an aspect of this setting)\n* Results indeed outperform many previous methods\n\nWeakness\n* Synthesizing domains in fourier space clearly does not generate realistic images - and so the idea diesn't really work in that sense - even though it did generate improvements on the baseline. The method resolves some of the issue by using mixup, but this is obviously not the main idea the paper was selling. It makes the reviewer ponder if this is the fourier parametrization is the correct one for synthesising the target domain. \n\n* The method is only slightly novel compared to the previous baseline of fourier domain mixup. Most of the advantage comes form SWAD, which is orthogonal. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Lp_nb5ajb82", "reviewer": "Reviewer_71uu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets at solving domain generalization. The authors tackle the DG problem as DA task where the worst-case `target' domain is adversarially synthesized. Specifically, they generate Fourier amplitude images and combine them with source domain phase samples. MCD is further exploited to relate the target domain performance to the discrepancy of classifiers in the model hypothesis space. Bayesian hypothesis modeling is adopted to make adversarial MCD minimization feasible. Experiments show the effectiveness of the proposed method.", "summary_of_the_review": "Strengths:\n+ This paper studies an important problem\n+ This paper is well-written, it is enjoyable to read.\n+ Experiments are comprehensive and show the effectiveness of the proposed method.\n\nWeaknesses:\n+ The contributions are a little limited considering that MCD is an existing approach.\n+ This paper should concentrate more on true solutions in DG. More discussions about the true solution in DG should be provided.\n+ Why choosing Bayesian hypothesis modeling? It should be clearly discussed.\n+ More insightful discussions should be provided. For example, how to improve the generalization ability of deep models using other adversarial DA approaches (e.g. DANN and MDD)?\n[1] Domain-adversarial training of neural networks\n[2] Bridging theory and algorithm for domain adaptation.", "strength_and_weaknesses": "Strengths:\n+ This paper studies an important problem\n+ This paper is well-written, it is enjoyable to read.\n+ Experiments are comprehensive and show the effectiveness of the proposed method.\n\nWeaknesses:\n+ The contributions are a little limited considering that MCD is an existing approach.\n+ This paper should concentrate more on true solutions in DG. More discussions about the true solution in DG should be provided.\n+ Why choosing Bayesian hypothesis modeling? It should be clearly discussed.\n+ More insightful discussions should be provided. For example, how to improve the generalization ability of deep models using other adversarial DA approaches (e.g. DANN and MDD)?\n[1] Domain-adversarial training of neural networks\n[2] Bridging theory and algorithm for domain adaptation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CV1ckTVRtn", "reviewer": "Reviewer_M2ax", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to domain generalisation whereby they use Fourier to generate synthetic images which are then used as a worst-case target domain to improve the model's robustness and therefore its generalisation. This is reminiscent of a domain adaptation approach. The amplitude-based generator that creates the synthetic images is trained in an adversarial manner and is combined with the phase source domain data. Surprisingly, the method performs competitively compared to other methods proposed.\n ", "summary_of_the_review": "This is a good submission that seems to perform well. Authors have run several experiments some appearing in the main text and some others in the appendix. There is a good empirical component and the novelty is fair. I think this is a decent submission to be considered for acceptance.\n", "strength_and_weaknesses": "Strengths:\n\na) generating target images for modelling worst-case scenario\nb) repurposing mean classifier discrepancy to propose a supervised variant that leverages the labelled generated target domain data\nc) competitive performance\nd) relative simple implementation\n\nWeakness:\n\na) not a weakness per se, but I am trying to get my head round why fourier amplitude for adversarially generating images.... is that to create bad but not too bad images deliberately to force a better generalisation performance? It sounds rather arbitrary to me (not in a negative way) so I would like to know what the thought process was to arrive to that approach vs trying other ways,", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster