Mitigating Real-World Distribution Shifts in the Fourier Domain
Authors: Kiran Krishnamachari
See-Kiong Ng
Chuan-Sheng Foo
Venue: Accepted by TMLR ()

Abstract: While machine learning systems can be highly accurate in their training environments, their performance in real-world deployments can suffer significantly due to distribution shifts. Real-world distribution shifts involve various input distortions due to noise, weather, device and other variations. Many real-world distribution shifts are not represented in standard domain adaptation datasets and prior empirical work has shown that domain adaptation methods developed using these standard datasets may not generalize well to real-world distribution shifts. Furthermore, motivated by observations of the sensitivity of deep neural networks (DNN) to the spectral statistics of data, which can vary in real-world scenarios, we propose Fourier Moment Matching (FMM), a model-agnostic input transformation that matches the Fourier-amplitude statistics of source to target data using unlabeled samples. We demonstrate through extensive empirical evaluations across time-series, image classification and semantic segmentation tasks that FMM is effective both individually and when combined with a variety of existing methods to overcome real-world distribution shifts. Code is available at https://github.com/kiranchari/FourierMomentMatching.

Published in Transactions on Machine Learning Research (08/2023)
Mitigating Real-World Distribution Shifts in the Fourier Domain
Kiran Krishnamachari
kinank wus.euduuuuuuuuu edu
Institute for Infocomm Research RI² A R, Singapore School of Computing, National Universily of Singapore, Singopore
Sco-Kiong Ng
seckiong www.ggguuuuuuuu...uu..
Institute of Data Science, National University of Singapore, Singapore School of Computing, National University of Singopore, Singopore
Chuan-Sheng Foo
foo__hhwee chuan sheng@ilr.ac w-stetrrrssstr..r. cataaarrddus.. edu.sg
Institute for Infocomam Research R1) A STAR, Singapore Centre for Frontier AT Research (CFAR), A ASTAR, Singapore
Reviewed on OpenReview: https: openreviee.. net/ formmo id- lu4oAg55iX
Abstract
While machine learning systems can be highly accurate in their training environments, their performance in real-world deployments can suffer significantly due to distribution shifts. Real-world distribution shifts involve various input distortions due to noise, weather, device and other variations. Many real-world distribution shifts are not represented domain adaplation datasets and prior empirical work has shown that domain adaptation methods developed using these standard datasets may not generalize well to real-world dis- tribution shifts. Furthermore, motivated by obscrvations of the sensitivity of deep neural networks (DNN) to the spectral statistics of data, which can vary in real-workd scenarios, propose Fourier Moment Matching (FMM), model-agnostic input transformation that matches the amplitee statistics of source to target data using unlabeled samples. We demonstrate through extensive empirical evaluations across time-serics, image classifi- cation and semantic segmentation lasks that FMM is ellective both individually and when combined with variety of existing methods overcome real-world distribution shifts. Code available https:://githb...mm//rrrrre http://hhw..oookktaaaaa///////ta
standard
1 Introduction
DNNs are known to suffer significant drops in accuracy when deployed in real-world environments different from their training distribution. For example, systems trained to classify time-series signals of electroen- cephalogram (EEG) data undergo distribution shifts between hospitals as well as patients. Acoustic classifiers are susceptible to changes in recording device such as mobile phones as well as changes in background noise. Image classifiers encounter many corruptions the wild including noise and weather changes that degrade performance. Standard domain adaptation datascts usually comprise tasks with extreme shifts between photos and synthetic target domains such as sketches between digit datasets. Prior work has shown that domain adaptation methods developed using these datasets may not have the same level of performance in real-world scenarios, highlighting the need for methods effective on variety of real-world shifts (Sagawa
et al. 2022
Xie et al. 2021 Miller et al. 2021 Djolonga et al. 2021 Taori et al. 2020
Real-worle distribution shifts across modalities have often been analysed in the Fourier-domain. For example, prior work has analysod multiple common image corruptions the Fourier-domain (Yin al. 2019). Many time-series signals are also amenable spectral analysis. In audio, frequency-domain analysis is common via the spectrogram. Relatedly, recent work has shown that DNN performance highly sensitive to shifts

---

Published in Transactions on Machine Learning Research (08/2023)
Figure I: Importance of Fourier phase vs amplitude in natural images. a,b) Examples of natural images. c) Image which has the phase spectrum of image a, and the amplitude spectrum of image b. d) Image which has the phase spectrum of image b, and the amplitude spectrun image a. Images c and are perceptually similar to the image the phase spectrum was taken from, which indicates that the phase spectrum is more important for preserving image semantics.
in the Fourier-statistics of data (Jo & Bengio 2017 Hence, in this paper, we approach the problem of adapting to real-world distribution shifts in the fourier-domatt
A fundamental result in the Fourier analysis of natural images is that amplitude spectra follow 1/f relationship, where f is frequency (Hyvärinen el al. 2009). This means that amplitude falls off inversely proportional to frequency. Hence, if all natural images have amplitude spectra that approximately have 1/f shape, they cannot carry too much information about the signal or image and thus phase information is key to preserving image semantics. This view can be demonstrated by experiments in which we take the phase spectrum of one natural image and the amplitude spectrum of another image via their respective Fourier transforms. We then compute the inverse Fourier transform this combination phase and amplitude spectra to determine which image turns out to "look" like. Figure [ demonstrates such an experiment where we can see that the images turn out to resemble the image from which the phase spectrum was taken, Similar rosults holds for other types of signals c.g. audio I (Oppenheim & Lim 1981). However, DNNs have been shown to be sensitive to spurious shifts in amplitude spectra statistics between train and test data. This motivates our approach to match amplitude statistics while leaving the phase information intact.
We thus propose an efficient model-agnostic input transformation, Fourier Moment Matching (FMM), that matches first and second-order Fourier-amplitude statistics of source data to those of the largel data. Our method is also easily applied in different modalities, e.g. time-series and image datasets, using the Fast Fourier Transform (FFT) and does not require any changes to the model architecture. In summary, the main contributions of our work are follows:
1. We propose a novel and model-agnostic input transformation, Fourier Moment Matching (FMM), which matches the first and second-order Fourier-amplitude statistics of source to tar- gel domains using unlabeled data
We demonstrate through extensive empirical evaluations on real-world distribution shifts across time- series, image classification and semantic segmentation tasks that FMM effective as a standalone method and also improve the performance of existing domain adaptation methods
2 Background
2.1 Unsupervised Domain Adaptation
Unsupervised domain adaptation (UDA) is a commonly studied setting that assumes access to labeled samples from a source domain and unlabeled samples from a target domain where we wish the model to perform well. We consider the standard setting where source and target domains have the same label space Y (1, 2,.., K), K being the number of classes. In this work, focus on real-world time-series and based tasks with distribution shifts, as these are relevant to the practical deployment of machine learning models.
https://colab.. wew.aaac..eeeec......o.a.. gwo.lgooooooooooooooooooooooo wttpleeeeccm/r/veeeeeeeee http://///ePaaneeeeeoeeTeee.ee-. HMIS6_ http:///www..tttt......aaaaaa/ta
2

---

Published in Transactions on Machine Learning Research (08/2023)
2.1.1 UDA Algorithms
UDA methods commonly perform distribution matching between domains or domainn-nnvriinn learning, generally in the embedding space of a deep neural network. CORAL (Sun et al. 2016) and DeePCORAL (Smn
& Saenko, 2016) match sccond-order statistics between domains while HoMM (Chen et al. higher order moments. DDC (Teeng et al. (014)), DAN (Long et al. 2015) and JAN (Long et al. 2017) utilize
2020 matches
maximum mean discrepancy (MMD) to reduce the gap between source and target distributions. MMDA
Rahmman et al.) combines CORAL and MMD with conditional entropy minimization, which increases the classifier's 's confidence on unlabeled data. DANN (Ganin et al. 2016), ADDA (Tzong et al. (017)) and MCD (Saito et al. 2018) learn domain-invariant features using adversarial learning. CDAN (Long el al. 2018) uses dlass-conditiona adversarial learning to learn aligned features between domains. DIRT-T (Shu et al.
2018 uses virtual adversarial training, a teacher model as well as conditional entropy. MCC (Jin et al. 2020) also uses entropy-rogulartzatin to encourage the model to be confident on unlabeled data, but with instance dependent weight that discourages over-confidence on samples that are likely to be misclassified. MCC further minimizes instance- weiihted confusion between classes. Margin Disparity Discrepancy (MDD) (Zhang et al. 2019)) is a feature-based domain adaptation method which finds representation of the imput features that minimises the disparity discrepancy (DD) between source and target domains. CoDATS (Wilson ct al. 2020 a method for time-serics data, uses adversarial training with weak supervision. AdvSKM (Liu & Xue 2021 uses adversarial spectral kernel matching for non-stationary time-series data. We further consider the following self- training or superviinn methods benchmarked in the WILDS (Sagawa et al.,
2022) framework for real-world distribution shifts: Pseudo-Tabel (Lee) 2013), FixMatch (Sohn et al. 2020):
Noisy Student (Xie et al.) 2020) and SwAV (Caron et al. 2020 While many existing methods have been effective on standard UDA dalasets, they have been found to be less effective on real-world distribution
shifts (Sagawa et al.) 2022). Hence, it is highlighted in Sagawa et al. (2022) that new methods specifically
designed for rcal-world distribution shifts are neoded.
2.2 Approaches in the Fourier-domain
The
produces an amplitude and a phase spectrum, which together completely represent any signal. The amplitude and phase spectra contain different information about the signal (Oppenheim & Lim 1981).
Generally, changes in the amplitude spectrum alters the signal but does not affect its interpretation while altering the phase spectrum affects the interpretation of the signal. Hence, we propose a phase- preserving transformation for domain adaptation to real-world shifts. Relatedly, many works have used froquency analysis to analyse the gencralization performance of models (Zhang 2019 Vasconcolos et al.,
2021 Krishnamachari et al., 2022 Tsuzuku & Sato 2019: Yin et al., 2019). DNNs have been shown to rely on the frequency statistics of the dataset. they are trained on. When tested on data with shifted Fourier-statistics, DNNs suffer significant performance degradation (Jo & Bengio, 2017). Shifts in frequency statistics are common due to various reasons such as noise, distortions and other real-world variations between domains, and are a contributing factor to poor generalization of computer vision models (Yin et al. 2019). A related work in domain adaptive semantic segmentation is Fourier Domain Adaptation (FDA) (Yang & Soatto 2020), which proposed a transformation that replaces low-frequency amplitudes in source images with those of a randomly chosen target image. FDA also employs entropy minimization and self- traininiinn.... In this work, we consider statistics of all frequencies in the Fourier-domain instead of swapping low-frequencies.
3 Fourier Moment Matching
We now describe Fourier Moment Matching, domain adaptation method operating in the Fourier domain using unlabeled source and target domain samples. Let D. = {xr} be the set of labelled source domain samples with labels L, = {ud}. and y E (1, C}. Let D. = } be unlabelled target domain samples with ar af E RRD Let F be the Discrete Fourier Transform (DFT), J-1 be the inverse DFT. The output of F(x) is complex-valued and has the same dimensionality of x. For example, if x E RD F(x) E CD.
Let A;, AA be Fourier-amplitddes of source and target samples, respectively, i.c., A [F()))) and A = [[F(()))]|- Let pa pe be mean vectors of A and A ie., μé = E[A[], He = E[A]]. C., C, are the sample covariance matrices of A, and Ae, respectively where C., C2 R Let. N denote the sample size of the
3

---

Published in Transactions on Machine Learning Research (08/2023)
Step I: Compute statistics of source and target. domains
Step 2: Train on FMM-transformed source data
FMM
Source X,
Amplitude Spectiat
Train Transformed
Step Test target data
DFT
(M)
Predict by Trained Classifier
Target X:
Amplitude Spectza
Target
Figure 2: Overview of Fourier Momont Matching to mitigate real-world distribution shifts.
source domain samples and M denote the sample size of target domain samples. FMM matches moments of the source Fourier-amplitude distribution to those of the target domain using whitening-dewhitennng procedure. First, moments of the Fourier-amplitude distributions of unlabeled source and target data are computed individually. These statistics are then used to whiten and de-whiten the source data to match target statistics (Algorithm FMM operates in the input space and does not impose any restrictions on model architecture. We proposc two variants of FMM that match either the first-order moment or both first and second-order moments. We describe the two variants of FMM below. The overall FMM algorithm is visualized in Figure 2 We note that although the transformation of source images (Step 2 of Algorithm I can be performed offline before training, the FMM transformation is efficient enough to be computed online part of the data-auugennttiin pipeline. Both online and offline methods lead to the same results and be chosen based on the computational complexity associated with the dataset at hand. We used the online implementation in our experiments below.
3.1 First-Order FMM
The first-order moments of the amplitude distributions of source and targel data are μs and μt, respectively. First-order FMM transforms the mean of the source amplitudes to that of the target domain, i.e. FMM(A))
= A Pa + Pt.
3.2
Second-Order FMM
Second-order FMM matches both the first and second-order moments of the amplitude distributions of source and target data. The mean and covariance of the source amplitudes transformed to match that of the larget domain, ie.. FMM(A)) = (A:, pa) x 1/2 1/2 C1/22 + u4-
3.2.1 Estimating high- high-dimensional covariance matrices
While sample covariance is an unbiased estimator of the population covariance, in the low sample support scenario, ie.,, when the input dimensionality D large compared the sample size n, estimation error can be high especially when dealing with arbitrary non-Gaussian distributions. Moreover, the resultant sample covariance matrix is singular (non-invertible) since it has rank most (when D n), even though the true covariance matrix is invertible. A simple solution is adding the identity matrix scaled by small value, we used x 10-3), to the estimated sample covariance matrix. Morcover, socond-order FMM requires computing the inverse and matrix square roots of covariance matrices, which are D mairices. The space and time complexity these operations can grow as O(D)), which can be infeasible for high-resolution input on standard machines, c.g. ImageNet-size images. Hence, for high-dimensional input, we use block-diagonal approximations (Appendix Figure 7 of the covariance matrices. The size of the block sub matrices, b, along the diagonal must be chosen based available computational capacity and the sample size. A block diagonal
4

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm Fourier Moment Matching
Input: D,
{a v.} labeled source domain training samples with E (1,2,..,,N; D, = {ti} are unlabeled target domain samples with j E .., M.. A model f with initial parameters 6.
2: Step 1: Compute Fourier-statistics for source and target data (Ms, C.s He, Ct)
M
Ms 1/ Σ [F(())) and M # E |F(x
4: C. N-1 Σ (1
M
p.)²
and
1 C. M-1 - -46)
6: Step 2: Transform source data to match statistics of target data in Fourier domain
7: for = 1 to N do
8: A ||F(x2))|
ccompute DFT-amplitudes] {compute DFT-phase)
P phase(F(x)))
10: if FMM: 1st Order then
11:
FMM (At) A ps + Pt
12:
else if FMM: 2nd Order then
FMM((A) (Ar na) CF 1/2 c/ + p
13:
14:
end if
15:
re= ==1-1 (FMM(A)) P.))
Jinverse-DFT]
16: Store FMM transformed source data oy
17: end for
18: Step 3: Slandard braining FMM transformed source dala
19: T training iterations
20: for to T do
21: Sample K labeled and FMM-transformed source domain images
22: for i == to K do
23:
Compute classifier loss C(x v
24: end for
25: Update classifier f(::0) minimize loss
26: end for
27:
28: Step 4: Standard evaluation on target domain data
29: for j= to M do
30: predict f(x];0)
31: end for
approximation significantly reduces the computational costs of inverting and square-rooting the covariance matrix, which are needed to perform second-order FMM. Computing the inverse or square-root of block- diagonal matrices simplifies to operating on the individual block sub-matrices (Appendix Figure 7p), which significantly reduces memory and computational requirements. In case of very low available computational capacity sample size relative to the input dimensionality, diagonal covariance matrix (equivalent to b=1) can be used that excludes off-diagonal covariance terms.
4 Experiments
We first describe experimental settings for benchmarks below (Section 4.1) We then validate that FMM is more effective than existing methods in overcoming shifts the Fourier-domain using an artificial Fourier- shift (Section 4.3 We then perform unsupervised domain adaptation experiments using classification tasks
5

---

Published in Transactions on Machine Learning Research (08/2023)
containing real-world distribution shifts across time-series (Section 4.4.1 p, image classification (Section 4.5
and semantic segmentation (Section 4.6 In all experiments, ERM (empirical risk minimization) refers to the non-adapted baseline model trained on labeled source data. "ERM + FMM" (or just "FMM"" refer to standard training on source data that have been transformed using FMM. We also combined FMM with existing domain adaplation methods. For example, "DANN + FMM" refers to training using the DANN (domain adversarial neural network) method on source domain samples that have been transformed using FMM. We also include results for "Oracle" models on each task. The "Oracle" model refers to supervised classifier trained on labeled target domain data and uses the same architecture the domain adaptation methods. It is included only to provide an estimate of the upper-bound performance achievable by domain adaptation methods. As an additional baseline, we benchmarked moment matching in input-space, i.e., matching statisties of source lo larget data in the pixel-domain for images or raw waveform of time-series and audio data. We observed that FMM could outperform input-space moment matching significantly, demonstrating the advantage of matching statistics in the Fouricr-domain (results in Appendix B
4.1 Experimental settings
4.1.1 Time-series classification
a. Sleep-stage classification: We benchmarked methods on a real-world sleep-stage classification clas- sification task using electroencephslograapy (EEG) data. We adopted the Sleep-EDF dataset (Goldberger ct al.] 2000), which contains EEG readings from 20 healthy subjects. The source and target domains this lask refer to EEG readings from different subjects in the Sleep-EDF dataset. Studies have shown that EEG signals vary across subjects due to factors such as subject-fatigue, difference in electrode placement,
impedance etc. (Buttfield et al. 2006 Li et al. 2010). This significantly hampers the accuracy of automatic sleep-stage classifiers. Hence, the problem setting is to train on labeled EEG readings of one subject (source domain) and evaluating on FEG readings from another subject (target domain). We selected a single channel (i.e., Fpz-C2), and 10 different subjects to construct five cross-domain (cross-subject) scenarios as proposed in (Ragab et al.] 2023). We used the widely used 1D-CNN as the backbone network for all methods (Ragab) et al. 2023) (see Appendix C.2 for details). All methods were trained for 40 epochs with a batch-size of 128. The Adam optimizer with fixed weight-decay (1c-4) and (81,82 (0.5, 0.99) was used to train all models. For each method, learning rate and other lyper-parameters were chosen using an extensive random search including 100 hyperparameter combinations per method and target validation set (see Appendix C.1 for details). Input samples were 3,000 time steps in length and were consistently pre-processed across mothods for fair ovaluation. FMM was implemented using the 1-dimensional Fast Fourier Transform (FFT) of the input. When adding FMM to other methods, we added the FMM transformation to their input pre-processing pipelines. We used the AdaTime library for training and evaluation (Ragab et al. 2023): we report results averaged across three random seeds.
b. Acoustic scene classification: We benchmarked UDA methods real-world distribution shifts between recording devices for an acoustic scene classification task. We used the TAU Urban Audio (Heittola et al. 2020b dataset as provided in the development set of (Heittola et al.) 2020a). The development set contains data from 10 cities and 9 devices: 3 real devices (A, B, C) and 3 simulated devices (S1-S3). The main recording device comprises a Soundman OKM II Klassik /stssio/sttuio A3, electret binaural microphone and Zoom F8 audio recorder, referred to as device A. The other devices are commonly available customer devices: device B is a Samsung Galaxy S7, device C is iPhone SE. The devices were used to record audio at 10 different acoustic scenes, e.g. airport, park, street-traffic (see Appendix D for details). We used 10 hours of labeled training data from device A as the source domain, while the smaller datascts of the other devices were used as larget domains. The audio is provided al 44.1kHz and each sample is 10 seconds long (each input is an array of length 441,000). As is standard, the input was converted to a MelSpectrogram (see Appendix D.1 for details) and fed to a ResNet18 model. FMM was performed on the raw audio input using the 1-dimensional FFT. We report results averaged across all touree-target pairs and three runs with different random seeds. We used the same method specific hyper-parameters from the sleep-stage classification task.

---

Published in Transactions on Machine Learning Research (08/2023)
4.1.2 Image classification
We benchmarked methods on four image classification tasks involving real-world distribution shifts. We evaluated methods on unsupervised domain adaptation from clean (source) to corrupted (target) images and
(Hendrycks & Dictterich) 2019)). We randomly sub-sampled 50 classes in ImageNet-C to create ImageNet50-C in order to reduce computational costs (see Appendix E.1 for details). We used the Transfer Learning Library (Junguang Jiang et al.) 2020g for bench- marking methods. On CIFAR10, we trained all models for 150 epochs using initial learning rate (Ir) that produced the best target domain validation set performance, selected from f0.1,0.01,0.00 ,0.001,, and Ir decayed by a factor of 0.1 every 50 epochs. On ImageNet50,, we trained all models for 90 epochs with an initial learn- ing rate that produced the best target domain validation set performance, selected from (0.1,0.01,0.001), and Ir decayed by a factor 0.1 every 30 epochs). For each method, hyper-parameters were selected on one task and applied to other tasks, requiring the hyper-parameters to generalize across tasks (see Appendix Table 12 This selection strategy is practical and widely adopted by many competitions. We performed the FMM algorithm the input using 3D-FFTT across the three color channels. When combining FMM with other methods, we additionally fine-tuned ir and the trade-off parameters for methods. We used the ResNet50 architecture for training on CIFAR10 and ImageNet50. For benchmarking on iWildCam- WILDS (Beery ict al. 2020: Sagawa et al. 2022) we used the Transfer Learning Library to train DenseNet 121 architectures initialized with
in (CIFAR1O-CCFARII)
preerraaid weights and finetuned for 18 epochs. Hyper-parameters including learn- ing rate and trade-off parameters that produced the best target risk were chosen per method (see Appendix Table 13 For benchmarking on Camelyon 17 7-WILDS (Bandi et al. 2018), we used the WILDS (Sagawal et al. 2022) library and the provided commande] Following the protocol in the WILDS framework, we used the model with the best validation domain performance, averaged across ten runs with different random seeds. We used the default model architecture and hyper-parameters for each method as used in WILDS.
4.1.3 Semantic segmentation
For semantic segmentation, we used the Transfer Learning Library library to train models using ERM, FDA (Yang & Soatto (020) and AdvENT (Vu et al. 2019]. We used the hyper-parameters found for each method using a target domain validation set (see Appendix Table 14p. For training HRDA (Hoyer al. 2022),, we used its official GitHub repository hrda) using the default architecture, syper-paramcters, and averaged results across three runs done in the HRDA paper.
architecture and
www.coomm/cc.. /hhoyrr//////
4.2 Selection of FMM mode
Applying FMM to a task requires selecting the statistics that are matched, i.e. first-order vs second-order FMM. In second-order FMM, we must also select an approximation of the covariance matrix, c.g. diagonal, block-diagonal or the full-covariance matrix. While both the sample mean and covariance are consistent estimators of the true parameters, computing statisties using limited datasets in practice introduces estima- tion errors. Hence, the FMM mode for task is treated as a hyper-parameter chosen using the validation set used to choose syper-parameterr. of other UDA methods as well. On the high-dimensional ImageNet50- C dataset. (D=150,528; N=65,000; M=2,500), it is not possible to perform operations such as inversion on the full covariance matrix due to memory restrictions on standard machines, necessitating the use of block-diagonal approximations. Moreover, we observed that using large block-sizes (b) at small sample sizes (N<20,000) can introduce significant estimation error that deteriorates FMM performance (Figure 3 Hence, the benefit of matching more statistics can be offset by estimation error at small sample sizes. On the other hand, first-order FMM and smaller block-sizes (b) were robust at small sample sizes (we report results using b=50 below). When combining FMM with other methods on ImageNNe500-C C, used second-order FMM with a diagonal covariance approximation (6=1). On the lower dimensional CIFAR10-C dataset (D=3,072; N=50,000), sccond-order FMM using the full-covariance matrix achieved the best accuracy generally (sce Appendix Table 15 On the small Sleep-EDG (EEG) dataset (D=3,000; N=14,000; M=6,000), first-order FMM was chosen and on the acoustic scene classification task (D=441,000; N=3,600), we chose second-order

---

Published in Transactions on Machine Learning Research (08/2023)
ERM
60
FMM (Ist order)
FMM (b=1)
FMM (b 50)
FMM (b 300)
40
FMM (b=2,000)
Accuracy
FMM 2,500)))
FMM (b=3,072)
1000
2500
20000
65000
Sample Size
Figure 3: Effect of sample size on different modes of FMM for adaptation to ImageNet50-C (Gaussian Noise). The input-dimensionallly (D 150, 528) here is large compared to the sample size. b is the block-size in the block-diagonal covariance matrix approxination.
Target
Oracle
ERM
FMM
MCC
MCD
ADDA
CDAN
DANN
11
92.2
73.1
91.9
58.0
77.3
83.4
84.5
84.4
7
88.2
17.6
87.5
44.7
63.2
74.5
75.5
75.8
5
82.4
14.9
81.4
41.1
54.1
64.3
63.3
64.2
Table 1: Domain adaptation performance (accuracy) from clean to Fourier-filtere CIFAR1O. Smaller corresponds to more filtering.
FMM with a diagonal covariance (b=1) approximation. On the high-dimensional WWildCan-WILDs dataset (D=) we used second-order FMM with b-50 for standalone FMM and b-1 when combining FMM with other UDA methods. On the Camelyon 7-WILDS dataset, we again used second-order FMM with b=20 for standalone FMM and b=1 in combination with other UDA methods. On the semantic segmenta- tion tasks, where the images have very high-resolution (D=3,145,725 we used socond-order FMM with a diagonal covariance approximation (b=1).
A rule of thumb in practice is to first apply first-order FMM, which is expected to contain the least statistical estimation error, as a first approximation. Further optimisation can be then done by using larger block- diagonal approximations of the covariance matrix depending on the available sample size. Another factor that may affect FMM's performance is the Fourier-sensitivityy (Krishnamachari et al. 2022 of models, which shows that model performance more sensitive to somee frequencies than others. Hence, matching the statistics of frequencies that a model most relies on may affect its performance more than other frequencies. We leave the study of the performance of FMM in relation to the Fourier-sensitfice of models for future work.
4.3 Validating FMM on artificial Fourier-shifts
DNNs rely on superficial Fourior-statisties of their training datasots (Jo Bengio 2017). which motivates our approach to match frequency statistics using FMM. To validate this, we generated visually similar Fourier- filtered CIFAR10 test. images using filtering in frequency space to artificially shift the Fourier-statistios of
8

---

Published in Transactions on Machine Learning Research (08/2023)
images (sce Appendix A for examples and details). This operation modified the frequency statistics of the test samples compared to the train samples by setting high-frequencies, to xero. A model trained on data with non-zero high-frequeecies significantly degraded in performance suffering up to ~-00% absolute drop in accuracy when evaluated on the filtered images (Table [ This demonstrates that mismatched Trequency statistics between training and lesting degrades performance. When we applied FMM (second- order), the model was able adapt to the shifted unlabeled data more effectively than other UDA methods we evaluated and nearly matched Oracle performance. This demonstrates that other UDA methods are not able to completely overcome the distribution shift in the Fourier-domaii effectively.
4.4
Benchmarking on time-series classification
We benchmarked methods on two time-series classification tasks with real distribution shifts.
4.4.1
Stagp classification
We benchmarked methods the Sleep-EDF dataset (Goldberger et al. 2000 to categorize EEG signals into five stages i.e. Wake (W), Non-Rapid Eye Movement stages (N1, N2, N3), and Rapid Eye Movement (REM). We benchmarked canonical UDA methods that can be applied to time-series tasks, as proposed in RRaab et al.) 2023). Standalone FMM, i.c., ERM + FMM, performed better than all other UDA methods
on this task (Table 2 Moreover, when combined with all other methods evaluated, FMM significantly improved their performance. The best performance across methods was achieved by the combination of MMDA and FMM with an accuracy of 77.0% (see Appendix Table 10 for results on each pair of source and target domains). We include results with mean and standard deviation across runs in Appendix J
4.4.2 Acoustic scene classification
For acoustic scene classification, we used the popular TAU Urban Audio dataset (Heittola el al. 2020b) FMM outperformed other adaptation methods when evaluated as a standalone method, i.e. ERM + FMM. FMM also improved other methods when added the input as pro-processing (Table 2). The best perfor- mance was achieved by the combination of CDAN, an adversarial learning method, and FMM (see Appendix Table I for results each pair of source-target domains). We include results with mean and standard de- viation across runs in Appendix
CIFARIO
CIFARIO-C
FMM
Saturate
Mosion Blur
Gaussian Noise
Snow
Contrast
Fog
Brightnes
Glass Blur
Impulse Noise
Figure& 4: FMM bridges the visual gap between source (CIFAR10) and target (CIFARIO-C) images. CIFAR1O (top-row), CIFAR10-C (middlo-row) and corresponding FMM transformed images (bottom-row).
9

---

Published in Transactions on Machine Learning Research (08/2023)
Sleep-EDD EDF
TAU Audio
Method
Standalone
Method
FMM
Standalone
Method
FMM
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
ERM
64.5
54.4
74.1
61.7
24.2
19.1
30.3
27.8
DANN
70.3
59.2
72.5
61.0
26.1
25.1
36.4
35.3
DeepCORAL
69.3
57.7
74.5
63.0
26.7
24.0
33.0
31.8
MMDA
72.3
61.4
77.0*
65.3*
22.2
20.3
29.8
29.6
DIRT-T
68.4
58.1
73.5
62.1
28.5
24.6
34.4
33.0
CDAN
71.2
59.6
76.1
63.3
29.9
38.9
37.9*
HoMM
70.1
58.3
74.3
62.4
28.7
27.1
34.6
33.4
CoDATS
67.0
58.5
72.0
60.2
27.3
25.4
33.1
31.1
DDC
69.4
57.8
74.5
62.9
27.1
24.5
34.1
32.8
AdvSKM
62.4
74.7
62.8
28.6
25.6
33.0
Oracle
87.5
77.6
87.5
77.6
45.2
44.8
45.2
44.8
Table 2: Performance Sleep-EDF (EEG) and TAU (audio) datasets. Results are in bold if FMM improves performance when added to baseline method. Rosults with are best across all methods. Results were averaged across all source-target domain pairs and three runs with different random seed.
Target
Oracle
ERM
FMM
MCC
FMM
MCD
+
FMM
ADDA
FMM
CDAN
FMM
DANN
FMM
Contrast
92.3
56.3
92.8
19.0
72.8
43.9
66.6
28.4
76.0
29.6
76.7
27.3
76.9
Elastic Transform
90.0
73.2
76.1*
68.0
72.1
70.5
66.5
72.3
74.8
73.7
74.8
73.9
74.4
Pixelate
91.8
73.5
60.3
79.2
JPEG
88.0
73.5
77.7
75.3
72.3
76.8
71.9
79.5
77.7
$1.0
78.8
80.8*
80.8
Delocus Blur
92.1
34.9
87.9*
47.5
77.7
70.0
72.4
72.1
79.1
73.5
80.2
73.8
80.4
Glass Blur
88.6
49.5
64.7
60.7
62.0
64.8
58.7
70.3
67.6
71.2
66.0
71.7*
66.4
Motion Blur
93.0
67.3
87.6*
50.0
70.9
65.9
67.3
67.5
75.2
68.3
75.7
68.7
76.2
Blur
92.5
77.2
72.1
92.9
75.8
$7.6
Prost
92.7
65.5
85.9*
55.0
72.0
68.1
68.4
69.2
75.6
69.0
75.2
69.6
76.7
Pog
93.1
74.2
89.6*
23.0
63.4
56.5
63.6
47.9
70.1
47.4
68.7
47.2
69.3
Brightness
93.7
92.0
92.4*
65.1
81.2
73.5
79.7
72.5
83.2
73.5
84.7
73.8
84.0
Gaussian Noise
88.8
31.5
77.3*
64.6
68.4
72.0
65.7
74.7
74.9
75.6
72.8
76.3
73.7
Shot Noise
90.0
74.7
76.9
Impolse Noise
93.7
33.8
83.9"
44.2
59.0
63.0
58.2
67.6
64.9
66.8
64.6
67.9
65.4
Table 3: Unsupervised adaptation performance (accuracy) from CIFAR10 (clean) to CIFAR10-C dataset for each corruption (severity 5). Results are in bold if FMM improves performance when added to baseline method. Results with are best across all methods.
4.5
Benchmarking on image classification
4.5.1
Adapting to common image corruptions
We benchmarked methods for adaptation from CIFAR10 and ImageNet to images with common corruptions in CIFARIO-C and ImageNet-C, respectively (Hendrycks & Dietterich/ 2019). These corruptions represent real-world distribution shifts that image classifiers encounter in the wild such noise, weather and compres- sion artifacts (Appendix Figure 6). FMM significantly bridged the visual gap between clean and corrupted images (Figure 4 and significantly outperformed benchmarked UDA methods many corruptions and also improved other methods when combined with them (Table 3 Notably, on some target corruptions, some common UDA methods had worse performance than the unadapted baseline ERM model, i.e, the domain adaptation method actually hurt performance. This is agreement with observations made in (Sagawa) et al. 2022) of the deteriorated performance of common adaptation methods on some real-world distribution shifts. On ImageNet50-C C, a significantly higher resolution dataset compared CIFAR10-C, FMM was both effective standalone, i.e. FRM+FMM, and when combined with other methods (Table across most target
10

---

Published in Transactions on Machine Learning Research (08/2023)
Targel
Oracle
ERtM
FMM
MCC
+ FMM
MCD
FMM
ADDA
FMM
CDAN
+ +PMM
DANN
FMM
Contrast
68.8
51.6
74.0
14.5
66.0
48.7
53.8
68.7
71.6
61.3
74.7*
71.6
72.8
Elastic trunsfou
71.9
66.3
71.2
63.9
62.9
57.0
55.6
71.1
71.8
74.0*
73.2
72.6
72.2
Pixelate
80.8
66.2
79.5
79.9
78.5
JPEG
77.8
69.8
73.7
66.4
67.5
61.4
61
74.6
76.6
75.4
76.1
76.3
76.8*
Delocus blur
77.0
55.0
74.7
49.4
66.4
55.1
58.7
70.4
76.6"
69.8
76.6"
71.0
74.4
Glass blar
76.7
59.7
68.9
58.2
67.5
57.7
61.4
73.2
75.1
73.8
76.0
75.5
77.2*
Motion blur
77.2
66.2
77.2*
57.7
67.4
56.9
61.2
74.9
76.4
74.6
77.2"
75.0
75.8
Zoom
68.1
57.0
58.0
73.8
73.6
57.7
47.2
Prost
70.4
51.8
62.1
25.5
47.3
46.9
50.1
62.4
67.2
57.6
66.1
62.7
68.2*
Pog
76.0
51.8
76.9*
25.3
55.7
64.0
58.5
67.0
72.2
66.3
71.6
65.2
71.2
Brightness
79.2
77.9
78.5
63.6
68.0
65.4
65.5
77.5
78.22
77.2
76.4
79.0*
77.5
Gaussian noise
77.2
49.0
63.3
47:7
49.5
54.1
52.4
71.9
71.0
70.6
70.6
72.0
73.6*
Shot moise
44.1
54.0
46.4
49.2
54.7
51.6
72.3*
70.4
68.5
70.2
71.6
Impulse nolse
54.9
50.1
47.0
68.2
70.5*
Table 4: Unsupervised adaptation performance ImageNet50-CC dataset (severity 2). Results are in bold if FMM improves performance when added to baseline method. Results with are best across all methods.
corruptions. The best performance was often achieved by a combination of FMM and another method (Table
Method
Standalone
Method + FMM
Method
Standalone
Method + FMM
Test Acc.
Test Acc.
Test Acc.
Test Acc.
ERM
72.6
74.9
ERM
82.0
84.1
MDD
73.5
75.7
CORAL
77.9
82.0
CDAN
71.2
73.0
DANN
68.4
70.2
JAN
68.7
75.4
Pseudo-Label
67.7
74.7
DAN
69.5
76.4"
FixMatch
71.0
83.9
DANN
70.1
72.6
Noisy Student
86.7
86.6
SwAV
91.4"
91.0
Oracle
96.5
96.5
Table 6: UDA performance on WILDS WTIDD-CCamlyon11 dataset. Results are bold FMM improves perfor- mance when added bascline method. Results with are best across all methods. Results were averaged across 10 runs with different random seeds.
Table UDA performance on WILDS-WiIdCAM dataset. Results are bold if FMM improves perfor- mance when added to baseline method. Rosults with are best across all methods.
4.5.2 Animal species classification
We benchmarked methods on domain adaptation between different camera traps in the WILDS-WiilCAM dataset. The task involves classifying animal species in images taken at different camera traps (domains), creating variation in illumination and background. We found that FMM consistently improved the perfor- mance of methods (Table 5). The best performance was achieved by the combination of DAN with FMM with an accuracy of 76.4%.
4.5.3 Tumor identification across different hospitals
We further benchmarked methods on domain adaptation between hospitals for tumor identification task using the WVLDD-CCamlyooi datasct. The task is to classify image patches as tumor or normal tissuc on data from different hospitals (domains), which can diller in their patient demographics and data acquisition protocols. We benchmarked methods evaluated in WILDS (Sagawa et al.) 2022 and found that FMM significantly improved the performance of ERM (+2.1%), CORAL (+4.1%), DANN (+1.8%). FixMatch (+12.9%) and Pseudo-Label (+6.0%). FMM did not improve or worsen the performance of Noisy Student and SwAV this task.
11

---

Published in Transactions on Machine Learning Research (08/2023)
4.6 Benchmarking semantic segmentation
We benchmarked methods for domain adaptive semantic segmentation from Cityscapes to Fog yscapes Cordts et al., 2016) and Synthia (Ros et al.) 2016) to Cityscapes. We combined FMM with baseline methods and observed improved performance on both datasets (Table 7 Notably, FMM improved the performance of FDA, which replaces only low-frequencies, demonstrating the benefit in matching statistics across all frequencies instead of only low-frequencies as in FDA. FMM also improved the performance of FRM and AdvENT (uu et al. 2019) across both tasks. On Synthia to Cityscapes, FMM improved the performance of HRDA (Hoyer el al. 2022 the current state-of-the-art method, to achieve new improved mIoU of 66.2%
(across 16 classes) or 72.7% (across 13 classes).
5 Discussion and future work
Here we discuss further analysis of our proposed method and its applicability to different tasks. We observed that there was a correlation between the degree of Fourier-shift between source and larget domains and the performance improvement provided by FMM on ImageNet50-C (Appendix F). This suggests that the degrec of shift maybe indicative of the performance gain that FMM could provide. We also evaluated FMM on standard domain adaptation tasks such as SVHN - MNIST and Office-Home (Appendix D. As these domain-shifts cannot be captured in the Fourier- amplitue spectrum alone, FMM was less effective here. Hence, FMM maybe better suited for the types of natural distribution shifts that we have considered rather than drastically different domains. We also investigated the effectiveness of matching only low-frequency statistics using FMM. However, we found that matching statistics across all frequencies provided better performance than matching only low-frequency statistics (Appendix H. Finally, we further experimented with test-time adaplation using FMM i.e. matching frequency statistics from target to source domains at test-time rather than from source to target domains at training-tinm time. This approach is suitable when it is not possible to re-train models on the source domain but we wish to adapt models at test-time. We found that test-time FMM could improve performance over ERM in many cases, although not as effectively train-time FMM. We leave exploration of the test-time adaptation setting to future work (see Appendix for preliminary results and discussion).
miol/
Road
S walk
Dicid
Wall
Fence
Pole
Light
Sign
Veget.
Toroln
Slyy
Peeson
Rider
Car
Truck
Bus
Trats
M.lile
Bike
Citysrapes
31:2
10.3
70.2
31.9
30.02
30.7
333.1
51.1
56.4
21.1
803.5
F3IM
59.0
00.5
74.8
72.0
33.2
43.0
39.6
46.5
60.3
51.4
56.3
07.6
50.1
88.8
69.2
05.6
38.0
48.7
02.5
61.8
96.5
75.1
76.4
46.2
42.6
39.8
43.6
58.0
67.5
50.4
70.6
61.7
30.9
47.9
AdvRos FMM
68.8*
87.0
76.5
62.0
13.0
41.8
41.3
48.2
61.5
81.4
54.3
78.9
68.8
52.0
80.3
71.7
73.6
43.6
50.8
65.1
FDA (3-0001)
96.9
77.2
75.8
46.5
42
89.8
47.1
61
54.6
68.4
72.8
68.
35.5
FDA (3=0000) + PMMM
64.2
46.9
76.5
77.5
45/0
45.4
40.9
48.7
00.9
76..1
54.8
67.6
68.1
50.9
80.3
72.1
74.4
56.2
52.5
05.1
FDA (3:005)
64.7
15.9
76.2
51.7
13.55
65.8
22.1
4%
61.5
82.0
53.5
731.6
08-4
51.1
143.5
73.5
73.4
301.4
50.4
83.5
FDA (3-0.0) a: FMM
65.0
96.9
76.5
$2.6
12.2
45.8
43.2
45.4
61.0
8L.8
63.7
76.5
08.7
51.5
00.4
70.2
74.1
55.2
51.6
65.4
Syothla
Chtyscapps
ERM
40.2
57.0
20.8
75.8
8.7
18.0
50.3
17.6
22.2
12.2
25.3
FMM
52.2
AdvRRn
19.8
11.7
52.5
52.5
20.7
AdvBac FMM
47.1
60.6
33.0
61.0
10.2
15.3
70.6
08.2
19.8
70.0
31.5
17.9
30.9
(3-0000)
41.0
08.9
75.8
14.0
18.4
48.9
19.5
17.7
25.1
FDA (3-0000) + FMM
45.8
78.9
34.1
77.6
10.4
17.6
77.9
82.7
47.5
15.8
72.5
83.1
10.0
50.1
(3-000)
42.4
87.5
35.5
75.3
7.4
15.8
48.9
17.5
71.5
31.9
25.2
FDA (3-0000) FM
44.1
71.2
29.0
77.8
12.7
18.8
70.4
44.8
16.9
70.8
50.7
15.6
80.4
HRDA
72.4
47.7
88.8
65.7
60.5
70.4
52.8
80.0
61.7
63.9
HRDDA+ + FMM
72.74
85.3
50.5
57.4
00.2
63.5
85.6
60.7
70.5
54.8
58.8
65.2
64.5
64.7
Table 7: Benchmark for domain adaptive semantic segmentation in Cityscapes to Pegyyiiyssaaps and Synthia to Cityscapes. Results are in bold if adding FMM to the baseline method improves performance. We report mIoU across 13 classes for Synthia to Cityscapes to be consistent with the literature. Results for classes not found or evaluated in the Synthia dataset are replaced with '_. Results marked with **: are best across methods.
12

---

Published in Transactions on Machine Learning Research (08/2023)
6
Conclusion
Matching frequency statistics between domains is simple and efficient approach to mitigate many real-world distribution shifts. Hence, we proposed Fourier Moment Matching, an input transformation that matches Fourier-statistics from source to target domains using unlabeled data. We demonstrated using extensive empirical evaluations across time-series and image tasks that FMM can improve the performance of existing methods and can also be used as a standalone method for many real-world distribution shift scenarios. We hope our work motivates further study mitigating and analysing real-world distribution shifts using the Fourier domain.
13

---

Published in Transactions on Machine Learning Research (08/2023)
References
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, and others. From detection of individual metastases to classification of lymph node status at the patient level: the CAMELYON17 challenge. IEEE Transactions on Medical Imaging, 2018.
Sara Beery, Elijah Cole, and Arvi Gjoka. The iWildCam 2020 Competition Dataset arXiv preprint arXiv:2004.13340, 2020.
A. Buttficld, P.W. Ferrez, and Jd.R. Millan. Towards robust BCI: crror potentials and online learning, IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(2):164-168, 2006. doi: 10.1109/ 7NSRE.2006.875555
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised Learning of Visual Features by Contrasting Cluster Assignments. In Advances in Neural Information Processing Systems, 2020.
Chao Chen, Zhihang Fu, Zhihong Chen, Sheng Jin, Zhaowei Cheng, Xinyu Jin, and Xian-sheng Hua. HoMM: Higher-Order Moment Matching for Unsupervised Domain Adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, April 2020. doi: 10.1609 v34i04..54 URL https:
//ojs. //ojs.aaai.. aaai
www...hh//AAAe phpp/MAT/artticl///tee//// 5744
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes Dataset for Semantic Urban Scene Under- standing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
June 2016.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' D'Amour, Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucie. On Robustness and Transferability of Convolutional Neural Networks. In 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Learning Research, 17(59):1 35, 2016. ISSN 1533-7928. URL http://jmlr wttpp//re/pp/p///////////////// html
Training of Neural Networks. Journal of Machine
A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Hausdorff, P. C. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. PhysioBank, PhysioToolkit, and PhysioNel: components of a new research resource for complex physiologic signals. Circulation, June 2000.
Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. Acoustic scene classification in DCASE 2020 Challenge: gencralization across devices and low complexity solutions. In Proceedings of the Detection and Classification of Acoustic Scenes and Events 2020 Workshop (DCASE2020), pp. 56-60, 2020a. URL https://arxiv.ott org/ abs/ 22055.002 14623
Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. TAU Urban Acoustic Scenes 2020 Mobile, De- velopment dataset, February 2020b. URL
htwp://nww..oggeeerr////111/1/1/ 3819999
Dan Hendryeks and Thomas Dictterich. Benchmarking Neural Network Robustness to Common Cor- ruplions and Perturbations. In International Conference on Learning Represenlations, 2019. URL
Lukas Hoyer, Dengxin Dai, and Lue Van Gool. HRDA: Context-Aware liigh-eeolution Domain- Adaptivee
Semantic Segmentation. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (ods.), Computer Vision ECCV 2022. Springer Nature Switzerland, Cham, 2022. URL
fttts:://lin.sprrnnge ottp:ppwwgg..aao//ooo.ooo//////.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Aapo Hyvärinen, Jarmo Hurri, and Patrick O. Hoyer. Natural Image Statistics: A Probabilistic Approach to Early Compulational Vision. Springer Publishing Company, Incorporated, 1st edition, 2009. ISBN 1-84882-490-4.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum Class Confusion for Versatile Domain Adaptation. In ECCV, 2020.
Jason Jo and Yoshua Bengio. Measuring the tendency of CNNs to Learn Surface Statistical Regulari- ties. XrXim fes, statj, November 2017. URL http://aarra...rr//ab///11.. 11561 arXiv:
1711.11561.
Junguang Jiang, Baixu Chen, Bo Fu, and Mingsheng Long. Traasfer-Learning-iie 2020. URL https:
//github. http://uuuul/aaaattt-aaaaaaaaa Library Publication Title: GitHub repository.
Kiran Krishnamachari, See- Ng, and Chuan-Sheng Foo. Fourier Sensitivity and Regularization of Computer Vision Models. Transactions on Machine Learning Research, 2022. URL https:: net/forrun3dd--aatyyyyo..
Dong-Hyur Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning, 2013.
Yan Li, Hiroyuki Kambara, Yasuharu Koike, and Masashi Sugiyama. Application of Covariate Shift Adap- talion Techniques in Brain- Computer Interfaces. IEEE Transactions Biomedical Engineering, 57(6): 1318-1324, 2010. doi: 10.1109/TBME. T0ME.2009 2039997..
Qiao Liu and Hui Xue. Adversarial Spectral Kernel Matching for Unsupervised Time Series Domain Adaptation. volume 3, pp. 2744 2750, August 2021. doi: 10.24963 10.24963/jjcai.221// 20c11..021/378. URL https:
//www. wwwai.jca..om oecei.org//ree
ISSN: 1045-0823.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning Transferable Features with Deep Adaptation Networks. In Proceedings of the 32nd International Conference on Machine Learning, July 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep Transfer Learning with Joint Adaptation Networks. In Proceedings of the 34th International Conference on Machine Learning, August 2017.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael I Jordan. Conditional Adversarial Domain Adaptation. In Advances in Neural Information Processing Systems, 2018.
John P. Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the Line: on the Strong Correlation Between Out- of -f-tstiiitttion and In-Distribution Generalization. In Proceedings of the 38th International Conference on Machine Learning, pp. 7721-7735. PMLR, July 2021. URL htttps:/ mlr http://adddnn........rrses////// press/v139/ mil miller21b. html ISSN: 2640-3498.
A.V. Oppenheim and J.S. Lim. The importance of phase in signals. Proceedings of the IEEE, 69(5):529 541, 1981. doi:
Mohaned Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan-Sheng Foo, Zhenghua Chen, Min Wu, Chee- Kcong Kwoh, and Xiaoli Li. ADATIME: A Benchmarking Suite for Domain Adaptation Time Series Data. ACM Trans. Knowl. Discov. Data, 17(8), May 2023. ISSN 1556-4681. doi: //114/3777777. URL Place: New York, NY, USA Publisher: Association for Computing
httpps://ooi org/ Machinery.
10. 1111110111455 1145/3587937
Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. On Mini- mum Discrepancy Estimation for Deep Domain Adaptation. In Joint ICCA/ECA//AMA//ICCA 2018 Workshop on Domain. Adaptation for Visual Understanding. doi: 10. 10.100779977-33000-300777-- 6.
15

---

Published in Transactions on Machine Learning Research (08/2023)
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M. Lopez. The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Fereshteh Sadeghi and Sergey Levine. CAD2RL: Real
Flight Without a Single Real Image. In Nancy M. Amato, Siddhartha S. Srinivasa, Nora Ayanian, and Scott Kuindersma (eds.), Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cumbridge, Massachusetts, USA, July 12-16, doi: 10. (556077RSS.2017.XI1 p34.html
2017, 2017.
URL http://www.ootticcppoeeeeeeee ort//ss13//r///
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang, Ex- tending the WILDS Benchmark for Unsupervised Adaptation. In International Conference on Learning Representations (ICLR), 2022.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum Classifier Discrepancy for Unsupervised Domain Adaptation. In 2018 IEEE/CVF Conference on Compuler Vision and Paltern Recognition, June 2018
Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-T Approach to Unsupervised Domain Adaptation. International Conference on Learning Representations, 2018. URI. https:://ppenreevie
net/forumiid-Hiq AW
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekkn Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. In Advances in Neural Information Processing Systems, 2020. URL https: .ddf
//proceedings.cu ienteedddinn..euuiiggg
neur ips
Baochen Sun and Kate Saeuko. Deep CORAL: Correlation Aligument for Deep Domain Adaptation. In Gang Hua and Hervé Jégou (eds.), Computer Vision ECCV 2016 Workshops, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of Frustratingly Easy Domain Adaplation. In Proceed- ings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Mca- suring Robustness to Natural Distribution Shifts in Image Classification. In Advances in Neural Informa- tion Processing Systems, 2020.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Picter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. Tn 2017 IEFE/ RSSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23-30, 2017. doi: 10.1109/IR09 2017.8202132
Yusuke Tsuzuku and Issei Sato. On the Structural Sensitivity of Deep Convolutional Networks the Directions of Fourier Basis Functions. In 2019 IEEE/CVE Conference on Computer Vision and Pattern
Recognilion (CVPR), pp. 51 31-60, Long Beach, CA, USA, June 2019. IEFF. ISBN 978-1-72813-293 doi: URL https::/ ieee org//oocmeet//85500666
Eric Theng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep Domain Confusion: Maxi- arXiv:1412.3474
mizing for Domain Invariance, December 2014.
[cs].
Fric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial Discriminative Domain Adaptation.
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Rob Romijaders, Nicolas Le Roux, and Ross Goroshin. Impact of Aliasing on Generalization Deep Convolutional Networks. In 2021 IEEE/CVI International Conference on Computer Vision (ICCV), pp. 10509-10518, October 2021. doi: 10.1109/ CCCV48892.2021.000033 ISSN: 2380-7504.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthicu Cord, and Patrick Perez. ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation. In 2019 /EFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. URL https://ieeexplore ieee. brg/document/894425
Garrett Wilson, Janardhan Rao Doppa, and Dianc J. Cook. Multi-Source Deep Domain Adaptation with Weak Supervision for Time-Series Sensor Data. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2020.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V. Le. Self-Triining With Noisy Student Improves TmageNet Classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), June 2020. URL https:://ieeexplore.. wwwpppporreeeeee......o. feeee.org/bbcmmmtt//1566000
Sang Michael Xic, Ananya Kumar, Robbic Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In-N- Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness. In International Conference on Learning Representations, 2021. URL http://///ppeeeeeeeeeeettttooooo jznizqvr15]
Yanchao Yang and Stefano Soatto. FDA: Fourier Domain Adaptation for Semantic Segmentation. In 2020 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2020. URL
wewe:iiieeerrr. dbcument//9557228/
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, and Justin Gilmer. A Fourier Perspec- live on Model Robustness Computer Vision. In Proceedings of the 33rd International Conference on Newrul Information Processing Systems. 2019.
Richard Zhang. Making Convolutional Networks Shift-Invariant Again. In Proceedings of the 36th Interna- tional Conference on Machine Learning, pp. 7324-7334. PMLR, May 2019. URL https::/ ISSN: 2640-3498.
mlr. ppwps/v97// h0t1//99//hhhg9/0 19a. html
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging Theory and Algorithm for Domain Adaptation. In International Conference on Machine Learning, pp. 4404-7413, 2019.
17

---

Published in Transactions on Machine Learning Research (08/2023)
A Fourier-filtering
A.1 Radial Fourier-filtering
The mask radius determines Fourier components that are preserved with larger radii preserving more components. We use (Ca, Co) to denole the centre of the mask and d(-:-) to denote Euclidean distance. The mask is applied on the Fourier transform each image, denoted X, followed by the inverse transform, i.e. fittered (F(() @ M,), where is the element-wise product. Fouricr-fitcerin performed on cach color channel independently, Formally, the radial mask M,
Mf(u,e) 2) = 1 0,
ifd((u, v), (Ca: Co)) r
otherwise
unfiltered
r=11
Figure 5: First image in each row the mask in Fourier space (lowest frequency at centre). White pixels preserve and black pixels set Fourier components to zero. Top row are original CIFAR10 images, other rows are Fourier-filtred with different radial masks.
18

---

Published in Transactions on Machine Learning Research (08/2023)
B Input-space moment matching
We trained models with input-space moment matching (termed IMM) (scc Table g below) using the same architecture as FMM. While IMM could improve upon the unadapted ERM baseline, the FMM transform resulted in better accuracy, especially on time-series classification and noise distortions.
Method
Sleep-EDD
TAU Audio
CIFAR10 - CIFARIOC
Acc.
F1
Acc.
F1
Noise
Blur
Weather
Digital
ERM
64.5
54.4
24.2
19.1
34.4
59.2
76.9
61.0
ERM + IMM
65.4
54.9
23.8
18.2
63.1
77.0
85.2
78.1
ERM + FMM
74.1
61.7
30.3
27.8
79.8
81.1
88.9
80.0
Table 8:
Performance Sleep-EDF (EEG), TAU (audio) and CTFAR10C. ERM+IMM is a supervised classifier trained on input-space moment matched (LMM) source samples.
19

---

Published in Transactions on Machine Learning Research (08/2023)
C Time-series: Sleep Stage Classification
C.1 Method typer-parametert
We used hyper-parameters found for cach method by Ragab et al. (2023)
were chosen using an extensive random search involving 100 hyper-parameter combinations. The hyper-parameters that produced the best target risk were chosen our experiments. The chosen hyper-parameters are listed in Table
C.2 Architecture
As used in Ragab et al. 2023), the network is a 3-block CNN with each block comprising a 1D convolutional layer, followed by Batch Normalization layer, a ReLU non-linearity and a 1D MaxPooling layer. The convolutional layer in the first block has a kernel size of 25 and stride The implementation can be accessed in the GitHub repository relcased in Ragab al. (2023
Table Hyperparameters for Sleep Stage Classification.
Method
Hyperparameter
Value
Method
Hyperparameter
Value
ERM
Learning Rate
5e-4
AdvSKM
Learning Rate
5e-4
Source loss weight
2.50
DANN
Learning Rate
5e-4
Domain loss weight
2.50
Source loss weight
8.30
Domain loss weight
0.324
MMDA
Learning Rate
5c-4
Source loss weight
4.48
Deep CORAL
Learning Rate
5e-4
MMD weight,
5.951
Source loss weight
9.39
CORAL weight
3.36
CORAL weight
0.19
Conditional Eutropy weight
6.13
DDC
Learning Rate
5e-4
CDAN
Learning Rate
1e-3
Source loss weight 2.951
Source loss weight 6.803
Domain loss weight 8.923
Domain loss weight
4.726
HoMM
Learning Rate 5e-4
Conditional Entropy weight
1.307
Source loss weight 0.197
DIRT-T
Learning Rate
5c-3
Domain loss weight
1.102
Source loss weight
9.183
CODATS
Learning Rate 1e-2
Domain loss weight
7.411
Source loss weight 9.239
Conditional Entropy weight
2.564
Domain loss weight
1.342
VAT loss weight
3.583
20

---

Published in Transactions on Machine Learning Research (08/2023)
mean
0 + 11
12 + 5
16 +
- 18
9 14
Method
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
ERM
64.5
54.4
5144
47.0
71.2
50.7
5414
52.0
62.5
57.7
79.9
64.5
FRM FMM
74.1
61.7
70.0
55.0
67.1
51.6
76.2
63.9
75.6
71.2
81.9
66.7
DANN
70.3
43.7
73.6
60.1
67.2
58.3
74.2
DANN FMM
72.5
60.2
71.4
58.6
78.2
63.5
77.4
64.3
DeeLCORAL
69.3
57.7
54.2
41.9
69.0
53.8
68.7
58.8
74.2
80.6
65.5
DeePCORAL + FMM
74.5
63.0
67.7
55.9
68.0
54.3
78.3
65.3
75.8
71.8
82.6
67.7
MMDA
72.3
61.4
52.6
41.9
90.7
65.6
71.4
59.8
77.1
72.3
79.6
67.3
MMDA + FMM
77.0
65.3
67.2
53.3
78.7
67.2
77.1
64.5
76.6
71.7
85.4
69.6
DIRT-T
68.4
58.1
33.0
29.2
75.7
62.7
74.8
61.3
74.1
68.5
84.7
68.7
DIRT-T + FMM
73.5
62.1
53.5
41.4
74.7
63.6
80.6
65.2
73.2
67.4
85.7
72.9
CDAN
71.2
59.6
48.5
39.7
80.7
65.9
69.8
59.6
77.1
69.1
79.9
63.9
CDAN FMM
76.1
63.3
62.0
49.2
77.1
64.2
80.6
65.1
76.5
70.5
84.3
67.7
HoMM
70.1
58.3
52.3
38.9
71.6
57.3
69.2
58.6
75.0
69.2
82.6
67.5
HoMM FMM
74.3
62.4
64.1
50.8
69.9
56.7
78.7
65.2
76.7
72.4
82.0
67.1
CoDATS
67.0
58.5
60.3
69.2
60.3
CoDATS FMM
72.0
71.0
DDC
69.4
57.8
54.3
42.1
69.0
53.7
68.9
58.9
74.3
68.6
80.7
65.6
DDC FMM
74.5
62.9
67.8
55.9
67.8
54.0
78.3
65.0
75.8
71.8
82.6
67.8
AdvSKM
74.1
62.4 4
68.7
59.1
74.2
58.5
72.9
61.3
72.0
65.9
82.6
67.3
AdvSKM + FMM
74.7
62.8
67.8
55.
69.8
56.7
75.7
61.1
78.5
74.1
81.7
66.7
Oracle
87.5
77.6
85.3
68.0
86.5
76.4
89.1
83.6
83.8
78.5
92.6
81.6
Table 10: Domain adaptive acoustic scene classification performance on Slecp-EDF datasct. Rosults are bold if FMM improves performance when added to baseline method. Results are averaged over three runs with different random seed.
21

---

Published in Transactions on Machine Learning Research (08/2023)
D Time-series: Acoustic Scene Classification
D.1 Melspectrogram Pre- Pre-processing
mel_ spectrorn = haABpecroorraassaaee._raao///// n o_fft=2048, hop_ 1e._length=10224 m2m f f_min=0. O, f max=44100/2, mel scale="htk"" norm=None)
mean
A + B
A - C
A + S1
A + S2
S3
Method
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
ERM FRM FMM
21.7
16.7
26.5
22.2
32.1
27.2
21.4
15.1
20.0
15.4
20.7
15.6
30.4
28.0
36.2
35.6
31.4
29.6
29.9
27.0
27.0
21.9
27.2
25.0
CDAN
28.8
27.1
37.6
36.4
44.3
22.5
20.4
18.9
CDAN FMM
38.4
35.2
CoDATS
26.2
24.6
30.1
29.3
41.8
39.6
22.6
20.3
19.2
17.0
22.9
20.8
CoDATS FMM
30.1
27.5
36.6
36.0
33.5
31.5
31.6
29.7
29.3
27.0
34.3
31.5
DANN
25.8
24.7
35.2
34.1
43.7
41.9
19.5
18.4
14.9
14.8
17.2
16.5
DANN + FMM
35.1
34.0
41.9
41.6
41.6
39
35.3
333.9
29.2
27.7
33.9
34.1
DDC
25.2
22.8
30.8
29.0
38.8
35.8
20.7
17.5
20.9
17.6
24.3
22.5
DDC FMM
33.6
32.3
37.9
37.5
39.1
38.2
29.5
28.4
29.4
25.8
34.6
34.2
HoMM
28.1
26.3
34.9
33.9
43.3
40.4
23.9
22.3
18.0
16.9
23.5
22.1
HoMM FMM
34.3
32.9
37.8
36.9
40.3
39.2
32.0
32.4
28.9
26.3
33.8
32.4
MMDA
21.9
20.1
27.4
26.0
36.8
32.9
1-1.9
13.6
13.6
12.2
18.4
16.6
MMDA FMM
28.6
28.0
32.4
32.8
44.1
43.3
22.1
22.9
22.5
22.0
28.1
26.9
AdvSKM AdvSKM + FMM
38.1
22.6
36.1
399.7
38.5
26.6
33.6
DeePCORAL
25.0
22.6
29.9
28.4
42.4
39.1
19.6
16.7
21.1
17.6
20.7
18.1
DeeCCORAL + FMM
32.8
31.6
37.4
36.9
30.8
29.0
27.3
32.9
32.1
DIRT-T
27.7
24.0
38.7
34.7
45.7
40.6
19.6
17.8
14.7
10.4
23.7
19.3
DIRT-T + FMM
33.3
31.8
40.4
38.6
44.8
43.4
26.8
25.9
29.2
26.9
30.6
30.1
Oracle
45.2
44.8
52.6
51.1
51.4
50.7
40.3
39.4
42.4
42.9
39.4
39.8
Table 11: Domain adaptive acoustic scone classification performance on TAU Urban Audio datasct. Results are bold if FMM improves performance when added baseline method. Results are averaged over three runs with different random seed.
22

---

Published in Transactions on Machine Learning Research (08/2023)
E
Image Experiments
Clean
Impulse Noise
Glass Blur
Pixelate
JPBG
0)
b)
15
15
11
28
Figure 6: a) Mean (first order moment) of Fourier amplitude-spectrum, iee, (z))ll)) of clean CIFAR1O images in log-scale. b,c,d.e) Difference between means of the Fourier-amplitude spectra of clean and corrupted images (severity 5) in CIFARIO-C, absolute values are in log-scale.
Full Covariance Matrix
Block Diagonal Approximation
a)
b)
A
0
0
0
200
0
B
0
B
100
0 C
Figure Block diagonal approximation of Fourier amplitude covariance matrix. a) CIFAR10 covariance matrix absolute values fow-frequencies in top-left). The top-left diagonal sub-matrix comprises ~60% of the total I norm. b) Block diagonal matrices be inverted square-rooted by operating individual diagonal sub-matrices. This reduces peak memory requirements compared to operating on the entire matrix.
E.1 ImageNet50
Randomly chosen ImageNet classes in ImageNet50, which contains 65,000 training images each of size 224x224:
n01440764, n01484850, n01494475, n01531178, n01632777, n01665541, n01687978, n01695060, n01749939,
n01775062, n01795545, n01818515, m01820546, n01824575, n01833805, n01914609, n01924916, n01930112,
n01950731, m01978455, n01984695, m02007558, n02012849, m02018795, n02037110, n01443537, n01514668,
n01514859, n01537544, n01592084, n01608432, n01677366, n01698640, n01728572, n01729977, n01735189,
m01740131, n01753488, n01770081, n01773157, n01773549, n01773797, n01774384, m01843383, n01955084,
n02018207, n02027492, n02028035, m02058221, m02077923.
23

---

Published in Transactions on Machine Learning Research (08/2023)
CIFARIO-C
ImageNet50-C
Method
Hyperparameter
Standalone
Method
FMM
Standalone
Method
FMM
Baseline
Learning
10
10
x 10
1 10
Momentum
0.9
0.9
0.9
0.9
Batch size
128
128
64
64
Weight deray
10
x 10-4
x 10
1 10
Epochs
150
150
90
90
Step-LR (epochs)
50
50
30
30
MCC
Learning Rate
10
10
5 10-
5 10-3
trade-off (A)
1.U
0.1
1.0
0.5
temperature
2.5
2.5
2.5
2.5
MCD
Learuing Rate
10
1 10-1
1 x 10-3
1 10-1
trade-off (A)
0.1
1.0
0.1
trade-offeentropy
0.03
0.03
0.03
0.03
ADDA
Learning Rate
10
10
10-2
10-1
trade-off (A)
0.1
0.1
0.1
0.5
CDAN
Learning Rate trade-off (A)
10-2
10-2
10-20-
10-1
1.0
1.0
1.0
DANN
Learning Rate
1 x 10
1 1x10
1 x010
1 x 10
trade-off (A)
1.0
1.0
1.0
1.0
Table 12: Hyper-parameters for methods
CIFAR1O-C and ImageNet50-C datasets.
Method Hyperparameter
Standalone
Method + FMM
Bascline
Learning Rate Epochs Batch size
1.0
18
18
24
24
DANN
Learning Rate trade-off (A)
1.0
0.3
0.3
DAN
Learning Rate trade-off (A)
0.3
0.3
MDD
Learning Rate
0.3
0.3
trade-off (A)
0.3
CDAN
Learning Rate trade-off (A)
0.3
0.3
0.3
0.3
JAN
Learning Rate
0.3
0.3
trade-off (A)
0.3
0.3
Table 13: Hyper-parameters for methods on iWildCAM dataset.
Method
Hyperparameter
Standalone
Method
FMM
Baseline
Learning Race
-3
Weight decay
10-4
10-1
Momentum
Epochs
Ratch size
AdvENT
trade-off (X)
0.001
0.001
FDA
trade-off (A)
0.001
0.001
ITA (robust entropy)
2.0
2.0
Table 14: Syper-parameters for semantic segmentation methods.
24

---

Published in Transactions on Machine Learning Research (08/2023)
+ FMM (b=1)
+ FMM (b=3,072)
Target
ERM
Contrast
56.3
86.1
92.8
Elastic Transform
73.2
76.5
76.1
Pixelate
41.0
58.7
73.5
JPEG
73.5
72.5
77.7
Defocus Blur
54.9
86.7
87.9
Glass Blur
49.5
59.3
64.7
Motion Blur
67.3
84.2
87.6
Zoom Blur
65.2
79.5
84.0
Snow
75.8
84.2
87.8
Frost
65.5
79.1
85.9
Fog
74.2
86.6
89.6
Brightness
92.0
92.3
92.4
Gaussian Noise
31.5
60.0
77.3
Shot Noise
37.9
63.8
78.1
Impulse Noise
33.8
53.7
83.9
Table 15: UDA performance (accuracy) on CIFAR10-C for socond-order FMM with diagonal (b=1) full-covariance (b=3,072) matrix approximations.
25

---

Published in Transactions on Machine Learning Research (08/2023)
F Degree of Fourier-shift and FMM performance improvement on ImageNet50-C
We computed the correlation between the degree of Fouricr-shift, mcasured by the difference in the first or second statistical moment between source and target domains in the ImageNet-C task, and the relative performance improvement provided by FMM over ERM. While we did not find any significant correlation between performance improvement and shift in the mean of the amplitud spectrum between source and target data (Figure &h, Pearson correlation -0.03), we observed higher correlation between performance improvement. and shift in the covariance structure (Figure D Pearson correlation +0.47). This maybe indicative of the importance of matching the covariance structure this dataset.
Mean Frequency Shift Relative Improvement
b)
Covariance Frequency Shift Relative Improvement
Mean Frequency St Trancline for Mean FrequencySinn
Covar sncs Prequency CN
Thand ins for Covertance Frequency Shit
50000
0.0
10000
0.04
30000
0.0
20000
Coealanee
0.22
10000
001
0
0.00
Reletive Imprommerr,
Relative (nprosmennt())
Figure 8: a. Relative performance improvement of FMM over ERM and the norm of the difference between the first moments (mean) of source and target domains in ImageNet50-C. b) Relative performance improve- ment of FMM over ERM and the norm of the difference between the second moments (covariance) of source and target domains ImageNet50-C.
26

---

Published in Transactions on Machine Learning Research (08/2023)
G Test-time adaptation using FMM
We further evaluated FMM at test-time i.. rather than matching source to target statistics, we matched larget samples to match source statistics at test time. This setting is called lest-lime adaplation and does not require any modification to the model or training procedure on the source domain. Hence, this approach is beneficial when it not possible to re-train models on the source domain but we wish to adapt models test time. We found that test-time FMM could improve performance in many cases, although effectively as train-time FMM (Tables T6 17 This suggests that there is some additional benefit in training the model the transformed source samples rather than just modifying the target samples at test-time. In the case of adapting to distorted images, one reason for this maybe that it is easier to distort clean images compared to denoising distorted images back to their clean versions at test-time. This could explain why test-time FMM worsens performance in some cases on the CIFAR10-C task (Table 17 We leave further exploration of the test-time adaptation setting for future work.
Sleep-EDF
TAU Audio
Method
Acc.
F1
Acc.
F1
ERM
64.5
54.4
24.2
19.1
ERM + FMM (test-time)
71.1
59.1
33.0
30.7
ERM FMM (train-time)
74.1
61.7
30.3
27.8
Table 16: Performance of test-time FMM on Sleep-EDF (EEG) and TAU (audio) datasets. Results were averaged across all source-target domain pains and three runs with different random seed.
+ FMM (test-time)
+ FMM (train-time)
Target
ERM
Contrast
56.3
76.5
92.8
Elastic Transform
73.2
51.4
76.1
Pixclate
41.0
52.1
73.5
JPEG
73.5
69.0
77.7
Defocus Blur
54.9
63.6
87.9
Glass Blur
49.5
52.6
64.7
Motion Blur
67.3
62.4
Zoom Blur
65.2
73.0
84.0
Snow
75.8
29.1
87.8
Frost
65.5
74.5
85.9
Fog
74.2
82.0
Brightness
92.0
88.0
92.4
Gaussian Noise
31.5
62.8
77.3
Shot Noise
37.9
64.8
78.1
Impulse Noise
33.8
57.1
Table 17: UDA performance (accuracy) on CIFAR10-C for ERM vs FMM (Lest-time) vs FMM (train-time).
27

---

Published in Transactions on Machine Learning Research (08/2023)
H Matching low-frequency statistics
We restricted FMM to the low-frequenciies to investigate the effectiveness of matching statistics of only low- frequencies instead of all frequencies. We found that matching only low-frequency statistics, which similar to Fourier Domain Adaptation (Yang & Soatto, 2020,, was not as effective as matching statistics across all frequencies. Here we varied 3, where / refers to the proportion of frequencies used to perform FMM, from the lowest highest frequencics. Smaller values of / incorporate only low frequencies while higher values incorporate both low and high frequencies (B refers to using all the Irequencies in standard FMM). We observed improved performance increased B 1 (Tables 18
19
Sleep
EDF
Method
Acc.
F1
ERM
64.5
54.4
ERM + FMM (3 0.01)
64.5
54.0
ERM + FMM (3 0.02)
64.7
54.1
ERM + FMM (3 0.2)
66.2
55.4
ERM + FMM (3 0.4)
69.8
58.1
ERM + FMM (3 = 0.6)
69.9
58.5
ERM + FMM (3 0.8)
70.0
58.6
ERM + FMM (3 1.0)
74.1
61.7
Table 18: Performance of FMM on Sleep-EDF (EEG) al varying 8. Smaller 3 values uses only lower- frequencies while higher values use lower and higher frequencies. ==1 corresponds to standard FMM (uses all frequencies). Results were averaged across all source-targer domain pairs and three runs with different random seed.
Method
Gaussian Noise
Frost
Defocus blur
Contrast
ERM
31.5
65.5
54.9
56.3
ERM + FMM (B = 0.2)
31.6
69.5
56.3
57.0
ERM FMM (B 0.4)
35.7
71.9
61.2
58.9
ERM + FMM (8 0.6)
38.9
74.7
75.5
78.6
ERM + FMM (8 0.8)
42.9
80.2
81,2
ERM + FMM (B 1.0)
77.3
85.9
87.9
92.8
Table 19: Performance FMM on CIFARIO-C at varying B. Smaller values uses only lower-frequencies while higher values lower and higher frequencies. corresponds standard FMM (uses all frequen- cies).
28

---

Published in Transactions on Machine Learning Research (08/2023)
Evaluating FMM on standard domain-shiff datasets
FMM was specifically designed for tackling natural variations in real-world applications many existing domain adaptation methods were shown to be less effective here. Existing domain adaptation methods were largely developed on standard domain-shift datasets e.g. SVHN + MNIST or Office-Home. As these domain-shifts cannot be captured in the Fourier-amplitude spectrum alone, FMM was less effective here Hence, FMM is suited for natural distribution shifts in real-world applications rather than drastically dillerent. domains.
(Tables 20 21
SVHN MNIST
MNIST
USPS
USPS
MNIST
ERM
76.9
74.1
74.6
ERM + FMM
77.6
74.4
81.4
76.9
CDAN
95.8
93.8
CDAN + FMM
96.0
93.5
96.6
97.
JAN
87.0
90.3
84.0
JAN + FMM
88.8
90.3
85.1
90.9
DANN
92.5
90.8
91.7
95.2
DANN + FMM
91.0
83.0
93.6
96.5
Table 20: Evaluating FMM on shifts between MNIST, SVHN USPS.
Ar
Rw
Cl
Rw
Pr
Ar
C
Ro:
Av
Rw
CI
Rw
Pr
52.2
71.8
ERM FMM
58.7
40.0
68.1
73.7
52.5
62.1
64.2
33.2
37.5
73.0
65.2
40.7
76.0
CDAN
69.7
62.4
75.5
CDAN FMM
69.3
51.2
71.6
77.3
61.2
70.8
71.5
60.7
64.5
80.3
75.0
60.8
84.1
DANN
53.8
62.6
55.8
67.3
55.8
55.1
77.9
60.7
DANN FMM
65.2
52.3
62.2
73.7
55.1
66.7
07.4
39.8
54.6
78.4
70.7
59.7
JAN
65.9
50.8
71.9
26.5
60.6
63.3
68.7
60.5
49.6
70.9
71.0
55.9
80.5
JAN FMM
65.1
49.8
70.6
73.9
50.3
67.7
69.1
60.1
47.9
76.8
71.7
56.2
80.5
Table 21: Evaluating FMM on shifts between Office-Home domains: Ar (Art), Cl (Clipart), Pr (Product), Rw worll-
29

---

Published in Transactions on Machine Learning Research (08/2023)
J Benchmarks
Sleep-EDF
TAU Audio
Method
Standaloue
Method + FMM
Standalone
Method
FMM
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
ERM
3.8
54.4
+
74.1 +
61.7
24.2
19.1
30.3
2.2
DANN
70.3
3.9
59.2
+ 3.6
72.5 + 2.7
61.0 +: 3.5
26.1 + 3.7
25.1
1
3.8
36.4
2.4
35.3 +: 3.0
DeePCRRAL
69.3 + 2.5
57.7
+ 1.7
74.5 + 3.1
63.0 = 3.9
26.7 = 2.1
24.0
2.3
33.0
2.0
31.8 = 19
MMDA
72.3 1 4.7
61.4
1 4.3
77.0* - 2.8
65.3* 2.5
22.2 1 4.6
20.3
1. 4.7
29.8 1.2
29.6 1 1.7
DIRT-T
68.4 + 4.2
58.1
+ 3.5
73.5
2.7
2.
28.5
24.6
+ 3.5
34.4
33.0 = 2.3
CDAN
71.2 6.6
59.6
1
76.1 1 1.6
63.3 16
29.9 1 2.7
28.4
2.6
2.6 38.9"
37.9* 2.7
HoMM
70.1 2.0
58.3
1.8
+
62.4
34.6
3.5
CoDATS
4.2
58.5 1:
72.0 +: 10.4
60.2 6..
422
25.4
4.3
33.1 3.9
31.1 =
DDC
69.4 2.4
57.8 + 1.7
74.5 3.1
62.9 +.9++39 3.9
27.1 + 3.4
24.5
3.3
34.1 1.5
32.8 1.4
AdvSKM
74.1
2.7
62.4 +: 1.6
74.7 +: 3.3
62.8 2.2
28.6
2.3
25.6
1:
2.6
33.0
3.5
31.9 3.6
Oracle
87.5 1
1.03
77.6
1 2.2
87.5 1 1.03
77.6 1. 2.2
45.2 1.8
44.8
2.0
45.2 1.8
44.8 1 2.0
Table 22: Performance Sleep-EDF (EEG) and TAU (audio) datasets. Results are in bold if FMM improves performance when added to baseline method. Results with are best across all methods. Results were averaged across all source-target domain pairs and three runs with different random sced.
Method
Standalone
Method FMM
Method
Standalone
Method + FMM
Test Acc.
Test Acc.
Test Acc.
Test Acc.
ERM
72.6 + 1.2
74.9 + 1.4
ERM
82.0 + 7.4
84.1 + 5.4
MDD
73.5 + 1.9
75.7 + 1.3
CORAL
77.9 + 6.6
82.0 + 6.5
CDAN
71.2 + 1.7
73.0 + 1.5
DANN
68.4 + 9.2
70.2 = 7.5
JAN
68.7 = 1.6
75.4 1.2
Pseudo-Tabel
67.7 + 8.2
74.7 8.1
DAN
69.5 2.0
76.4" + 2.1
FixMatch
71.0 + 4.9
83.9 6.0
DANN
70.1 1.4
72.6 1.6
Noisy Student
86.7 + 1.7
86.6 2.0
SwAV
91.4" = 2.0
91.0 = 2.5
Oracle
96.5 + 1.9
96.5 + 1.9
Table 24: UDA performance on dataset. Results are bold FMM improves perfor- mance when added to baseline method. Results with are best across all methods. Results were averaged across 10 runs with different random seeds.
Table 23: UDA performance on WILDS-IWiICCAN datasct. Results are bold if FMM improves perfor- mance when added to baseline method. Results with are best across all methods.
30

---

Published in Transactions on Machine Learning Research (08/2023)
Target
Oradl
BBM
FM
MCC
FMM
MCD
PMM
ADDA
PMM
CDAN
FMM
DANN
FMM
Controst
92.3
92.8
40
76.0
76.7
4.3
76.9
Teaasiou
86.0
9L.
73.5
73.4
77.6
T8.5
79.2 1.2
JPRG
53.0
77.7
72.3
70.8
79.5
71.3
78.8 3.1
90.97
80.81
Deforus Blur
80.1
87.0"
77-7
700
72.1
19.1
N0.2
73.8
80.4
Glase Bour
58.6
40.5
64.7
2 3.2
60.7
2.2
62.0
1.9
64.8
= 2.1
56.7
1.8
70.3
1.5
57.6
11.2
06.0
2.0
T1.1" 4: 1.3
66.4
1.9
Motton Dlur
10.0
67.3
2.2
87.0"
1:
2.9
50.0
1.0
70.0
1: 2.1
65.9
1.0
67.3
1.8
07.3
1.9
T0.2
2.1
68.3
1.9
75.7 1: 2.1
08.7
76.2
2.8
Zoom Hluz
92.5
60.2
8
$4.0"
2 9.1
40.0
2:
70.3
6.1
60.9
8.2
71.2
2.1
71.8
1.9
77.2
71.2
79.3
8.2
72.3 1.8
70.0 =:
Suuw
92.9
70.8 8:
87.8"
8: 2.3
57.6
73.8
3.2
69.2 2.8
T1.9 1.7
69.2
8.3
70.1
09.5
70.5 2.1
70.2 1.8
76.3 1.4
Froes
W2.7
05.5
65.11*
2.7
72.0
75.6
75.2
1.3
71.77
Fog
80.7
92.4"
72.3
53.2
54.7
77.3"
72.0
72.8
70.7
8000
68.5
722
74.3
76.9
1.2
748
Imprlio Notse
00.7
33.8
83.9
39.0
20
030
3.1
58.2
02.0
84.9 311
08.8
64.0 2.4
07.9
1.0
65.4
Table 25: Unsupervised adaptation performance (accuracy) from CIFAR10 (clean) to CIFAR10-C dalaset for each corruption (severity 5). Results are in bold if FMM improves performance when added to baseline method. Results with are best across all methods.
Target
Onalle
FRM
PVM
MCC
PMM
MCD
FMM
ADDA
FMM
CDAN
FMM
DANN
PYIM
Chureast
00.
51.6 1 3.4
74.0 : 2.4
34
66.0 = 2.6
68.7 = 1.9
58.8 1
68.7 1.8
71.0 2.2.1
61
T4.7 28
7L6 0.9
72.8 1.1
Fluatle trandfrm
71.
683
11.2 3.5
659
03. 1.1
$7.0 1.5
558
71.1
1.0
71.8 1.7
71/0 +
13.3 9.
72.6 0.9
72.2 + 1.
Pisclate
80.8
81.3"
T8.5 0.9
JPEG
75.4
57.7
Buu
77.2"
67.4
766
77.2"
75.8
bler
78.4
08.1
$7.0 0.8
71.5
75.3
73.6
74.4
67.7
01:2
47.0
0.
02.1
47.3
18.9
50.1
67.22
57.6 50
68.1
00.7
For
70.0
51.8
76.9*
25:1 3.6
65.7
43
$4.0 21
58.5
67.0 2.1
72.2 3.1
06.3 1.9
71.6 2: 25
15.2 0.9
71.2 2.1
Beighttune
79.2
77.9
1: 0.7
78.5 2 0.9
63.6 03.6.2111 1.0
68.0
1.2
66.4
5 0.5
65.6 1 0.3
77.5 0 0.8
78.2
1.0
77.2 1.2
76.4 2.1
70.0P 1 2.1
77.6 4: 2.0
Causeion minaa
77.2
190
+ 2.3
03.3 8: 3.1
47.7 19
49.5
2.0
51.1 1.9
52.1 2.5
71.9 0.0
71.0 2:4
70.6 23
70.6 5 3.0
72.0 2.1
75.8"
Shor molse
75.8
43.1
8 01
54.6
8 2.1
8644 12
49.2
13
54.7 8 41
514 311
723" 8 3.2
70.0
23
08. 8 066
70.2 8: 122
71.6 8 2.1
71.00
2.1
Imgelse undae
70,0
11.3 1
54.9
23
4L9 12
1.
50. 41
470
3 2.4
07.7 1.2
69.3
1.3
68.2
666 2: 1.2
70.0" 4 03
68.5 a: 1.8
Table 26: Unsupervised adaptation performance ImageNet50-C dataset (severity 2). Results are in bold if FMM improves performance when added to baseline method. Results with are best across all methods.
31

---

Published in Transactions on Machine Learning Research (08/2023)
K Domain Randomization using Fourier-moments
We experimented with domain randomization (DR), which is an orthogonal approach to domain adaptation (DA). Domain randomization aims to achieve improved generalization to real-world scenarios by simulating multiple environments during training by randomizing properties such as lighting conditions, position of objects etc (Tobin et al. 2017 Sadeghi & Levine 2017). We explored randomization of the Fourier-moment of the source domain rather than matching the statistics of the target domain. We randomized the Fourier- moments of the source data by adding varying degree of random Gaussian noise to the Fourier-moments. This approach exposes the model to different frequency statistics at training and hence, may potentially help improve its robustness in the target domain. However, did not observe any improvement in performance on the target domain due to this procedure (Table 27], which suggests that merely randomizing frequency statistics is less beneficial than matching target domain statistics to achieve domain adaptation.
Sleep-F.D FDF
Method
Acc.
Fl
ERM
64.5
54.4
ERM DR (Ju 0; 1.0)
63.7
53.5
ERM + DR (μI 0; = 2.0)
63.7
53.5
ERM DR. (μ1 0; 3.0)
63.7
53.5
ERM + (μI 0; a = 4.0)
63.7
53.5
ERM DR. (JI 0; a 5.0)
63.7
53.5
ERM + (μI 0g 6.0)
63.7
53.5
ERM + FMM
74.1
61.7
Table 27: Performance of domain randomization (DR) and FMM on Sleep-EDF (EEG). Results were aver- aged across all source-target domain pairs and three runs with different random soed.
32


Reviews:
[{"review_id": "AKgbK73Bzc", "reviewer": "Reviewer_R9yh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Straightforward proposal, the Fourier moment matching is conceptually and implementation-wise very simple, yet quite effective.\n- Broad range of experiments show that the approach is effective at reducing the domain shifts.\n- Unsupervised approach, of practical use\n\nWeaknesses:\n- Motivations are quite ad hoc, it seems that the neural networks are sensitive to some spectral elements in images and time series, but this is not verified in a more formal way, out of the experiments with FMM.\n- Technical contribution remains thin (although it leads to surprisingly good results).\n- The approach has some similarities to domain randomization, used in robotics. I think comparison with some baselines for domain randomization would better position the approach and make a comparison with other state-of-the-art approaches.\n- The results are reported for domain adaptation settings where the normalization in spectral domain would be likely to help (e.g., image corruption, time series) and where the gaps between the domains are not that big.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to make domain adaptation through aligning data in the spectral domain, that is matching the Fourier moment statistics of source to target domain samples. That approach is based on the assumption that neural networks are sensitive to spectral elements, such that the Fourier moment matching (FMM) allows a normalization that is effective for reducing the effect of the distribution shift between source and target. The matching is done through the first two moments in the Fourier domain, that is to normalize the mean and covariance values of amplitude of the source domain to match those of the target domain. For large images (high resolution) matching of covariance is done through blocks of subimages. Experiments show significant improvement regard unsupervised domain adaptation both for image corruption (CIFAR-10, ImageNet), time series (EEG, Audio), different capture settings (WILDS-iWildCAM), different collections settings (WILDS-Camelyon17), semantic segmentation (Cityscapes, FoggyCityscapes, Synthia), where FMM appears to provided gains in most situations, when combined with other domain adaptation methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some comparison with other similar baselines would be required to ensure that the proposed FMM approach. For instance, domain randomization has been used in robotics for several years, see the following blog post for a good description of it (with references herein).\n\nhttps://lilianweng.github.io/posts/2019-05-05-domain-randomization/\n\nThis has been proposed for sim2real settings, where the source domains are simulated, allowing a fine control over randomization done over the simulation configurations. The current proposal is to make normalization in the Fourier domain between source and target domains. I would guess that making randomization over the Fourier domain on the source domains may be a way to achieve similar results than the FMM approach proposed. The difference between randomization and normalization over the first and second statistical moment in the Fourier domain would provide further insights on how the current proposal is effective and how sensitive neural networks are to shift in the frequency domain.\n\nAlso, the experiments have been presented for situations that appears favorable to a signal processing approach acting on the frequency domain (e.g., image corruption, EEG, audio), where shifts are likely to be limited. I am wondering whether the authors have experimented with other datasets where the shifts are more fundamental, like on digits datasets (MNIST, MNIST-M USPS, SVHN) or the PACS dataset (https://paperswithcode.com/dataset/pacs), where the shift is less associated to spectral elements, but rather shifts on the content of the images. These can be interesting results to provide, providing some more insights on how widely applicable is the proposed approach.", "broader_impact_concerns": "There are no specific ethical concerns to mention for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jNs7Yx8zM2", "reviewer": "Reviewer_z9wq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n* Figure 1 brings a traditional insight from Fourier phase spectra to the training and generalization of deep neural networks. \n\n* The proposed method is evaluated on two domains and two datasets each. \n \n\n# Weaknesses\n\n* Tables 1-4 show the effect of matching the Fourier amplitudes with the proposed method. The results suggest that including this method improves accuracy. However, these results do not allow us to distinguish the performance of FMM from data augmentation in general. Have results been compared to other data augmentation techniques with similar computational costs? (I.e. calculating the Fourier spectrum etc, also requires computing)\nTo be specific, Figure 3 compares FMM to ERM. However, it is unclear what data augmentation is used for ERM and how that compute/intensity compares to FMM.\nSecond, could the evidence be corroborated by showing that changing the Fourier amplitudes could worsen the results also? The argument for 'matching Fourier amplitudes could improve generalization' would be more robust by showing that 'unmatching the Fourier amplitudes would worsen generalization'. Section 4.3 explores this, but the sentences are unclear. \n\n* Table 4 does not compare against existing work or established methods. What argument is made for the current comparison? For example, results from AugMix [3] seem significantly better than those presented here. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of out-of-distribution generalization, where a model is trained on a particular dataset but tested on another dataset with slightly different characteristics. The paper is motivated by observations from Fourier Spectra that phase spectra seem to preserve objective image content (Figure 1), while the amplitude spectra may vary (Figure 1). However, the results don't compare with other works, and a quick comparison with literature shows that existing approaches already achieve better error rates.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main points are made in 'strengths and weaknesses' section; these are minor comments.\n\n\n# Minor comment:\n\n  * These works combine ideas from signal processing, Fourier theory, and deep learning. As such, works like [1][2] could be useful for reference.\n\n  * Section 4.2 explores block diagonal approaches for the covariance matrix. Is the separation of the Red, Green, and Blue channels used here?\n\n* I would guess that some parts of CIFAR-C and IMAGENET-C have more shifts in the Fourier spectrum than others. Could the proposed method be shown to be better or worse on such perturbations in the Imagenet-C set, compared to others?\n\n[1] Zhang, Richard. \"Making convolutional networks shift-invariant again.\" ICML, 2019.\n\n[2] Vasconcelos, Cristina, et al. \"Impact of aliasing on generalization in deep convolutional networks.\" ICCV 2021.\n\n[3] Hendrycks, Mu, Cubuk, Zoph, Gilmer, Lakshminarayanan \"Augmix: A simple data processing method to improve robustness and uncertainty.\" arXiv 2019\n", "broader_impact_concerns": "N.A. The research works on distribution shift, which could have beneficial effect for critical applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VLLORKlQOk", "reviewer": "Reviewer_3oxK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n========\n- The paper is well-written and easy to follow for the most part\n- The experimental investigation is very thorough, covering different modalities (audio and images), different application scenarios, different “base” domain adaptation methods\n- The results are very strong, demonstrating that FMM can be successfully combined with several domain adaptation methods to improve their performance\n- I really appreciated the transparency w.r.t how hyperparameters are tuned (and the treatment of first- vs second-order as a hyperparameter).\n\nWeaknesses\n===========\n- Some minor clarity issues\n- Some experimental details missing or unclear\n- Would be nice to perform further analyses of results (see below)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method, Fourier Moment Matching (FMM) for domain adaptation which is based on the idea of mitigating the distribution shift in the Fourier domain. Their approach is motivated by observations that deep neural networks are sensitive to the spectral statistics of their training data and thus may not generalize well to “target domains” of different spectral statistics. They find in particular that images are perceptually similar to other images with similar phase rather than similar amplitude (so the phase spectrum is more important for perceptual similarity than the amplitude spectrum). Therefore, they propose a phase-preserving transformation to match the first- (and, optionally, also the second-) Fourier amplitude statistics of the source and target domains. Note that computing these statistics requires only unlabeled data (which DA methods assume we have available even for the target domain). Their approach proceeds by first computing the Fourier statistics for the source and target data, independently. Then, they transform the source data to match the target statistics (in Fourier domain), and store a copy of the modified source dataset. Finally, they train the chosen domain adaptation method (ERM, or any other) on the modified source data and proceed as usual.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify the scope of the proposed method’s motivation. Specifically, the motivation was based on the finding that the phase spectrum matters more than the amplitude spectrum for perceptual similarity. However, that finding was in the context of images. Are there similar results for other modalities, like audio?\n\n- It would be useful to state what exactly is F(x^i_s) / F(x^i_t), and what is its dimensionality.\n\n- The setup for the Sleep-stage classification task was not clear to me. The authors describe what the classification task is but not really what the distribution shift is. The authors mention constructing “five subject-wise cross-domain scenarios” but this isn’t sufficient information to understand the setup.\n\n- It sounds like FDA is the most related out of the related works, but it seems to be specific to image segmentation (and thus the authors only compare against it for their segmentation experiments). Is it possible to derive an FDA-like approach for the other tasks too, that considers only the low-frequency amplitudes? It would be interesting to investigate whether / under which conditions considering only low-frequency suffices, over FMM which matches statistics across all frequencies.\n\n- Do the authors have some intuition for which conditions (in terms of the task, the distribution shift, the “base” domain adaptation method, etc) lend themselves well to seeing improvement by combining FMM with that domain adaptation method? The experimental results show that in many cases there is an improvement, but there are some in which FMM decreases the performance of other domain adaptation methods (e.g in Tables 4, 7, among others). What is common among these failures? Is there a rule of thumb we can derive for when using FMM is most beneficial?\n\n- How did the authors assess statistical significance (and thus bolding in tables)? Were confidence intervals computed?\n\n- Could one do the reverse of FMM? That is, instead of adjusting the source statistics to match the target, could one adjust the target statistics to match the source? How are these two different and what are some pros and cons?\n", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple approach to mitigating distribution shifts in the Fourier domain. The paper is well-written, the presented idea is simple, and the experimental evaluation is very strong (spans different domains and datasets considered, and demonstrates the proposed method works surprisingly well given its simplicity). The authors have addressed reviewers questions and concerns during rebuttal period.
Decision: Accept (poster)