Proportional Amplitude Spectrum Training Augmentation for Synthetic-to-Real Domain Generalization
Authors: Prithvijit Chattopadhyay
Kartik Sarangmath
Vivek Vijaykumar
Judy Hoffman
Venue: Submitted to ICLR 2023 (2023)
Keywords: Synthetic-to-Real Generalization
Fourier Space Augmentation
Single Domain Generalization

Abstract: Synthetic data offers the promise of cheap and bountiful training data for settings where lots of labeled real-world data for some task is unavailable. However, models trained on synthetic data significantly underperform on real-world data. In this paper, we propose Proportional Amplitude Spectrum Training Augmentation (PASTA), a simple and effective augmentation strategy to improve out-of-the-box synthetic-to-real (syn-to-real) generalization performance. PASTA involves perturbing the amplitude spectrums of the synthetic images in the Fourier domain to generate augmented views. We design PASTA to perturb the amplitude spectrums in a structured manner such that high-frequency components are perturbed relatively more than the low-frequency ones. For the tasks of semantic segmentation (GTAV→Real), object detection (Sim10K→Real), and object recognition (VisDAC Syn→Real), across a total of 5 syn-to-real shifts, we find that PASTA  either outperforms or is consistently competitive with more complex state-of-the-art methods while being complementary to other generalization approaches.

Under review as a conference paper at ICLR 2023
PROPORTIONAL AMPLITUDE SPECTRUM TRAINING AUGMENTATION FOR SYN-TO-REAL DOMAIN GENER- ALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Synthetic data offers the promise of cheap and bountiful training data for settings where lots of labeled real-world data for some tasks is unavailable. However, models trained on synthetic data significantly underperform this paper, we propose Proportional Amplitude Spectrum Training Augmentation (PASTA), simple and effective augmentation strategy improve out-of-the-box synthetic-to-real (syn-to-real) generalization performance. PASTA involves per- turbing the amplitude spectrums of the synthetic images in the Fourier domain generate augmented views. We design PASTA to perturb the amplitude spectrums in structured manner such that high- more than the low-frequency ones. For the tasks semantic segmentation (GTAV object detection (SimJOK-RReal and object recognition (VisDA- C Syn Rea), across a total of 5 syn-to-real shifts, we find that PASTA either outperforms or is consistently competitive with more complex state-of-the-art methods while being complementary to other generalization approaches.
real-world data. In
components are perturbed relatively
INTRODUCTION
Performant deep models for complex tasks heavily rely access to substantial labeled data during training. However, gathering labeled real-world data can be expensive and often only captures a portion of the real-world seen at test time. Therefore, training models synthetic data to better generalize to diverse real-world data has emerged as a popular alternative. However, models trained on synthetic data have a hard time generalizing to real world data e.g., the performance of vanilla DeepLabv3+ (Chen et al., 2018a) (ResNet-50 backbone) architecture on semantic segmentation drops from 73.45% mloU on GTAV to 28.95% mloU on Cityscapes for the same set of classes. Several approaches have been considered prior work to tackle this problem.
In this paper, propose an augmentation strategy, called Proportional Amplitude Spectrum Training Augmentation (PASTA), for the synthetic-to-real generalization problem. PASTA involves perturbing the amplitude spectrums of the source synthetic images in the Fourier domain. While prior work in domain generalization has considered augmenting images in the Fourier domain (Xu et al., 2021; Yang & Soatto, 2020; Huang et al., 2021a), they mostly rely on the observations that - (1) low- frequency bands of the amplitude spectrum tend to capture style information / low-level statistics (illumination, lighting, etc.) (Yang & Soatto, 2020) and (2) the corresponding phase spectrum tends to capture high-level semantic content (Oppenheim et al., 1979; Oppenheim & Lim, 1981; Piotrowski & Campbell, 1982; Hansen Hess, 2007; Yang et al., 2020).
In addition to the observations from prior work, we make the observation that synthetic images have less diversity in the high-frequency bands their amplitude spectrums compared to real images (see Sec. 3.2 for detailed discussion). Motivated by these key observations, PASTA provides structured way to perturb the amplitude spectrums of source synthetic images to ensure that a model is exposed to more variations in high-frequency components during training. We empirically observe that by relying on such a simple set of motivating observations, PASTA leads to significant improvements in synthetic-to-real generalization performance e.g., out-of-the--on GTAV CTAV-CCityscapes generalization performance of vanilla DecpLabv3+ (ResNet-50 backbone) semantic segmentation architecture improves from 28, 18.95% mloU to 44.12% mloU.
PASTA involves the following steps. Given an input image, we apply 2D Fast Fourier Transform (FFT) to obtain the corresponding amplitude and phase spectrums in the Fourier domain. For every spatial frequency (m, n) in the amplitude spectrum, we sample multiplicative jitter value from

---

Under review as a conference paper at ICLR 2023
Image
RandAugment
PASTA
VisDA-C GTAV
Figure 1: PASTA augmentation samples. Examples of images from different synthetic datasets when augmented using PASTA and RandAugment (Cubuk et al., 2020). Row includes examples from GTAV and row 2 from VisDA-C.
oLL nl) such that o(m, n| increases monotonically with (m, n) (specifically vm² n²), thereby, ensuring that higher frequency components in the amplitude spectrum are perturbed more compared to the lower frequency components. The dependence of a a(m, n) on (m, n.) can be controlled using a set of hyper-parameter that govern the degree of monotonicity. Finally, given the perturbed amplitude and the original phase spectrums, we can apply an inverse 2D Fast Fourier Transform (iFFT) to obtain the augmented image. Fig. 1 shows a few examples of augmentation by PASTA,
In termss of Fourier domain augmentations, closest to PASTA are perhaps the approaches Amplitude Jitter (AJ) (Xu et al., 2021) and Amplitude Mixup (AM) (Xu et al., 2021). The overarching principle across these methods is to perturb only the amplitude spectrums of images (while keeping the phase spectrum unaffected) to ensure models are invariant the applied perturbations. For instance, AM, which is a type of mixup strategy (Zhang et al., 2018; 8; Verma et al., 2019), performs mixup between the amplitude spectrums of distinct intra-source images, while AJ uniformly perturbs the amplitude spectrums with a single jitter value c. Another frequency randomization technique, Frequency Space Domain Randomization (FSDR) (Huang et al., 2021a), first isolates domain variant and invariant frequency components by using SYNTHIA (Ros et al., 2016) (extra data) and ImageNet and then sets up a learning paradigm. Unlike these methods, PASTA applies fine- grained perturbations and does not involve sampling a separate mixup image or the use of any extra images. Instead, PASTA provides a simple strategy to perturb the amplitude spectrum of images in structured way that leads to strong out-of-the-box generalization. We will release our code and data upon acceptance.
In summary, we make the following contributions.
We introduce Proportional Amplitude Spectrum Training Augmentation (PASTA), simple and effective augmentation strategy for yyntheticto-real generalization. PASTA involves perturbing the amplitude spectrums of synthetic images in the Fourier domain so as to expose a model more variations in high- high-frequency components.
We show that PASTA leads considerable improvements or competitive results across three tasks - (1) Semantic Segmentation: GTAV - Cityscapes, Mapillary, BDD100K, (2) Object Detection: Sim10K - Cityscapes and (3) Object Recognition: VisDA-C Syn - Real covering total of syn-to-real shifts across multiple backbones.
We show that PASTA (1) often makes baseline model competitive with prior state-of-the-art approaches relying on either specific architectural components, extra data, or objectives, (2) is complementary said approaches and (3) is competitive with augmentation strategies like FACT (Xu et al., 2021) and RandAugment (Cubuk et al., 2020).
2 RELATED WORK Domain Generalization (DG). DG involves training models on single or multiple labeled data sources to generalize well to novel test time data sources (unseen during training). Since its incep- tion (Blanchard et al., 20 Muandet et al., 2013), several approaches have been proposed to tackle the problem of domain generalization. These include decomposing model into domain invariant and specific components and utilizing the former to make predictions (Ghifary et al., 2015; Khosla et al., 2012), learning domain specific masks for generalization (Chattopadhyay et al., 2020), using meta-learning to train a robust model by mimicking the DG problem during training (Li et al., 2018; Wang et al., 2020; Balaji et al., 2018; Chen et al., 2022; Dou et al., 2019), manipulating feature statistics to augment training data (Zhou et al., 2021; Li al., 2022: Nuriel et al., 2021). and using models crafted based on risk minimization formalisms (Arjovsky et al., 2019). Recently, properly tuned Empirical Risk Minimization (ERM) has proven to be competitive DG approach (Gulrajani & Lopez-Paz, 2020) with follow-up work adopting various optimization and regularization techniques on top of ERM (Shi et al., 2021; Cha et al., 2021).
2

---

Under review as a conference paper at ICLR 2023
GTAV (Synthetic)
GTAV BDO-103K
Chysappe Vap ltry
Venu- (ontherst
VIDA e (Reat
0.55
0.55
6.5
0.5
04
Cityscapes (Real)
0.25
02
11gh Ionds
Band
Band
(a)
Precuenccy Flancle
(b)
Trepanery Barate
Figure 2: Amplitude spectrum characteristics. (a) Sample amplitude spectrums (lowest frequency at the
center) for one channel of syathetic and real images. Note that the amplitude spectrums tend to follow specific patter- statistics of natural images have been found exhibit the property that amplitude values tend follow an inverse power law w.r.t. the frequency (Burton Moorhead, 1987; Tolhurst et al.. 1992), i.e. roughly, the amplitude at frequency f. A(f) x fr, for some yy determined empirically, (b) Variations in amplitude values across images. Synthetic images have less variance in high-frequeney components of the amplitue spectrum compared to real images.
Single Domain Generalization (SDG). Unlike DG which leverages diversity across domains for generalization, SDG considers generalizing from only one source domain. Some notable approaches for SDG involve using meta-learning (Qiao et al., 2020) by considering strongly augmented versions of source images as meta-target data (by exposing the model to increasingly distinct augmented views of the source data (Wang et al., 2021; Li et al., 2021)) and learning feature normalization schemes with auxiliary objectives (Fan et al., 2021).
Synthetic-to-Ret Generalization (syn-to-real), Approaches specific to syn-to-real generalization in prior work (most relevant to our experimental settings) mostly consider either learning specific feature normalization schemes so that predictions are invariant style characteristics (Pan al., 2018; Choi et al., 2021), collecting external data to inject style information (Kim et al., 2021; Kundu et al., 2021), learning to optimize for robustness (Chen al., 2020b), leveraging strong augmentations / domain randomization (Yue et al., 2019; Kundu et al., 2021) or using contrastive techniques to aid generalization (Chen al., 2021). To adapt to real images from synthetic data (Chen al., 2018b) trained Faster R-CNN in an adversarial manner, (Saito et al., 2019) leveraged adversarial alignment loss to emphasize globally similar images, (Chen et al., 2020a) proposed method to harmonize transfcrability and discriminability of features a hicrarchical method, and (Vibashan et al., 2021) ensures category-awan feature alignment for learning domain domain-invariant features. We consider three of the most commonly studied settings for syn-to-real generalization in this paper datasets, (2) Object Detection Siml0K-PRee and (3) Object Recognition the VisDA-C (Peng et al., 2017) dataset. To appropriately characterize the "right" synthetic data for generalization, (Mishra et al., 2021) has recently considered tailoring synthetic data for downstream generalization.
(I) Semantic Segmentation
Fourier Domain Generalization, Adaptation and Robustness. Prior work has considered aug- menting images in the frequency domain as opposed to the pixel space. These approaches rely on the empirical observation (Oppenheim et al., 1979; Oppenheim & Lim, 1981; Piotrowski & Campbell, 1982: Hansen & Hess, 2007) that the phase component of the Fourier spectrum corresponds the semantics of the image. PASTA is in line with this style of approach. Closest and perhaps most relevant to work are that of (Xu et al., 2021; Yang & Soatto, 2020) which consider perturbing the amplitude spectrums of the source synthetic images. Building on top of (Xu et al., 2021), (Yang al., 2021) adds significance mask during linear interpolation of amplitudes. (Huang et al., 2021b) decomposes images into multiple frequency components and only perturbs components that capture little semantic information. (Wang et al., 2022) uses an -deoodr architecture to obtain high and low frequency features, and augments the image by adding random noise to the phase of high frequency features and to the amplitude of low frequency features. More generally, (Yin et al., 2019) finds that perturbations the higher frequency domain increase robustness of models to high-frequency image corruptions.
METHOD
In this section, we first cover preliminaries and then describe our proposed approach.
3.1 PRELIMINARIES
Problem Setup. In this work, we study the single domain generalization (SDG) problem in the synthetic-to-real context. SDG typically involves training model on a single labeled data source
3

---

Under review as a conference paper at ICLR 2023
?D Fast Fourier Transform
FFT
FFT
Inverse Fasl. Fourier Transform
IFFT
Souroe Image (2:
Amplituce A(z)
Phose
((x)
Elementtwise Procuct
IFFT
e)rn.]] (1, mm,
Augmented Source Image
Perturbed Ampll tude
Incressing of fmm si] with increasing spatial frequency
Sample jitter values per-spelial frequency
A(x)
Figure 3: PASTA. The figure outlines the augmentation pipeline involved in PASTA. Given image, first apply 2D Fast Fourier Transform (FFT) to obtain the amplitude phase spectrums. Following this, the Amplitude spectrum perturbed as outlined in Eqns. and S. Finally, we use the perturbed amplitude spectrum and the pristine phase spectrum to recover the augmented image by applying 2D inverse FFT.
generalize well to a target distribution(s), without access target data during training. For our experiments, source data is synthetic and target data is real.
Fourier Transform. Consider single-channel image HR be expressed as,
The Fourier transform for can
H-1 W-
F(r)|m, n] = Σ
Σ
x 2/h, w| exp -2ri
W W
h=0
where 1² 1. The inverse Fourier transform, (-), that maps signals from the frequency domain to the image domain can be defined accordingly. Note that the Fourier spectrum F(x) E C If (x. -)) and (m:)())] -]) denote the real and imaginary parts of the Fourier spectrum, the corresponding amplitude (-(x))()) and phase (P()))-)) spectrums can be expressed as,
A(x)(m,n] n] (Re(F(z)( n))² + Im(F(ax)(mm nl)²
(2)
Im Im(F(x)(m, ml)
P(=)m,, arctan
Re(F(s)(m m])
Without loss of generality, we will assume for the rest of this section that the amplitude and phase spectrums are zero-centered, i.e., the low-frequency components (low m, n) have been shifted the center (lowest frequency component is at the center). The Fourier transform and its inverse can be calculated efficiently using the Fast Fourier Transform (FFT) (Nussbaumer, 1981) algorithm. For an RGB image, we can obtain the Fourier spectrum (and [+x) and P())(-)) independently for each channel. For the following section, will use single channel image running example describe our proposed approach.
3.2 PROPORTIONAL. AMPLITUDE SPECTRUM TRAINING AUGMENTATION (PASTA)
We propose create a of augmented images through perturbations the Fourier Amplitude spectrum. Following prior work (Oppenheim et al., 1979; Oppenheim & Lim, 1981; Piotrowski & Campbell, 1982; Hansen & Hess, 2007), our augmentations perturb only the amplitude spectrum and leave the phase untouched to roughly preserve the semantics of the scene. The key steps involved in generating augmented image from original image
R Hx are:
1. Translating to the Fourier domain to obtain the Fourier, amplitude and phase spectrums (see Eqns 3) F(z), P(r)
2. Perturbing "only" the amplitude spectrum via perturbation function gx(+) (-) to obtain the perturbed amplitude spectrum A(x) ga (AAx))
3. Applying an inverse Fourier transform, F-)((), to the perturbed amplitude A(xx) and the pristine phase spectrum P(x:) to obtain augmented image.
4

---

Under review as a conference paper at ICLR 2023
Alternative approaches to augment images in the Fourier domain (Yang & Soatto, 2020; Xu et al., 2021) 1) differ mostly terms of how the function ()(+) operates.
Across a set of synthetic source datasets, we make the important observation that synthetic images tend to have smaller variations in the high-frequency components of their amplitude spectrum than real images. Fig. 2 shows sample amplitude spectrums of images from different datasets (on the left) and shows the standard deviation of amplitude values for different frequency bands per-dataset (on the right).' We find that the variance in amplitude values for high frequency components is significantly higher for real as opposed to synthetic data (see Fig. 2 (b)). In Sec. A.4 of appendix, we show how this phenomenon is consistent across (1) several syn-to-real shifts and (2) fine- une-ggrined frequency band discretizations. This phenomenon is likely a consequence of how synthetic images rendered. For instance, in VisDA-C, the synthetic images are viewpoint images of 3D object models (under different lighting conditions),
it's unlikely for them to be diverse in high-frequency details. For images from GTAV, synthetic renderings can lead to contributing factors such as low texture variations - for instance, "roads" (one of the head classes in semantic segmentation) in synthetic images likely have less high-frequency variation compared to real roads. Consequentially. to generalize well to real data, we would like to ensure that our augmentation strategy exposes the model to more variations in the high-frequency components of the amplitude spectrum during training. Our intent to ensure that the learned models are invariant to variations in high-frequency components thereby avoiding overfitting to a specific syn-to-real shift.
PASTA. This is precisely where we come in. PASTA proposes perturbing the amplitude spectrums of the images in a manner that proportional the spatial frequencies, i.e., higher frequencies are perturbed more compared to lower frequencies (see Fig. 3). For PASTA, we express 9A (-) as,
98 (A(x))]m, n] =[m, ml,A(x))
(4)
k + and A {a, B3 (5)
where <(m. ni] ~ (V11, n)) and a[m,n] = m² + n2
2² 2
W2
We ensure that the perturbation applied at spatial frequency (m, n)) has a direct dependence on (m, n) (Eqn. 5). A= {a, k 33 are controllable byper-parameters. 8 ensures baseline level of jitter applied to all frequencies and a, k govern how the perturbations grow with increasing frequencies. Note that setting either - 0 or k 0 (removing the frequency dependence) results in setting where the a [m, ni] is the same across all (m, n). In Sec. A.4 of appendix, we show quantitatively how applying PASTA increases the variance metric measured in Fig. 2 (b) for synthetic images across fine-graieed frequency band discretizations.
EXPERIMENTS
We conduct our experiments across three tasks semantic segmentation, object detection and object recognition, where we train on synthetic data so as to generalize to real world data.
4.1 DATASETS
Semantic Segmentation. For segmentation, we consider GTAV (Richter et al., 2016) as our synthetic source dataset. GTAV consists of 25k driving- triving-scene images with 19 object classes. We consider Cityscapes (Cordts et al., 2016), BDD-1008 (Yu et al., 2020) and Mapillary (Neuhold et al., 2017) as our real target datasets which contain 5k, 8k, and 25k finely annotated real-world driving / street view images respectively. The 19 classes in GTAV are compatible with those of Cityscapes, BDD-I00K and Mapillary. We train all our models on the training split of the source synthetic dataset and evaluate on the validation splits of the real target datasets. We report performance in terms of mIoU (mean intersection over union).
Object Detection. For object detection, we consider Sim10K Kiohnson-Roberee et al., 2016) as our synthetic source dataset. Sim10K consists of ~ 10k images of street scenes obtained from GTAV with 59k bounding boxes for cars. We consider Cityscapes (Cordts et al., 2016) as our target
In Fig. 2(b), for every image, upon obtaining the amplitude spectrum, first take element- wise logarithm. Then, for a particular frequency band (pre-defined), we compute the standard deviation of amplitude values within that band (across all the channels). Finally, we average these standard deviations across images to report the same in the bar plots.
'When PASTA is applied, find that performance "road" increases by significant margin (see per-class gencralization results in See. A.2 of appendix).
5

---

Under review as a conference paper at ICLR 2023
Table 1: Synthetic-to-Reet generalization results for semantic segmentation and object detection. Ta- bles la, 1b and Ic summarize syn-to-real generalization results for semantic segmentation (across 3 runs) on Cityscapes (C), BDD-I00K (B) and Mapillary (M) when trained on GTAV (G). Table I summarizes syn-to-real generalization results for object detection Cityscapes (C) when trained Sim (S). indicates numbers drawn from published manuscripts. indicates trained with downsampled 1024 x 560 images due to restricted compute. h = 2 and 3 = 0.25 for PASTA (a = 3). 8 = 0.5 for PASTA (a = 0), Rows in gray font use different base architectures and / or extra data for training and have been included primarily for completeness (drawn directly from published manuscripts). Bold indicates best and underline indicates second best.
(a) SemSeg G-(C, M) (R-50, abb334)
(b) SemSeg G-{C, B, M) (R-101, Deepl ahv3++).
Method
C
B
M
Avg.
Method
C
M
Avg.
1 Baseline"
28.95
25.14
28.18
27.42
Baseline"
32.97
30.77
30.68
31.47
RandAug
31.89
38.28
34.54
34.54
Baseline"
29.36
23.61
28.14
27.04 1:386
3 PASTA (a ()
41.65
34.75
43.37
39.92.a.m
PASTA (a 3y
41.54
39.43
45.05
42.01
4 PASTA (a
44.12
40.19
47.11
43.81
2074
IBN-Net"
37.37
34,21
36.81
36.13
5 IBN-Net"
33.85
32.30
37.75
34.63
IBN-Nett
30.85
32.94
34.77
32.35 60070
6 PASTA (a
41.90
41.46
45.88
43.08
+ PASTA 3y
40.74
41.62
44.71
42.36
7 ISW"
36.58
35.20
40.33
37.37
ISW'
37.20
33.36
35.57
35.58
8 + PASTA (a
42.13
40.95
45.67
42.91
22:27
8 ISW'
31.74
31.90
34.94
32.86.10.6
9 DRPC
37.42
32.14
34.12
34.56
PASTA 3)
40.93
41.68
44.40
42.34
10 WEDGE
38.15
36.14
43.21
39.17
10 DRPC
42.53 38.72
38.05
39.77
11 ASG
31.89 N/A
N/A
N/A
11 WEDGE
43.60 41.62 48.82
44.68
12 CSG
35.27 N/A
12 ASG
32.79 N/A
N/A
N/A
13 WildNet
44.62
38.42
46.09
43.04
13 CSG
38.88 N/A
N/A
N/A
14 FSDR
44.80
41.20
43.40
43.13
15 WildNet
45.79
41.73
47.08
44.87
(c) SemSeg G-+[C, B. M) (MNv2, DeepLabv3+).
(d) ObjDet S-xC (R-50 101, (ester-RCNN)
Method
C
B
M
Avg.
Method
R-50
R-101
Baseline"
25.92
25.73
26.45
26.03
Generalization
Baseline!
24.27
23.30
23.07
23.55
0.47
Baseline"
N/A
+ PASTA (a = 3)
37.76
36.32
39.07
37.71
0.54
Baseline
39.4
43.3
IBN-Net'
30.14
27.66
27.07
28.29
PD
51.5
52.2
IBN-Net
29.13
26.29
28.34
27.92
1.76
+ RandAug
52.8
5 + PASTA (a: =
52.9
53.9
PASTA (ax 3)
35.37
36.15
38.60
36.71
PASTA (a =
56.3
55.2
ISW"
30.86
30.05
30.67
30.53
7 + PASTA (a 0)
57.0
$7.1
ISW1
30.11
27.20
27.54
28.28
1.68
8 + PD + PASTA (a = 3)
58.0
56.6
PASTA (a 3)
35.15
36.49
38.85
36.83
Adaptation
EPM
N/A
51.20
10 ILLUME
N/A
53.10
Faster-RCNN (Oracle)
N/A
70.40
dataset. Following prior work (Khindkar et al., 2022), we (1) train on the entirety of Sim10K and report performance on the validation split Cityscapes and (2) consider detecting instances of the class "car" across Sim and Cityscapes. We use mAP@ 50 (mean average precision at an IoU threshold of 0.5) to report performance.
Object Recognition. For object recognition, we consider the VisDA-2017 (Peng et al., 2017) image- classification benchmark. The source synthetic domain, consists of 3D renderings of 12 object categories from different angles and under different lighting conditions resulting in a total of 152k synthetic images. The target real domain ("real" val split in VisDA-C) consists of 55k images of the 12 classes cropped from images from the COCO dataset. We split the source domain into an 80/20 train val split (for checkpoint selection) and evaluate the entirety of the target domain specified above. For comparisons with CSG on VisDA-C, we use the same experimental configurations as (Chen et al., 2021). We use accuracy evaluation metric.

---

Under review as a conference paper at ICLR 2023
4.2 MODELS, IMPLEMENTATION DETAILS AND BASELINES
Models and Implementation Details. For our semantic segmentation experiments, we consider the D (Chen et al., 201 (0) architecture with backbones- ResNet-50 (R-50) (He et al., 2016), ResNet-101 (R-101) (He ct al., 2016) and MobileNetv2 (MN-v2) (Sandler et al., 2018) (see Tables 1a, lb and lc). Unlike Table. Ia (R-50), for Tables. lb (R-101) and Ic (MN-v2), we downsample source GTAV images to the resolution 1024 x 560 for faster training (due to limited computational resources). For our object detection experiments, we consider the Master-RCNN (Ren et al., 2015) architecture with R-50 and R-101 backbones (see Table. Id). For segmentation and detection, PASTA is applied with some con- sistent positional and photometric augmentations. For our object recog- nition experiments, we consider two sets of experiments (1) where train a baseline ResNet-50 CNN from seratch with different augmentation strategies (see Table 2) and (2) where compare our proposed approach with (Chen et al., 2021) with a ResNet-101 backbone (see Table 3). We provide more details in Sec. A. of the appendix.
Table 2: VisDA-C (ResNet-50) generalization. Vamilla ResNet-
50 CNN trained (3 runs) on the synthetic source data VisDA-C is evaluated on the real (val split) target data of VisDA-C. & = 3 and / = 0.875. I.D. and O.O.D. are regular in and out-of-domain accuracies. O.O.D. (Bal.) is class balanced accuracy on out-of- domain data. Bold indicates best and underline indicates second best.
Accuracy
Method
L.D.
O.O.D
O.O.D (Bal.)
Baseline
93,87
55.86
49.77
2+ RandAug
90.83
60.48 :0.34
56.40
3+ PASTA (a = 0)
92.13 10:41
56.29 1:27
51.61 E188
4+ PASTA (a = 4)
90.26
60.20 90:71
56.91 2
Baselines and points of comparison. In addition to prior work, we compare PASTA with (1)
RandAugment (Cubuk et al., 2020) and (2) the setting where a in PASTA to assess the extent to which monotonic increase in a (m,, r2] makes a difference. For segmentation and detection, we consider only the pixel-level / photometric transforms in the RandAugment vocabulary. For our object recognition experiments, we consider the entire RandAugment vocabulary.
5 RESULTS AND FINDINGS
5.1 PETHETICCTO-REAAE GENERALIZATION RESULTS
Our semantic segmentation results are summarized in Tables. 1a, lb and Ic, object detection results in Table. 1d and object recognition results in Tables. 2 and 3.
PASTA consistently improves performance. We observe that PASTA offers consistent improvements for all three tasks and the considered synthetic-to-real shifts.
Semantic Segmentation. From Tables. 1a, 1b and Ic, we observe that PASTA significantly improves performance of a baseline model (~16%, ~10% and ~ ~11% mloU for R-50, R-101 and MN-v2 respectively) and outperforms Rand/Augeett (Cubuk al., 2020) by a significant margin- - 9.3% for R-50 (Table. Ia rows 2,4).² Additionally, we find that PASTA makes the baseline DeepLabv3+ model improve over existing approaches that use either extra data, modeling components, or objectives by significant margin. For instance, Baseline + PASTA rows across Tables. la, lb and Ic outperforms IBN-Net, ISW, DRPC (Yue al., 2019), ASG (Chen al., 2020b) and CSG (Chen et al., 2021). For WEDGE (Kim et al., 2021), FSDR (Huang et al., 2021a) & WildNet (Lee et al., 2022) we find that Baseline PASTA outperforms only for R-50 and not for R-101. We would like to note that DRPC, ASG, CSG, WEDGE, FSDR, WildNet use either more data (the entirety of GTAV Or additional datasets) or different base architectures, making these comparisons unfair to PASTA. We would also like to note that for our experiments using R-101 and MN-v2, due to limited computational resources, we downsample GTAV images to lower resolution while training. This slightly hurts PASTA performance relative to experiments run at the original resolution (e.g- performance of Table. 1a row 4 drops from 43.81 to 1.90 when downsampling during training). In Sec. A.2 of appendix, we also show that for the SYNTHIA -> Real shift (1) PASTA provides strong improvements over the vanilla baseline (+3.91% % absolute improvement) and (2) is competitive with RandAugment.
Object Detection. For Object Detection (see Table. Id), we compare PASTA to EannAttment and PhotoMetric Distortion (PD). PD randomly applies a sequence of operations alter the contrast, brightness, hue, saturation and other photometric properties of the input image (details in Sec. A. of appendix). For both R-50 and R-101, we first note that PASTA (I) improves the performance of
In Sec. A.2 of appendix, we show how PASTA improves across several classes for the considered shifts.
7

---

Under review as a conference paper at ICLR 2023
baseline model (~ 17% and ~ 12% for R-50 and R-101 respectively) and (2) improves over PD and RandAugment for R-50 and is competitive with RandAugment for R-101. More interestingly, for R-101, we find that not only does PASTA improve performance on real data but also significantly outperforms the state-of-the-art adaptive object detection method - ILLUME (Khindkar et al., 2022), method that has access to target data at training time!
Object Recognition. For object recognition (see Tables. 2 and 3), we find that while PASTA improves over vanilla baseline, the offered improvements are competitive with that of RandAug- ment. From rows 1, 2, 4 in Table. 2. we can observe that while both RandAugment and PASTA outperform the Baseline, Rand Augment and PASTA (a 4) are competitive when we look at best syn-to-real generalization performance. In Table. 3 (rows 8, 9, 10) we can observe that CSG is competitive with versions where Augment in CSG is replaced with PASTA.
PASTA is complementary to existing gener- alization methods. In addition to ensuring that baseline model is competitive with or signif- icantly improves over existing methods, PASTA can also complement to existing generalization methods. For semantic segmentation, from Tables. 1a, lb and 1c, we find that applying PASTA to IBN-Net and ISW significantly improves performance (see IBN-Net, IBN-Net + PASTA and ISW, ISW- + PASTA set of rows) with improvements ranging from ~6-9% mloU for IBN-Net and ~5-7% mIoU for ISW across different backbones. For object recognition, in Table. 3. we apply PASTA CSG (Chen al., 2021), one of the state-of-the-art generalization methods VisDA-C. Since CSG inherently uses RandAugment, we consider both settings where PASTA is applied to CSG with and without RandAugment. In both cases, applying PASTA improves performance.
Table 3: VisDA-C (ResNet-101) generalization. We apply PASTA to CSG (Chen et al., 2021). Since CSG inherently uses RandAug, we also report results with and without the use RandAug when PASTA is applied. indicates drawn directly from published manuscripts. We report balssebalanced accuracy on the real (val split) target data of VisDA-C. k = 1 and / = 0.5 for PASTA, Results are reported across 3 runs. Bold indicates best and underline indicates second hest.
Method
Accuracy
Oracle (IN-1K)
53,30
Baseline (Syn. Training)*
49.30
Weight /2 Distance"
56.40
4 Synaptic Intelligence*
57.60
Feature 22 Distance"
57.10
ASG
61.10
CSG"
64.05
CSG
63.84 1-0.9
w/o RandAug
PASTA vs Frequency- Augmentation Strategies. Closest to our work is FACT (Xu et al., 2021), where authors employ Amplitude Mixup (AM) and Amplitude Jitter (AJ) as aug- mentations in a broader training pipeline. On our experimental settings (semantic segmenta- tion using R-50 DeepLabv3+ baseline) we find that PASTA outperforms AM and AJ 41.90%
CSG + PASTA (a 0)
64.05 +0.00
10 CSG + PASTA (a = 10)
64.29 10.56
w RandAug
11 CSG + PASTA (a = 0)
65.38 10,90
12 CSG + PASTA (a = 10)
65.86 1113
(PASTA) vs 39.70% (AM) vs 30.70% (AJ) mloU on real datasets (single run, trained at 1024 x 560). Since FACT was designed specifically for multi-source domain generalization settings (not neces- sarily in syn-to-real context), we also compare directly with FACT on PACS (a multi-source DG setup for object recognition) by replacing the augmentation pipeline in FACT with PASTA. We find that FACT-PASTE outperforms FACT- accuracy. We further note that PASTA is also competitive (within 1%) with FSDR (Huang et al., 2021a), another frequency space domain randomization method, which uses extra data in its training pipeline (see Table. lb)."
- 87.97% vs 87.10% average leave-one-out domain
5.2 ANALYZING PASTA
Sensitivity of PASTA to a, k and B. We discuss how sensitive generalization performance is the choice of o, k and (8 in PASTA and attempt to provide general guidelines and pitfalls for appropriately selecting the same. We note the following:
Same choice of a, k and 3 leads to consistent improvements across multiple tasks (object detection and semantic segmentation). We find that the same set of values, a E [0,3), k = 2, 3 E (0.25, (.5), offer significant improvements across (I) both the tasks of detection and segmentation, (2) different backbones: R-50 and R-101 and (3) different shifts: (GTAV, Siml + { [Cityscapes, BDD Mapillary).
Optimal values of a, & and 3 are different across different kinds of shifts. We find that values of hhyper-parmeeter that lead to improved performance differ across different synthetic source
Wee discuss these experiments in Sec. A.3 of appendix.
8

---

Under review as a conference paper at ICLR 2023
60
60
60
60
OGD Balanced
Accuracyy (%)
0==
45
45
45
Q==
30
30
30
30
0== 5
15
15
15
15
0==
0
0
0
0
a1 9
2
6
8
2
4
6
8
2
4
6
8
2
6
8
1:
k-
B 0.25
B
0.5 (b)
3
0.75 (c)
1.0 (d)
(a)
Figure 4: Ablating 3, k in PASTA. We train vanilla ResNet-50 on VisDA-C source by applying PASTA
and varying lt and 3 within the sets {0, 5, 7, 7, {2, 4, 6, 8) and {0.25, 0.5, 0.75, 1.0) respectively. The trained models are evaluated the target (validation) real data with class-balanced accuracy as the metric.
datasets {GTAV, Sim10K) } and VisDA-C (Tables. 1a, 1b, 1c. 1d vs 2, 3). This is not entirely surprising since VisDA-C and (GTAV, Sim10K) images differ significantly synthetic street views and VisDA-C includes different viewpoints of 3D CAD models of objects. Appropriate selection of a and is more important compared to B. In Fig. 4 we evaluate R-50 trained on synthetic VisDA-C source on the real (validation) target split for different values of o, k and 33 and report generalization performance. More than anything, we find that performance is quite sensitive to the choice of k performance increases slightly with increasing k and then drops significantly. This holds across all values of B3. Furthermore, increasing for value of l: results in significant drops. Qualitatively, we observe that for high a and we get augmented images have their semantic content significantly occluded, thereby, resulting in poor generalization.
GTAV includes
Is PASTA complementary to other augmentation strategies? We first check if PASTA is complementary to RandAugment. To assess this, we conduct experiments with vanilla R-50 CNN on VisDA-C by using both PASTA and RandAugment. We find that, terms of absolute values, PASTA + RandAugment outperforms RandAugment but not PASTA, but all performances are within 1 1% of each other. We notice that combining the best PASTA setting with the best RandAugment one leads to very strong augmentations and further note that reducing the strength of PASTA in the combination leads to improved results. On semantic segmentation, we find that a simple combination of just RGB gaussian noise and color jitter itself leads to significant improvements in generalization (leading mloU of almost 40.96%; single run, trained 1024 560) with gaussian noise likely adding more high-frequency components and color jitter likely perturbing low-frequency components. PASTA on the other hand is more fine-grained method perturb all frequencies in a structured manner, which is likely why it outperforms this combination.
To summarize, across our set of extensive experiments, and effective augmentation strategy that - (1) ensures a baseline method is competitive with existing generalization approaches, (2) often improves performance over existing generalization methods and (3) is complementary to existing generalization methods.
observe that PASTA serves as simple
Limitations. By relying the observation that synthetic images can be lacking in tigh-frequency variations, even though PASTA leads to strong generalization results, akin to any augmentation strategy, it is not without it's limitations. Firstly, it is important to note that PASTA has only been tested on the syn-to-real generalization settings explored this work and it an important step for future work to assess the extent to which improvements offered by PASTA translate to more such settings. Secondly, as observed in Sec. 5.2, PASTA quite sensitive to the choice of k. with a small subset of values of a, k and 3 defining beneficial levels of augmentations.
6 CONCLUSION
We propose Proportional Amplitude Spectrum Training Augmentation (PASTA), an augmentation strategy for synthetic-to-real generalization. PASTA is motivated by the observation that the amplitude spectra are less diverse in synthetic than real data, especially for high-frequency components. Thus, PASTA augments synthetic data by perturbing the amplitude spectra, with magnitudes increasing for higher frequencies. We show that PASTA offers strong oft generalization performance on semantic segmentation, object detection, and object classification tasks. The strong performance of PASTA holds true alone (i.e., training with ERM using PASTA augmented images) or together with alternative
algorithms. We would like to emphasize that the strength of PASTA lies in its simplicity and effectiveness, offering strong improvements despite not using extra modeling components, objectives, or extra data. We hope that future research endeavors in syn-to-real generalization take domain randomization techniques like PASTA into account.

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
While simulation offers the promise of having more diverse synthetic training data, this is predicated on the assumption that simulators will be designed to produce the diversity of data needed to represent the world. Most existing work on developing simulated visuals focus on diversity such weather, lighting, and sensor choice. In order for syn-to-real transfer to be effective for the world population we will also need to consider design and collection of synthetic data that adequately represents all sub- cut--ooulations.
8 REPRODUCIBILITY STATEMENT
For all our experiments surrounding PASTA and associated baselines, we provide details surrounding the datasets and training, validation and test splits in Sec. of the main paper and Sec. A.l of the appendix. Sec. A. also provides details surrounding the choice of hyper-parameten optimization, and model selection criteria for all the three tasks semantic segmentation, object detection, and object recognition. Sec. A.6 of the appendix provides details about the frameworks and base code repositories used for our experiments. We will release our code and data upon acceptance.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint ExXiv:197.000 00093, 2019.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-reguularization. In Advances in Neural Information Processing Systems, pp. 998-1008, 2018
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24, 2011.
Geoffrey J Burton and Ian R Moorhead. Color and spatial structure in natural scenes. Applied optics, 26(1):157-170, 1987.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems. 34, 2021.
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference Computer Vision, pp. 301-318. Springer, 2020.
Chaoqi Chen, Zebiao Zheng, Xinghao Ding, Yue Huang, and Qi Dou. Harmonizing transferability and discriminability for adapting object detectors. 2020 IEFE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8866-8875, 2020a.
Keyu Chen, Di Zhuang, and J Morris Chang. Discriminative adversarial domain generalization with meta-learning based cross-domain validation. Neurocomputing. 467:418 426, 2022.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818. 2018a.
Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Animashree Anandkumar. Automated synthetic- to-real generalization. In International Conference on Machine Learning, pp. 1746-1756. PMLR, 2020b.
Wuyang Chen. Zhiding Yu, Shalini De Mello, Sifei Liu, Jose M. Alvarez, Zhangyang Wang, and An- ima Anandkumar. Contrastive syn-to-real generalization. In International Conference on Learning Representations, 2021. URL https: openreview net forum?
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Domain adaptive faster r-cnn for object detection in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3339-3348, 2018b.
10

---

Under review as a conference paper at ICLR 2023
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selec- tive whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11580-111 1900, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via -gdes-agnostic learning of semantic features. In Advances Neural Information Processing Systems, pp. 6447-6458, 2019.
Xinjie Fan, Qifei Wang. Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou. Adversari- ally adaptive normalization for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Bruce C Hansen and Robert Hess. Structural sparseness and spatial phase alignment in natural scenes. JOSA A, (4(7):1833-1885 2007.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomiza- tion for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6891-6 aa
Jaaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Rda: Robust domain adaptation via fourier adversarial attacking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8988-8999 2021b.
Matthew Johhnson-Roeesonn Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the matrix: Can virtual worlds replace human-gennn -enerrtee annotations 2016.
for real world tasks? arXiv preprint
Vaishnavi Khindkar, Chetan Arora, Vineeth Balasubramanian, Anbumani Subramanian, Rohit Saluja, and CV Jawahar. To miss-attend is to misalign! residual self-attentive feature alignment for adapting object detectors. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3632-3642, 2022.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In European Conference on Computer Vision, pp. 158-171. 1.8-1711.priiiiii Springer, 2012.
Namyup Kim, Taeyoung Son, Cuiling Lan, Wenjun Zeng, and Suha Kwak. Wedge: Web-image assisted domain generalization for semantic segmentation. arXiv preprint 11X.:22109.1199 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
11

---

Under review as a conference paper at ICLR 2023
Jogendra Nath Kundu, Akshay Kulkami,, Amit Singh, Varun Jampani, and R Venkatesh Babu. Generalize then adapt: Source-free domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7046-7056, 2021.
Suhyeon Lee, Hongje Seong, Seongwon Lee, and Euntai Kim. Wildnet: Learning domain generalized semantic segmentation from the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9936-9946, 2022.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 224-
233, 2021.
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling-Yu Yu Duan. Uncertainty modeling for out-o--distribution generalization. In International Conference on Learning Representations, 2022.
Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: preprint arXiv.1500.04579, 2015.
Looking wider to see better. arXiv
Samarth Mishra, Rameswar Panda, Cheng Perng Phoo, Chun-Fu Chen, Leonid Karlinsky, Kate Saenko, Venkatesh Saligrama, and Rogerio Feris. Task2sim: Towards effective pre-training and transfer from synthetic data. arXiv preprint ar20v:2112.00054, 2021.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pp. 4990-4999, 2017.
Oren Nuriel, Sagie Benaim, and Lior Wolf. Permuted adain: Reducing the bias towards global statistics image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms, pp. 80-111. Springer, 1981.
A. Oppenheim, Jae Lim, G. Kopec, and S. Pohlig. Phase in speech and pictures. In ICASSP '79. LEEE International Conference on Acoustics, Speech, and Signal Processing, volume 4, pp. 632-637, 1979. doi: 10. 1TT.O//AAAA......1111000.
A.V. Oppenheim and J.S. Lim. The importance of phase in signals. Proceedings of the IEEE, 69(5): 529 541, 1981. doi: 10.1 1110//00C..981.11200
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 479, 2018
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-peeformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. 0024-8035. Curran Associates, Inc., 2019. URL htatt:::/paaee neurips pdf.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv: 2017.
12

---

Under review as a conference paper at ICLR 2023
Leon N Piotrowski and Fergus W Campbell. A demonstration of the visual importance and flexibility of spatial- freuuency amplitude and phase. Perception, 11(3):33-334, 1982.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556-12565 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pp. 102-118 18. Springer, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3234-3243, 2016.
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-weak distribution alignment for adaptive object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1956-6965, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 4520, 2018.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09995 2021.
DJ Tolhurst, Yy Tadmor, and Tang Chao. Amplitude spectra of natural images. Ophthalmic and Physiological Optics, 12(2):229-233 232, 1992.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning. 2019.
V. S. Vibashan, Vikram Gupta, Poojan Oza, Vishwanath A. Sindagi. and Vishal M. Patel. Mega-cda: Memory guided attention for category-aware unsupervised domain adaptive object detection. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4514-4524, 2021.
Bailin Wang, Mirella Lapata, and Ivan Titov. Meta-lcarning for domain generalization in semantic parsing. arXiv preprint arXiv:2010.1 11988, 2020.
Jingye Wang, Ruoyi Du, Dongliang Chang, and Zhanyu Ma. Domain generalization via frequency- based feature disentanglement and interaction. arXiv preprint arXiv:2201 0080099 2022.
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning diversify for single domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 834 843, 2021.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14383-14392, 2021.
Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4085-4095, 2020.
Yanchao Yang. Dong Lao, Ganesh Sundaramoorthi. and Stefano Soatto. Phase consistent ecological domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 901 1-9020, 2020.
13

---

Under review as a conference paper at ICLR 2023
Yijun Yang, Shujun Wang, Pheng-Ann Heng, and Lequan Yu. Hodg: A hierarchical consistency frame- work for domain generalization on medical image segmentation. arXiv preprint arXiv:2109.05722 2021.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances Neural Information Processing Systems, 32, 2019.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636- 2645, 2020.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1110-2110, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2021.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
This appendix is organized as follows. In Sec. A.I, we first expand on implementation and training details from the main paper. Then, in Sec. A.2, we provide per-class synthetic-to- to-real generalization results (see Sec. 5.1 of the main paper). Sec. A.3 includes additional discussions surrounding different aspects of PASTA. Sec. A.4 includes additional analysis of the amplitudes across multiple frequency bands and datasets. Next, Sec. A.5 contains more qualitative examples of PASTA augmentations and predictions for semantic segmentation. Finally, Sec. A.6 summarizes the licenses associated with different assets used in our experiments.
A.1 IMPLEMENTATION AND TRAINING DETAILS
In this section, we outline our training and implementation details for each of the three tasks Semantic Segmentation, Object Detection, and Object Recognition. We also summarize these details in Tables. 4a, 4b, and 4c.
Semantic Segmentation (see Table. 4a). As stated in Sec. 4.2 of the main paper. for our semantic segmentation experiments, we consider the D (Chen et al., 2018a) architecture with back- bones - ResNe-50 (R-50) (He et al., 2016), ResNet-101 (R-101) (He et al., 2016) and MobileNetv2 (MN-v2) (Sandler et al., 2018) (see Tables. 1a, lb, and Ic in the main paper). In our experiments using R-101 and MN-v2, we resize the source GTAV images to the resolution 1024 x 560 for faster training under limited computational resources. We adopt the hyper-parameter (and distributed training) settings used by (Choi et al., 2021) for training. We train ResNet-50, ResNet-101 and MobileNetv2 based models across 4, 4 and 2 GPUs respectively in a distributed manner (similar to (Choi et al., 2021)). This includes the use of SGD as an optimizer with an initial learning rate of 10-2 and a momentum of 0.9. Similar to (Choi et al., 2021), we also use a polynomial learning rate schedule (Liu al., 2015) with a power of 0.9. Our models are initialized with ImageNet (Krizhevsky et al.. 2012) pre-trained weights. We train all our models for 40k iterations with a batch size of 16 for GTAV. Our segmentation models are trained on the train split of GTAV and evaluated on the validation splits of the target datasets (Cityscapes, BDD-100K and Mapillary). For segmentation, PASTA is applied with base set of positional and photometric augmentations (PASTA first and then the base augmentations) gaussian blur, color jitter, random crop, random horizontal flip and random scaling used. We use Pytorch (Paszke al., 2019) implementations of these augmentations. For RandAugment (Cubuk et al., 2020), we only consider the vocabulary of photometric augmentations for segmentation. We conduct ablations (within computational constraints) for the best performing Rugmenttent setting using R-50 for syn-to-real generalization and find that best performance is achieved when 8 augmentations are sampled from the vocabulary for application at the highest severity level (30). Whenever we train a prior generalization approach, ISW (Choi et al., 2021) or IBN-Net (Pan et al., 2018), we follow the same set of typer-paramete configurations as (Choi et al., 2021). All models are trained across 3 random seeds.
Object Detection (see Table. 4b). For object detection, we consider the Faster-RCNN (Ren et al., 2015) architecture with ResNet-50 and ResNet- 101 backbones (see Table. 1d in the main paper). We train on the entirety of Sim minl0K OOK Rooesonn et al. (2016) (source dataset) for 10k iterations and pick the last checkpoint for Cityscapes (target dataset) evaluation. We use SGD with momentum as our optimizer with an initial learning rate of 10-2 (adjusted according to a step learning rate schedule) and batch size of 32. Our models are initialized with ImageNet (Krizhevsky al., 2012) pre-trained weights. All models are trained on 4 GPUs. For detection, we also compare PASTA against Rand (Cubuk et al., 2020) and Photometric Distortion (PD). The sequence of operations in PD to augment input images are randomized brightness, randomized contrast, RGB-*HS conversion, randomized saturation & hue changes, HSV-RGB conversion, randomized contrast, and randomized channel swap.
Object Recognition (see Table. 4c). As stated earlier, for object recognition, we consider two sets of experiments (see Tables. 2 and 3 in the main paper) - (1) where we train a baseline Re CNN from scratch with different augmentation strategies (see Table. 2, main paper) and (2) where we check if PASTA is complementary to CSG (Chen al., 2021). when trained with ResNet- 101 backbone (see Table. 3, main paper). For (1), we split the synthetic source data into a training and validation split in an 80/20 ratio for checkpoint selection. We use SGD without momentum as an optimizer with a learning rate of 10-4, weight decay of 10-5 and batch size of 64. We train these models for 10 epochs. Our models are initialized with ImageNet (Krizhevsky et al., 2012) pre-trained weights. We use Pytorch (Paszke et al., 2019) implementations of color jitter, random resized crop and random horizontal flip as the base set of augmentations. For (2), we follow the same Sypem--aramettr
15

---

Under review as a conference paper at ICLR 2023
Table 4: Implementation & Optimization Details. We summarize details surrounding dataset, training.
optimization and model selection criteria for semantic segmentation, object detection and object recognition experiments.
(a) Semantic Segmentation Training
Config
Value
Source Data
GTAV (Train Split)
Target Data
Cityscapes (Val Split)
BDD-100K (Val Split)
Mapillary (Val Split)
Segmentation Architecture
DeepLaby3+ (Chen al., 2018a)
CNN Backbones
ResNet-50 (R-50) (He al., 2016)
ResNet-101 (R-101) (He et al., 2016)
MobileNetv2 (Mn-v2) (Sandler et al., 2018)
Training Resolution
Original GTAV resolution for R-50
1024 560 for R-101 Mn-v2
Optimizer
SGD
Initial Learning Rate
10-2
Learning Rate Schedule
Polynomial IJR (exp 0.9)
Initialization
Imagenet Pre-trained Weights (Krizhevsky al., 2012)
Iterations
40000
Batch Size
16
Augmentations w/ PASTA
Gaussian Blur, Color Jitter, Random Crop Random Horizontal Flip, Random Scaling In-domain (Source) Validation performance
Model Selection Criteria
(b) Object Detection Training
Config
Value
Source Data
SimlOK
Target Data
Cityscapes (Val Split)
Segmentation Architecture
Faster-RCNi (Ren al., 2015)
CNN Backbones
ResNet-50 (R-50) (He al., 2016)
ResNet- (R-101) (He et al., 2016)
Training Resolution
Original Sim 10K resolution for R-50, R-101
Optimizer
SGD (momentum = 0.9) 10-2
Initial Learning Rate
Learning Rate Schedule
Step-LR, Warmup 500 iterations, Warmup Ratio 0.001
Steps 6000 & 8000 iterations Imagenet Pre-trained Weights (Krizhevsky et al., 2012)
Initialization
Iterations
10000
Batch Size
32
Augmentations w/ PASTA
Resize, Random Flip
Model Selection Criteria
Checkpoint at 10k iterations
(c) Object Recognition Training
Config
Value (for Table. 2 main paper)
Value (for Table. main paper)
Source Data
VisDA-C Synthetic
VisDA-C Synthetic
Target Data
VisDA-C Real
VisDA-C Real
Backbone
ResNet-50 (R-50) (He et al. 2016)
ResNet-101 (R-101) (He et al., 2016)
Optimizer
SGD w/o momentum
SGD w/ momentum (0.9)
Initial Learning Rate
10-4
10-4
Weight Decay
10-5
5 x 10-4
Initialization
Imagenet (Krizhevsky et al., 2012)
Imagenet (Krizhevsky et al., 2012)
Epochs
10
30
Batch Size
64
Augmentations w/ PASTA
Color Jitter, Random Resized Crop
RandAugment (Cubuk et al., 2020)
Random Horizontal Flip
Model Selection Criteria
Best In-domain Val Performance
Best In-domain Val Performance
16

---

Under review as a conference paper at ICLR 2023
Table 5: GTAS per-class generalization results. Per-class IoU comparisons for syn-to-real generalization results when semantic segmentation models trained on GTAV are evaluated on Cityscapes. Base model involved is DeepLabv3+ (R-50 backbone). Results are reported across 3 runs. indicates drawn directly from published manuscripts. l = 2 and 3 = 0.25 for PASTA (a = 3). B = 0.5 for PASTA (a = 0). Class headers are in decreasing order of pixel frequency.
veretation
motorcycle
ruad
building
car
sidewalk
sky
pole
person
terrain
Fance
wall
bicycle
sign
bus
tnuck
rider
light
train
Method
mloU
1 Bascline
45.1 56.8 80.9 61.0
23.1
38.9
23.9
58.2
24.3
16.3
16,6
13.4
7.3
20.0
17.4
1.2
30,0
7.2
8,5
29.0
2 +RandAug
58.5 56.3
77.3
83.7
30.3
45.1
27
57.8
20.6
20.9
11.4
16.9
9.7
20.3
15.0
2.4
28.1
14.0
10.3
31.9
+PASTA (o= 0)
76.3
85.5
82.3
35.2
72.9
65.2
25.7
25.4
19.8
23.2
2.8
33.8
21.0
24.0
41.7
+PASTA 84.1
80,5 85.8 85.9
40.1
81.8
31.9
66.0
31.4
28.1
29.0
21.8
28.5
24.5
28.7
7.0
32.9
23.4
27.2
44.1
IBN-Net
51.3 59.7 85.0
76.7
24.1
67.8
23.0
60.6
40.6
25,9
14.1
15.7
10.1
23.7
16.3
0.8
30.9
4.9
11.9
33.9
6 4PASTA (a 3) 78.1 79.5 85.8 84.5
31.7
80.1
32.2 63.4
38.8
21.7
28.0
18.2
22.6
26.4
29.0
2.8
34.0
16.5
22,9
41.9
7 ISW
60.5 65.4
85.4 82.7
25.5
77.3 25.8 61.9 38.5 23.7
21.6
15.5
12.2
25.4
21.1
0.0
33.3
9.3
16.8
36.6
+PASTA (a 10=33 76.6 78.4 85.6 83.7 32.5 83.1
33.1 63.4 40.4 23.6
27.3
17.4
22.3
25.7
30.1
3.3
35.9
18.2
19.9
42.1
configurations adopted by (Chen et al., 2021). This includes the use of an SGD (with momentum 0.9)
optimizer with a learning rate of 10-4, weight decay of 5 x 10-4 and a batch size of 32. These models are trained for 30 epochs. CSG (Chen et al., 2021) also uses RandAugment (Cubuk al., 2020) as an augmentation. For both (1) and (2), models are trained on single GPUs across 3 random seeds.
A.2 SYNTHETIC-TO-RELT GENERALIZATION RESULTS
Overall GTAV Generalization Results. We first note that using RandAugment (Cubuk et al., 2020) for the Baseline DeepLabv3+ model with ResNet-101 and MobileNetv2 backbones leads to overall syn-to-real generalization performances of +0.44 (PASTA performance being 02201=0.26) and 11.4771.22 (PASTA performance being 37. 37.71+0.54) respectively. Overall, across MNv2, R-50 and R-101, we observe that the gap between RandAugment and PASTA reduces as the number of parameters in the backbone increase. In Tables. 1a, 1b and Ic of the main paper, present overall synthetic-to- ro-rell generalization results when models trained on GTAV are evaluated on Cityscapes, BDD-1008 and Mapillary. As stated in Sec. 5.1 of the main paper, for ResNet-50 and ResNet- (in Tables. Ia and 1b of the main paper), including comparisons with DRPC (Yue et al., 2019), ASG (Chen et al., 2020b), CSG (Chen et al., 2021), WEDGE (Kim et al., 2021), FSDR (Huang et al., 2021a) and WildNet (Lee et al., 2022) is not entirely fair to PASTA since these approaches use either more data or different base architectures for training. For instance, WEDGE (Kim et al., 2021) and CSG (Chen et al., 2021) use Lee2Labv2, ASG (Chen et al., 2020b) uses FCNs, DRPC (Yue et al, uses the entirety of GTAV (not just the training split) and WEDGE uses ~5k extra Flickr images in it's overall pipeline. FSDR (Huang et al., 2021a) another such approach that, when trained with a ResNet-101 backbone, achieves overall mloU of 43.13% (within 1% of Baseline/ / IBN-Net ISW + PASTA) on real datasets. However, FSDR also uses FCNs and the the entirety of GTAV for training. FSDR (Huang et al., 2021a) and WildNet (Lee et al., 2022) also use extra ImageNet (Krizhevsky et al., 2012) images for stylization / randomization. For FSDR, the first step in the pipeline also requires access to SYNTHIA (Ros et al., 2016).
Per-class GTAV Generalization Results. Tables 5, 6 and 7 include per-class synthetic-to-rea generalization results when a DeepLabv3+ (R-50 backbone) model trained on GTAV is evaluated on Cityscapes, BDD- and Mapillary respectively. We briefly discuss in the main paper how PASTA (a 2 0) improves performance across several classes for GTAV Real. For GTAV - Cityscapes (see Table. 5), we find that Baseline- PASTA consistently improves over Baseline and Rundnngeent. For IBN-Net and ISW in this setting, we observe consistent improvements (except for the classes terrain and fence). For -BDV outperforms RandAugment on the majority of classes, both are fairly competitive and outperform the vanilla Baseline approach. For IBN-Net and ISW, PASTA (a = 3) almost always outperforms the vanilla approaches (except for the class wall). For GTAV -aailllry (see Table. 7), for Baseline, we find that PASTA (a 2 0) outperforms the vanilla approach and RandAugment. For IBN-Net and ISW, PASTA (a 3) outperforms the vanilla approaches with the exception of the classes train and fence. Overall SYNTHIA Real Generalization Results. We conducted additonal experiments using SYNTHIA (Ros et al., 2016) as the source domain and Cityscapes, BDD- 100- and Mapillary as
(see Table. 6). we find that for the Baseline, while PASTA
17

---

Under review as a conference paper at ICLR 2023
Table 6: GTA5-EDD-I00K per-class generalization results. Per-class IoU comparisons for syn-to-real generalization results when semantic segmentation models trained on GTAV are evaluated on BDD-100K, Base model involved is DeepLabv3+ (R-50 backbone). Results are reported across 3 runs. indicates drawn directly from published manuscripts. l = 2 and 3 = 0.25 for PASTA (a = 3). B = 0.5 for PASTA (a = 0). Class headers are in decreasing order of pixel frequency.
vegetation car
motorcycl
ruad
sky
building
sidewalk
fence
Kerain
tnuck
pole
bus
wall
sign
person
light
bicycle
rider
train
Method
mloU
1 Bascline
48.2 32.3 34.3 58.2
67.3
23.0
19.8
21,4
11.3
28.4
5.6
3.4
18.1
43.9
30
11.1
16.0
5.1
0.0
25.1
2 +RandAug
75.4 82.8
67.7
74.7
74.1
39.3
32.7
26.6
22.7
37.0
16.5
5.0
23,9
51.7
35.8
12.0
29.3
20.2
0.0
38.3
+PASTA (o= 0)
65.9
50.6
78.5
28
34.1
6.4
25,4
52
35.8
10.4
38.5
11.5
34.8
+PASTA
86.0 86.6 74.8
72.7
82.8
38.4
31.2
24.9
23.7
34.8
6.4
11.2
26.2
55.1
37.0
13.3
38.3
19.9
0.0
40.2
IBN-Net
68.9 66.9
56.7
66.6
70.3
28.8
214
22.1
12.8
31.9
7.2
6.0
21.7
50.2
35.0
18.1
23.2
5.8
0.0
32.3
6 4PASTA (a: 3) 86.1 87.6 74,9 72.3 82.3
36.6
30.6
26.2
25.3
37.1
10.8
13.2
25,5
56.0
36.8
21.4
38,9
26.0
0.0
41.5
7 ISW
74,9 77.4 65.2 69.0 72.4
30.4
22.6
26.2
16.2
34.9
6.1
11.5
22.2
50.3
36.9
11.4
31.3
10.0
0.0
35.2
+PASTA (a 86.5 87.9 74.0 73.0 83.2 37.7
28.6
28.1
23,4
37.2
7.8
11.3
25.0
55.1
37.8
23,6
35.5
22.4
0.0
41.0
Table 7: GTAS-Mapillary per-class generalization results. Per-class 1oU comparisons for syn-to-real generalization results when semantic segmentation models trained on GTAV are evaluated on Mapillary. Base model involved is DeepLabv3+ (R-50 backbone). Results reported across runs. indicates drawn directly from published manuscripts. headers are decreasing order pixel frequency.
2 and 0.25 for PASTA (a 3). B 0.5 for PASTA (a 0). Class
sky
road
vegetation
building
sidewalk
car
fonce
pole
tamain
wall
sign
trick
person
bus
light
blcycle
rider
moturcycle
train
Method
mloU
Baseline
42.2 64.9
33.5
24.9
72.3
14,4
27.7
23.8
6.7
23.7
53.7
7.0
35.8
18.4
4.9
10.8
28.2
+RandAug
51.7 59.6 75.5
39.5
33,9
81.3
22.6
37.2
24.6
4.2
32
31.2
56.5
13.4
36,2
18.0
11.9
21.4
5.0
34.5
3 +PASTA (a 0)
88.4 76.5
73.7
65.5
36
82.
22.7
38.9
32.9
14.5
49.9
35.8
60.0
17.0
42.6
27.3
16.8
31.
11.2
43.4
4PASTA (a 3)
93.3 83.0 76.3
76.9
40.2
83.3
27.1
40.9
37.1
19.2
50.4
3
63.3
19.5
29.4
25.2
38
15.2
47.1
IBN-Net
82.0 66.4 73.5
57.1
32.9
73.1
24.9
31.5
28.4
10.5
38.9
30.7
56.4
16.0
38.0
18.6
9.1
16.6
12.6
37.7
+PASTA (a 3) 94.4 81.7 76.1 76.9
40.4
80.8
27.1
40.3
38.7
19.0
43.2
38.0
62.9
20.5
39.3
25.7
23,6
31.6
12.4
45.9
7 ISW
88.2 74.8 74.3 66.1 36.2
78.7
26.0
35.4
30.2
15.2
36.6
33,3
58.6
14.4
37.9
17.8
11.1
20.4
11.0
40.3
+PASTA (a 3) 94.6 82.9 76.7 76.0
41.9
81.8
25.8
40.4
40.9
18.8
43.1
34.1
61.6
19.9
40.2
24.5
22.3
30.5
11.6
45.7
the target domains with the same set of hyper-parameters for PASTA (a 3, 0.25). For a baseline deepLLabv3+ model (R-101), we find that PASTA - (1) provides strong improvements over the vanilla baseline (31.77% mloU, +3.91 % absolute improvement) and (2) is competitive with RandAugment (32.30% mloU). More generally, we find that syn-to-real generalization performance is worse when SYNTHIA is used as the source domain opposed to GTAV for instance, ISW (Choi et al., 2021) achieves an average mloU of 31.07% (SYNTHIA) as opposed to 35.58% (GTAV). SYNTHIA has significantly fewer images compared to GTAV (9.4k vs 25k), which likely contributes to relatively worse generalization performance.
PASTA and Base Augmentations. As stated in Sec. A. PASTA is applied with some consistent color and positional augmentations. To understand if PASTA alone leads to any improvements, we trained a baseline DeepLabv3+ model (R-50) without these augmentations. We find that from the settings reported in Table. la, Row of the main paper plus training at a resolution of 1024 x 560, average performance on real datasets drops from (1) 41.90% to 40.37% mloU when the photometric augmentations (color jitter and gaussian blur) are removed and (2) further drops 40.25% mloU when both positional and photometric augmentations are removed. Therefore, applying PASTA without any base augmentations leads to performance that is still significantly above vanilla baseline (40.25% vs 26.99% mloU).
A.3 PASTA ANALYSIS
In this section, we provide more discussions surrounding different aspects of PASTA. Functional form of PASTA. We now discuss the specific functional form of the perturbations introduced by PASTA per Eqns. 4 and 5 in the main paper. Eqn. 5 is constructed to ensure we have
18

---

Under review as a conference paper at ICLR 2023
multiplicatiiee jitter interaction. Unlike the multiplicative interaction, an additive jitter interaction in the Fourier space where the perturbation is sampled from a gaussian distribution can be expressed as gaussian jitter perturbation in the RGB space easier to attain by adding gaussian noise to the images. For Eqn. 5, we first considered uniform perturbations (a- = 0) independent of the spatial frequencies. As a natural extension based on the observations made in Section 3.2 of the main paper, we considered a linear dependence on spatial frequencies hence, the inclusion of a. The inclusion of l: decides how much attention we pay to the high-frequency bands compared to the low frequency ones. For fixed a, since the frequency dependent term being exponentiated in Eqn. 5 is normalized m22n22 (2v/ H=+W2
[0, 1D,, increasing perturbs the lower frequency components less.
PASTA and Amplitude Mixup (AM). We also compare PASTA with the Amplitude Mixup (AM) technique proposed in (Xu al., 2021). Amplitude Mixup involves perturbing the amplitude spectrum of the image of concern by performing a convex combination with the amplitude spectrum of another " "mixup" image drawn from the same source data. For DeepLabv3+ architecture with a R-50 backbone, we find that when trained on GTAV (single run trained at a resolution of 1024 x 560), AM achieves an mIoU of 99% (vs 41.90% for PASTA) on the real datasets. We further observe that for AM. performance isn't 't altered even if we reduce the set of mixup images to a very small set (10 randomly selected images from GTAV) instead of sampling one for every source image. We observe that this variant of AM also ends up achieving an mloU of 39.77%. 70% (single run). Overall, we observe that while AM tends to underperform compared to PASTA, the pipeline involved in AM - sampling "mixup" images - ends up being an overkill.
PASTA and Amplitude Jitter (AJ). We also compare PASTA with the Amplitude Jitter (AJ) technique considered in (Xu al., 2021). Amplitude Jitter (AJ) perturbs the amplitude spectrum with a single jitter value for every spatial frequency and channel. is a multiplicative factor sampled from a gaussian distribution centered at We use AJ with e ~ (L1,, 0.5) for our experiments with DeepLabv3+ 3+ (R-50). We find that AJ ends up achieving an mloU of 30. 70% (single run trained at a resolution of 1024 x 560) on the real datasets on average, which is significantly lower than PASTA, Qualitatively, we observe that multiplying the entire amplitude spectrum by a constant value results in uniform change of brightness in the augmented image. If we sample separately per-channel, we find that the performance of this AJ variant improves to an mIoU of 32.83% (single run trained at a resolution of 1024 x 560). Here, observe that multiplying the amplitude spectrum channel- wise by three distinct constants leads to one of image channels dominating over the others, resulting in red-ish, green-ish
blue-ish hue in the augmented images. Finally, if we consider PASTA (a = 0, 3 = 0.5), which is a more fine fine-graing grained setting where e is sampled per-spatial frequency and per-channel, we find that generalization performance improves significantly to 39.19% mloU, Therefore, we observe that approaches along the lines of AJ for generalization lead to better results implemented in a fine-grained manner as PASTA.
Constrained PASTA (a 0). Note that sampling an c per spatial frequency creates a jitter image per-channel, which is then multiplied in an elementwise manner to the original amplitude to perturb the same. Similar to the Amplitude Mixup (AM) set of experiments, we also consider a setting where we sample these jitter images from a fixed set, instead of sampling a new one for every source image, We create these jitter image sets (each jitter image randomly sampled) of sizes 1, 2, 5 and 10 and observe that for DeepLabv3+ (R-50) (single runs trained at a resolution of 1024 x 560) we achieve mloUs of 39.81 %, 40.33%, 38. 78%, and 39.16% respectively on the real datasets. Therefore, even we severely restrict the source of randomness variations in e for PASTA (a = 0), we don't observe significant loss in generalization performance.
Comparison with Fourier Domain Adaptation (FDA) (Yang & Soatto, 2020). FDA is a recent approach for syn-to-real domain adaptation and naturally requires access to unlabeled target data. In FDA, low frequency bands of the amplitude spectra of source images are replaced with those of target- essentially mimicking cheap style transfer operation. Since we do not assume access to target data in our experimental settings, a direct comparison is not possible. Instead, we consider a proxy task where we intend to generalize to real datasets (Cityscapes, 100K, Mapillary) by assuming additional access to real world street view images under different weather conditions (for style transfer) sunny day, rainy day, cloudy day, etc. in addition to synthetic images from GTAV. We find that FDA in this setting achieves an mloU of 33.0 04% (vs 26.99% for Baseline and 41.90% for Baseline + PASTA (a = 3, k = 2)) for DeepLabv3+ with an R-50 backbone trained at resolution of 1024 x 560.
19

---

Under review as a conference paper at ICLR 2023
Algorithm Pseudocode of PASTA in a PyTorch-like style.
imago: Cxcx input image for PASTA, pixel velues to e, by k: sipha, beta, and & PASIA hyperpacameters
fft = fft (imago, dim- [1, 2]) get 2D fft for each channel aba(ttt) get amplitude ot image phase argleifft) get phasa of image
C, shage $ get channel, height, and width of image
m zanget H/2,, E/21 range for m. a - =angel-W/2, 0/21 range for n
Hack grid dist srid = sqrt (NV++2 + MM++2) # RxN grid where each entry is diatance from center pixel horm_distt_riae dist_grid / sqrt (Rxx2 Wx*2) normalize distance grid by diagonal longth
KN meshgrid(n,m)
values and
grld standard deviations sigma tile IC, 1, 11) CXUXи grid of standard deviations
sigma 112 x norm ntrm_dist dim_dii__grddd grrdd) xx
Cxlix array where an entry [m sampled N(1, sigma [m, epsilon normal (mean-1, std-sigma)
pasta_amp amp epsilon PASTA perturbed amplitude
seconstruc: fft using perturbed amplitude and original phase [xp phase]
paste_ fft = pasta
pasta_image ifft (pasta_fft, dim-(1, 21) take 2D inverse Eft for each channel pasta_ mage real (pasta_ image) knep real va luea from inverse
basta_image = c pastt__mmags, 6, 1) + constrain pixel values from 0 to return pasta_imaçe
Table 8: Ablating & for GTAV as synthetic source.
Overall sensitivity to a, k. Careful selection of a; k is more important compared to 3. Sim- ilar to our observations on VisDA-C (Section 5.2 main paper), we find that restricting the k E [1,4] offers stable improvements in semantic segmentation GTAV->Real for a DeepLabv3+ model (RN-50) backbone trained a resolution of 1024x560 (see Table. 8). For & [1,4], 4], to ob- tain stable improvements for vanilla baseline (not designed specifically for syn-to-real gen- eralization), restricting (0, 4] offers stable generalization improvements. Exceeding these ranges leads to severe augmentations that obfus- cate task-relevant semantic information.
apply PASTA with varying values to DecpLabbv3e (RN-50) semantic segmentation trained at a source in- put resolution of 1024 x 560 for the GTAV-Rcal (Cityscapes, BDD-100K, Mapillary) task.
Method
mIoU (Real)
Baseline
26.99
Ablating &s for (a 3)
2 PASTA (l db=1)
41.50
3 PASTA (k
41.90
4 PASTA (k = 4)
41.09
5 PASTA (k=6)
30.90
A.4 AMPLITUDE ANALYSIS
PASTA is motivated by the empirical observation that synthetic images have less variance in their high frequency components compared to real images. In this section, we first show how this observation is widespread across set of syn-to-real shifts over fine-grained frequency band discretizations and then demonstrate how PASTA helps counter this discrepancy.
Fine-grained Band Discretization. For Fig. (b) in the main paper, the low, mid and high frequency bands are chosen such that the first band is 1/3 the height of the image (includes all spatial frequencies till 1/3rd of the image height), second band is up to 2/3 the height of the image excluding band I frequencies, and the third band considers all the remaining frequencies. We begin by splitting the amplitude spectrum into 3, 5, 7, and 9 frequency bands in the manner described above, and analyze the diversity of these frequency bands across multiple datasets. Across domain shifts (see Fig. and 6) (GTAV, SYNTHIA) - [Cityscapes, BDD-100K, Mapillary). and VisDA-C Syn-+Renr Real, we find that (1) for every dataset (whether synthetic or real), diversity decreases as we head towards higher frequency bands and (2) synthetic images exhibit less diversity in high-frequency bands at all the levels of granularity we consider.
Increase in amplitude variations post-PASTA. Next, we observe how PASTA effects the diversity of the amplitude spectrums on GTAV and VisDA-C. Similar to above, split the amplitude spectrum into 3, 5, 7, and 9 frequency bands, and we analyze the diversity of these frequency bands before and after applying PASTA to images (see Fig. 7 and 8). For synthetic images from GTAV, when
20

---

Under review as a conference paper at ICLR 2023
61AV
SYNIHA
G Olysoopes
B00-TOOK
Mapitary
61W
SYN
Chyccapee
B00-TOOK
Macii bary
0.53
0.53
Mest c(os.A020)
0.433
0.488
03117
0.11
0.9
0.9
Band
Dambo
bando
bards
Dand
Meat elboc.b22l)
alboe A/ai|
0.402
0.492
0.43
0.77
0.278
0317
0208
0258
02
09
Band1
Bah52
3und5
Sandd
Stard
Stand2
Sunda
Hrequency Bando
Precuericy Bands
Figure Variations in amplitude values across fine-grained frequency bands (GTAV Real and SYNTHIA- - Reel). Across domain shifts GTAV RReal and SYNTHIA- RReal, and four settings corresponding to fine-grained frequency bands (3, 5, 7 and bands; increasing in frequency from Band-1 to Band-n), we find that synthetic images have less variance in high-frequency components of the amplitude spectrum compared real images.
ViDA C (Symheric)
V&DAA C (faal)
V VDDA (Sendieni))
VaDa C (feaa)
Maan Alee
0.492
0.492
0:438
0.433
0.37
0317
022
02
Bonds
Rond
A4
Bands
Rund
Dand-d
0.56
0.56
Meen Alst
0.492
0.400
0.376
0.375
0950
0998
Banda
Binda
Bond
Band
Beed
Band
Dend2
Dend-
Bondee
Bondi
Frequency Bands
Frequencyy Bandi
Figure 6: Variations in amplitude values across fine-grained frequency bands (VisDA-C Synthetic++Real For the VisDA-C Sonthettttthhtttteen to fine-erained frequency bands (3, 5, 7 and bands; increasing in frequency from Band-1 to Band-n), find that synthetic images have less variance in high-frequency components of the amplitude spectrum compared to real images.
domain shift, and four settings corresponding
PASTA (a = 3, 2) is applied, observe that the standard deviation of amplitude spectrums increases from 0.4 to 0.497, 0.33 to 0.51 and 0.3 to 0.52 for the low, mid and high frequency bands respectively. As expected, observe maximum increase for the high-frequency bands.
21

---

Under review as a conference paper at ICLR 2023
GTAV
o-33
Q14
GTAY
Q-3
Moen e(log.A22))
Meen Milg..laa.Aaa Azal]
0.575
0.578
0.455
0.425
0.275
0.27
Sand
berd-2
Dendh33
Derd-1
berd-3
Band-44
Derdi
0.06
0.05
Meat c(log,Axx))
0.675
ollog_Azoi
0.5
0.0
0.35
0275
0.78
Bands
Band5
Band
banda
bandh2
bends
Bands
Figure 7: Variations in amplitude values across fine-grained frequency bands for synthetic images POSt-PASTA (GTAV). For GTAV, we find that applying PASTA increases variations in amplitude values across different frequency bands. Four plots correspond to fine-gr fine-grained frequency bands 5, 7 and 9 bands; increasing in frequency from Band-1 to Band-n). We find the maximum amount of increase for the highest frequency bands across different granularity levels.
VSDM-C (Syttheic)
c:4
ViSDW-C (9yntheicc)
Moan aljoe, Alx]]
Mean elboe_Atar)
0.575
0.575
0.425
0.425
0356
035
02
0.2
Band1
Rund
Sand1
Rund
Dond5
0.66
0.66
Meen eljor Aell
0.778
0.276
Bandl
Bendl
Band5
Baods
Bend7
Wondo
Bondo
Fregy Ranvil
Frequencc Rendi
Figure 8: Variations in amplitude values across fine-grained frequency bands for synthetic images POSt-PASTA (VisDA-C). For VisDA-C (Synthetic). we find that applying PASTA increases variations in amplitude values across different frequency bands. Four plots correspond to fine-grained frequency bands (3, 5, and bands; increasing in frequency from Band-1 to Band-n). We find the maximum amount of increase for the highest frequency bands across different granularity levels.
22

---

Under review as a conference paper at ICLR 2023
Image
RandAugment
PasTA
SIM
VisDA-C
Figure 9: PASTA augmentation samples. Examples of images from different synthetic datasets when augmented using PASTA and RandAugment (Cubuk et al., 2020). Rows 1-3 include examples from GTAV and rows 4-6 from VisDA-C.
A.5 QUALITATIVE EXAMPLES
PASTA Augmentation Samples. Fig. 9 includes more examples of images from synthetic datasets (from GTAV and VisDA-C), when RandAugment and PASTA are applied.
Semantic Segmentation Predictions. We include qualitative examples of semantic segmentation predictions on Cityscapes made by Baseline, IBN-Net and ISW (DeepLLabv3+, ResNet-50) trained on GTAV (no downsampling. corresponding to Table. 1a in the main paper) Fig. 10, 12 and 14 respectively when different augmentations are applied (RandAugment and PASTA). The Cityscapes images we show predictions on were selected randomly. We include Rand predictions only for the Bascline. To get a better sense of the kind of mistakes made by different approaches, we also include the difference between the predictions and ground truth segmentation masks in Fig. 11, 13 and 15 (ordered accordingly for easy reference). The difference images show the predicted classes only for pixels where the prediction differs from the ground truth.
A.6 ASSETS LICENSES
The assets used in this work can be grouped into three categorics Datasets, Code Repositories and Dependencies. We include the licenses of each of these assets below.
Datasets. We used the following publicly available datasets in this work GTAV (Richter et al., 2016), Cityscapes (Cordts et al., 2016), #DDD-00K (Yu et al., 2020), Mapillary (Neuhold et al., 2017), and VisDA-C (Peng et al., 2017). For GTAV, the codebase used to extract data from the original GTAV game is distributed under the MIT license." The license agreement for the Cityscapes dataset dictates that the dataset made freely available academic and non-acane academic entities for non-commercial purposes such as academic research, teaching, scientific publications, or personal
23

---

Under review as a conference paper at ICLR 2023
Baseline + Randaug
Baseline
Baseline 0) + PASTA(a 3)
Image
Ground Truth
Baseline
PASTA(a
road
build
wall
fence
pole
tlight
tsign
veg
sky
person
rider
mcycle bicycle
unlabel
Figure 10: -rtAV--Ctysccpps Baseline segmentation predictions. Qualitative predictions made on randomly selected Cityscapes validation images by Baseline DeepLabv3+ model (R-50 backbone) trained on GTAV synthetic images. The first two columns indicate the original image and the associated ground truth and rest indicate the considered approaches.
24

---

Under review as a conference paper at ICLR 2023
Baseline + Randaug
Baseline
Baseline 0) + PASTA(a 3)
Image
Ground Truth
Baseline
PASTA(a
road
swalk
build
wall
fence
pole
tlight
tsign
veg
terrain
sky person
rider
truck
mcycle bicycle
unlabel
Figure 11: GTAV -itys--psscapps Baseline segmentation prediction diffs. Differences between prediction and ground truth for predictions made randomly selected Cityscapes validation images by a Baseline DeepLabv3+ model (R-50 backbone) trained on GTAV synthetic images. The first two columns indicate the original image and the associated ground truth and rest indicate the considered approaches.
25

---

Under review as a conference paper at ICLR 2023
IBN-Net PASTA(a:
Image
Ground Truth
IBN-Net
road
swalk
build
wall
fence
pole
tlight
tsign
veg
terrain
sky person
rider
car
truck
bus
train
mcycle
bicycle
unlabel
Figure 12: GTAV->Cityscapes IBN-Net (Pan et al., 2018) segmentation predictions. Qualitative predictions made on randomly selected Cityscapes validation images by IBN-Net (DeepLabv3+ model with R-50 backbone) trained on GTAV synthetic images. The first two columns indicate the original image and the associated ground truth and rest indicate the considered approaches.
26

---

Under review as a conference paper at ICLR 2023
IBN-Net
Image
Ground Truth
IBN-Net
PASTA(o 3)
road
swalk
build
wall
fence
pole
tlight
tsign
veg
terrain
sky person
rider
car
truck
bus
train
meycle
bicycle
unlabel
Figure 13: +TAV-=Cityscappe IBN-Net (Pan et al., 2018) segmentation prediction diffs. Differ- ences between prediction and ground truth for predictions made on randomly selected Cityscapes validation images by IBN-Net (DeepLabv3+ model with R-50 backbone) trained GTAV synthetic images. The first two columns indicate the original image and the associated ground truth and rest indicate the considered approaches.
27

---

Under review as a conference paper at ICLR 2023
ISW PASTA(c:
Image
Ground Truth
ISW
road
swalk
build
wall
fence
pole
tlight
tsign
veg
terrain
sky
person
rider
car
truck
bus
train
mcycle
bicycle
unlabel
ISW (Choi et al., 2021) segmentation predictions. Qualitative predictions made on randomly selected Cityscapes validation images by ISW (DeepLabv3+ model with R-50 backbone) trained on GTAV synthetic images. The first two columns indicate the original image and the associated ground truth and rest indicate the considered approaches.
Figure 14: GTAV
28

---

Under review as a conference paper at ICLR 2023
ISW PASTA(a 3)
Image
Ground Truth
ISW
road
swalk
build
wall
fence
pole
tlight
tsign
veg
terrain
sky
person
rider
car
truck
bus
train
meycle
bicycle
unlabel
Figure 15: GTAV CityspCey ISW (Choi et al., 2021) segmentation prediction diffs. Differences between prediction and ground truth for predictions made on randomly selected Cityscapes validation images by ISW (DeepLabv3+ model with R-50 backbone) trained GTAV synthetic images. The first two columns indicate the original image and the associated ground truth and rest indicate the considered approaches.
29

---

Under review as a conference paper at ICLR 2023
experimentation and that permission to use the data is granted under certain conditions." 6 distributed under the BSD-3-Clause license." Mapillary images are shared under a CC-BY-SA license, which in short means that anyone can look at and distribute the images, and even modify on github does not have
them a bit, as long as they give attribution." The VisDA-C development license associated with it, but it does include Terms of Use, which primarily states that the dataset must be used for non-commercial and educational purposes only."
Code Repositories. For our experiments, apart from code that we wrote ourselves, we build on top of three existing public repositories RobustNet MMDetection' I and CSG RobustNet is distributed under the BSD-3-Claus license. MMDetection is distributed under Apache License 2.0 13 CSG, released by NVIDIA, is released under NVIDIA-specifie license."
Dependencies. We use Pytorch Paszke et al. (2019) as the deep-learning framework for all our experiments. Pytorch, released by Facebook, is distributed under Facebook-specifie license." 15
9 httpss//ggthbb.ooo///oooomaamaie
10 httpp:///gihuub.oommcccco///ooo
12 Phtpp://gituub.cmm//Wnaa Waass//////aaaiiiiaaaaaS
http://siiiicmmmoiiiii/CCccaoaan 15
30


Reviews:
[{"review_id": "k1nG1BXlay", "reviewer": "Reviewer_nJc4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with the problem of synthetic-to-real domain generalization and proposes a method for augmenting synthetic images. The proposed method first derives power and phase components from a given source image and adds random jitters onto only power components for generating a pseudo-real image. A variance of random jitters increases as a spatial frequency increases, which indicates that high-frequency components are perturbed more than low-frequency components. Although the proposed method is simple, it outperforms baselines and several possible competitors in synthetic-to-real domain generalization.", "summary_of_the_review": "I have a positive feeling about this paper since the proposed method is simple and effective and the documentation is clear. Also, as far as I  know, the proposed method for synthetic-to-real domain generalization is novel.\n\nOther minor comments:\n\n- The following paper can be regarded as an example of random amplitude perturbations. I do not think that this paper is critical for justifying the novelty of the proposed method.\n\"Adversarial Bone Length Attack on Action Recognition\" (AAAI2022) https://ojs.aaai.org/index.php/AAAI/article/view/20132\n\n- I am interested in the use of generative models for the same purpose.", "strength_and_weaknesses": "[Strength]\n\nS1. The problem dealt with in this paper is significant. Neural network models are getting larger and larger, and their training would require a huge amount of training data. Employing synthetic images as training examples is one of the promising solutions for this situation. Technically solid methods for filling the gap between synthetic and real images will draw attention from a broad range of researchers and engineers.\n\nS2. The proposed method is simple but effective, which will be good news for practitioners interested in large-scale model training with a few training examples. Also, the proposed method works well for many applications, such as object recognition, object detection, and semantic segmentation.\n\nS3. The documentation is clear and the proposed method is well justified.\n\n[Weakness]\n\nW1. I am not sure whether the variance given in Equation (5) is the best for this purpose. I understand that the proposed setting is better than the constant variance from the experimental results. However, I could find any theoretical justifications for this setting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PHgelwVSBvY", "reviewer": "Reviewer_vBft", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the data augmentation method to better utilize synthetic images in model training for various tasks. Here, an augmented image is generated by perturbing the synthesized image, wherein the perturbed signal is selected from a normal distribution having a large standard deviation in proportion to a high frequency band. This perturbation is applied to the amplitude signals after transforming synthetic image to the frequency domain using FFT and retransformed after the perturbation. This method was developed based on the observation that the high-frequency signal variations in the synthesized image are relatively smaller than in the real image. Using these naive augmentation methods has been effective in significantly improving accuracy in various tasks such as semantic segmentation, object detection, and object recognition.", "summary_of_the_review": "The proposed method has both the advantages of improving performance and the disadvantages of being developed too naively and not compared with other methods. I'm on the negative side now because I think the disadvantages outweigh the advantages, but my status may change after the discussion period with the authors is over.", "strength_and_weaknesses": "** Strength\n- This simple augmentation was effective in terms of accuracy on various tasks.\n- The observation that high frequency signal variations in the synthetic images are smaller than in the real images is interesting.\n\n** Weaknesses\n- The proposed method is so naive to grab attentions of researchers in ICLR. \n- There are various ways to increase the variation of the high frequency signal to take advantage of interesting observations. For example, increasing the signal size can be a method. Among these methods, what is the reason for using perturbation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NKxCIj0bH-", "reviewer": "Reviewer_tiZd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a data augmentation method PASTA for syn2real generalization. Based on the finding that real images exhibit larger high frequency variances than synthetic ones, PASTA perturbs the high frequency bands of the amplitude spectrum of the synthetic images. In doing so, the high frequency variance is larger while the semantic content is preserved. This augmentation scheme is applicable to different tasks. ", "summary_of_the_review": "In general, the empirical evidence is not sufficient, whereas the algorithm itself is rather hand-crafted. \n", "strength_and_weaknesses": "Strength:\n1. Simple approach but rather effective compared with some more complex methods.\n2. The effectiveness is showed across different tasks, e.g., semantic segmentation, object detection.\n\nWeakness:\n1. Frequency domain augmentation is an existing augmentation strategy, see listed papers below. It is also reviewed in the related work of this paper. However, there generally miss direct comparison with data augmentation schemes, especially those which are also based on spectrum perturbation. In the paper, the authors mainly compared with one and reported the number in the text for a single task. This is far from sufficient.\n2. The perturbation design is rather hand-crafted. It's also sensitive to hyperparameters such as alpha, k, and beta(also mentioned in the limitations). Therefore, without comprehensive comparisons with prior work on frequency domain augmentation, PASTA is not a convincing new solution.\n3. The scheme is more effective in SemSeg, whereas the performance in object detection is not convincing. \n4. There are multiple questionable places in the experiment part:\n- In Tab. 1-d), it is strange why a larger backbone would cause worse performance. \n- For SemSeg in Tab 1, it is also unclear why the authors did not choose the best performing method WildNet as the baseline and apply PASTA with it for a better generalization performance. \n- The in-distribution performance is not provided in Table 1. \n- From Table 2, PASTA (obviously)  hurts the ID performance. Is this always the case and any explanation for this?\n\nPapers related to frequency domain augmentation:\n- Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain\n- Frequency Spectrum Augmentation Consistency for Domain Adaptive Object Detection\n- Domain Generalization via Frequency-domain-based Feature Disentanglement and Interaction\n- Amplitude Spectrum Transformation for Open Compound Domain Adaptive Semantic Segmentation\n- A Fourier-based Framework for Domain Generalization\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BwFxMlT3gx", "reviewer": "Reviewer_PoSx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a synthetic-to-real generalisation augmentation method called Proportional Amplitude Spectrum Training Augmentation (PASTA). The proposed method specifically augments synthetic data by perturbing the amplitude spectra, with magnitudes increasing with frequency. PASTA's effectiveness is demonstrated through experiments on three tasks and different settings.", "summary_of_the_review": "Learning from synthetic data is an interesting and vital topic. This paper proposes perturbing the amplitude spectra of synthetic data for data argumentation, which may provide insights into how the community can better use synthetic data.", "strength_and_weaknesses": "Strength:\n1. The proposed method appears to be reasonable. It is an intriguing idea to manipulate the amplitude spectrums of synthetic images in the Fourier domain for data augmentation.\n2. The topic of this paper is interesting and important.\n3. A large number of experiments are carried out to demonstrate the effectiveness of the proposed method.\n4. The paper's writing is good.\n\nWeaknesses:\n1. Despite the fact that the authors conduct numerous experiments, they do not provide a sufficient discussion of the experimental results, so it is unclear how and why the proposed method works well. For example, for different tasks, the performance of the proposed method will change with different values of \\alpha (\\alpha is set to 3 in Tables 1-2 and to 10 in table 3). There is no information on how to set \\alphal, or whether it can be directly set to 0 or 3 as mentioned in Section 5.2.\n2. I think the proposed method does not be limited to synthetic data augmentation. It is preferable to discuss other application scopes, such as whether it can be used on generated data.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject