Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a synthetic-to-real generalisation augmentation method called Proportional Amplitude Spectrum Training Augmentation (PASTA). The proposed method specifically augments synthetic data by perturbing the amplitude spectra, with magnitudes increasing with frequency. PASTA's effectiveness is demonstrated through experiments on three tasks and different settings.
strength_and_weaknesses: Strength:
1. The proposed method appears to be reasonable. It is an intriguing idea to manipulate the amplitude spectrums of synthetic images in the Fourier domain for data augmentation.
2. The topic of this paper is interesting and important.
3. A large number of experiments are carried out to demonstrate the effectiveness of the proposed method.
4. The paper's writing is good.

Weaknesses:
1. Despite the fact that the authors conduct numerous experiments, they do not provide a sufficient discussion of the experimental results, so it is unclear how and why the proposed method works well. For example, for different tasks, the performance of the proposed method will change with different values of \alpha (\alpha is set to 3 in Tables 1-2 and to 10 in table 3). There is no information on how to set \alphal, or whether it can be directly set to 0 or 3 as mentioned in Section 5.2.
2. I think the proposed method does not be limited to synthetic data augmentation. It is preferable to discuss other application scopes, such as whether it can be used on generated data.

clarity,_quality,_novelty_and_reproducibility: Clarity and quality: The paper is written clearly.

Novelty: This paper provides some novel insights

Reproducibility: I believe this paper is simple to replicate.

summary_of_the_review: Learning from synthetic data is an interesting and vital topic. This paper proposes perturbing the amplitude spectra of synthetic data for data argumentation, which may provide insights into how the community can better use synthetic data.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper deals with the problem of synthetic-to-real domain generalization and proposes a method for augmenting synthetic images. The proposed method first derives power and phase components from a given source image and adds random jitters onto only power components for generating a pseudo-real image. A variance of random jitters increases as a spatial frequency increases, which indicates that high-frequency components are perturbed more than low-frequency components. Although the proposed method is simple, it outperforms baselines and several possible competitors in synthetic-to-real domain generalization.
strength_and_weaknesses: [Strength]

S1. The problem dealt with in this paper is significant. Neural network models are getting larger and larger, and their training would require a huge amount of training data. Employing synthetic images as training examples is one of the promising solutions for this situation. Technically solid methods for filling the gap between synthetic and real images will draw attention from a broad range of researchers and engineers.

S2. The proposed method is simple but effective, which will be good news for practitioners interested in large-scale model training with a few training examples. Also, the proposed method works well for many applications, such as object recognition, object detection, and semantic segmentation.

S3. The documentation is clear and the proposed method is well justified.

[Weakness]

W1. I am not sure whether the variance given in Equation (5) is the best for this purpose. I understand that the proposed setting is better than the constant variance from the experimental results. However, I could find any theoretical justifications for this setting. 
clarity,_quality,_novelty_and_reproducibility: I think that the current manuscript has no problems in terms of clarity, quality, and reproducibility.
summary_of_the_review: I have a positive feeling about this paper since the proposed method is simple and effective and the documentation is clear. Also, as far as I  know, the proposed method for synthetic-to-real domain generalization is novel.

Other minor comments:

- The following paper can be regarded as an example of random amplitude perturbations. I do not think that this paper is critical for justifying the novelty of the proposed method.
"Adversarial Bone Length Attack on Action Recognition" (AAAI2022) https://ojs.aaai.org/index.php/AAAI/article/view/20132

- I am interested in the use of generative models for the same purpose.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the data augmentation method to better utilize synthetic images in model training for various tasks. Here, an augmented image is generated by perturbing the synthesized image, wherein the perturbed signal is selected from a normal distribution having a large standard deviation in proportion to a high frequency band. This perturbation is applied to the amplitude signals after transforming synthetic image to the frequency domain using FFT and retransformed after the perturbation. This method was developed based on the observation that the high-frequency signal variations in the synthesized image are relatively smaller than in the real image. Using these naive augmentation methods has been effective in significantly improving accuracy in various tasks such as semantic segmentation, object detection, and object recognition.
strength_and_weaknesses: ** Strength
- This simple augmentation was effective in terms of accuracy on various tasks.
- The observation that high frequency signal variations in the synthetic images are smaller than in the real images is interesting.

** Weaknesses
- The proposed method is so naive to grab attentions of researchers in ICLR. 
- There are various ways to increase the variation of the high frequency signal to take advantage of interesting observations. For example, increasing the signal size can be a method. Among these methods, what is the reason for using perturbation?
clarity,_quality,_novelty_and_reproducibility: The methodology is clear and easy to reproduce with the content of the paper. However, while this method is considered novel, it is too naive to be a significant development.
summary_of_the_review: The proposed method has both the advantages of improving performance and the disadvantages of being developed too naively and not compared with other methods. I'm on the negative side now because I think the disadvantages outweigh the advantages, but my status may change after the discussion period with the authors is over.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a data augmentation method PASTA for syn2real generalization. Based on the finding that real images exhibit larger high frequency variances than synthetic ones, PASTA perturbs the high frequency bands of the amplitude spectrum of the synthetic images. In doing so, the high frequency variance is larger while the semantic content is preserved. This augmentation scheme is applicable to different tasks. 
strength_and_weaknesses: Strength:
1. Simple approach but rather effective compared with some more complex methods.
2. The effectiveness is showed across different tasks, e.g., semantic segmentation, object detection.

Weakness:
1. Frequency domain augmentation is an existing augmentation strategy, see listed papers below. It is also reviewed in the related work of this paper. However, there generally miss direct comparison with data augmentation schemes, especially those which are also based on spectrum perturbation. In the paper, the authors mainly compared with one and reported the number in the text for a single task. This is far from sufficient.
2. The perturbation design is rather hand-crafted. It's also sensitive to hyperparameters such as alpha, k, and beta(also mentioned in the limitations). Therefore, without comprehensive comparisons with prior work on frequency domain augmentation, PASTA is not a convincing new solution.
3. The scheme is more effective in SemSeg, whereas the performance in object detection is not convincing. 
4. There are multiple questionable places in the experiment part:
- In Tab. 1-d), it is strange why a larger backbone would cause worse performance. 
- For SemSeg in Tab 1, it is also unclear why the authors did not choose the best performing method WildNet as the baseline and apply PASTA with it for a better generalization performance. 
- The in-distribution performance is not provided in Table 1. 
- From Table 2, PASTA (obviously)  hurts the ID performance. Is this always the case and any explanation for this?

Papers related to frequency domain augmentation:
- Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain
- Frequency Spectrum Augmentation Consistency for Domain Adaptive Object Detection
- Domain Generalization via Frequency-domain-based Feature Disentanglement and Interaction
- Amplitude Spectrum Transformation for Open Compound Domain Adaptive Semantic Segmentation
- A Fourier-based Framework for Domain Generalization


clarity,_quality,_novelty_and_reproducibility: The idea is rather easy to follow. The experiment part requires major revision, see the comments above. 
summary_of_the_review: In general, the empirical evidence is not sufficient, whereas the algorithm itself is rather hand-crafted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a data augmentation strategy, called PASTA,  for synthetic-to-real generalization, i.e., for improving the performance on real data of a model trained on synthetic data. PASTA involves perturbing the amplitude spectrums of the synthetic images in the Fourier domain to generate augmented views. It is motivated by the observation that the amplitude spectra are less diverse in synthetic than real data, especially for high-frequency components. Thus, PASTA augments synthetic data by perturbing the amplitude spectra, with magnitudes increasing for higher frequencies (this is where the novelty of the work lies).

The reviewers find the problem addressed interesting, the aforementioned observation insightful, the proposed method simple and yet effective, and the paper well-written.  However, there is a concern about the fact that PASTA is sensitive to hyperparameters and their choice is not discussed.  One possible way is to use a validation (real) domain, but this has not been tested.  Actually, it might not even be realistic to assume the availability of a separate validation domain in synthetic-to-real generalization. Other concerns include gains being not pronounced and even underperformed other data augmentation methods and the lack of comparisons with other data augmentation schemes.

While one reviewer votes for accept and another rates it marginally above the acceptance threshold, two other reviewers consider the work marginally below the acceptance threshold. 

justification_for_why_not_higher_score: Not sufficient support from the reviewers.
justification_for_why_not_lower_score: NA
Decision: Reject