Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a novel way of performing data augmentation in image-based reinforcement learning by perturbing regions in the frequency domain of the observed images. The method improves upon state-of-the-art data augmentation methods both during training and in generalization.
strengths_and_weaknesses: Strengths:
- The explanation and visualization of the Spectrum Random Masking method are excellent.
- The proposed approach is a novel and effective addition to the existing data augmentation methods.
- Ablation studies justify the specific implementation choices in the default SRM.
- The experiments are comprehensive.

Weaknesses:
- I would love to see a real-world experiment where data augmentation plays a more important role.
questions: - Simple data augmentation methods like cropping and flipping add almost no computational overhead in the RL training loop. How demanding is SRM in computation power?
limitations: - Typo in Figure 4: average performance is the last column.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new data augmentation technique that masks out components in the frequency space: the Spectrum Random Masking (SRM). To adapt with SRM, the authors propose new techniques to stabilize the Q value after augmentation. The proposed method achieves higher performance than SVEA on video background generalization tasks.
strengths_and_weaknesses: Strengths:
1. The paper is clear and easy to understand. 
2. The proposed method is straightforward and easy to implement.
3. The performance is better than SVEA on video background environments.
4. This explores new possiblities about using strong and useful augmentations for RL.

Weaknesses:
1. The proposed method is applicable only when the observation has a wide spectrum. For example, in the color background, the proposed method cannot outperform previous method. This largely compromise the generality of the augmentation and its potential to be applied on real-world tasks.
2. This paper only test on DMC-GB. However, this benchmark is far from enough to demonstrate the generalization ability. For example, all the videos are copy-pasted to the background.
questions: 1. Do you have a in-depth answer about when the SRM technique should be applied? If I am an RL practitioner, my application wouldn't distinguish whether it is color or video background.
2. Can you test your method on one more task other than DMC-GB? For example, you can use DrawerWorld.
limitations: The authors mention that when the input is not full image, the proposed method would fail. However, the authors did not mention the limitation even when the inputs are images. But we appreciate the authors for at least mentioning that.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on zero-shot policy generalization to new environments in vision-based deep reinforcement learning. It proposes a new data augmentation method by random masking in the frequency domain, dubbed "Spectrum Random Masking (SRM)". SRM can be plugged into existing policy generalization algorithms like DrQ and SVEA. Experimental results are demonstrated on the Distracting DMControl benchmark, where the environment background is replaced by random color or distracting videos.
strengths_and_weaknesses: # Strengths

1. This paper is quite easy to follow. Fig. 2 and Fig. 3 are very good visualizations to show the intuition of frequency-domain augmentation for policy learning.
2. The proposed Spectrum Rnadom Masking method is conceptually simple and easy to understand and adopt. It is also a plug and play data augmentation module that can be integrated into many policy learning methods. The paper demonstrates experiments on DrQ+SRM and SVEA+SRM. 


# Weaknesses

My primary concern with the paper is the experiment section. The only benchmark demonstrated in the paper is Distracting DMControl, which is a set of toy environments with artificial distractions, such as random color and noisy video background. The tasks (such as Half-Cheetah, Walker-run, etc.) are not grounded in any useful real-life scenarios, and the visual distractions are not from naturally occurring sources. Furthermore, Distracting DMControl is already quite saturated, as the baselines methods can achieve close to 100% score in a significant number of tasks. 

SECANT [1] includes a number of visual policy generalization benchmarks that feature test-time visual appearance shifts that are representative of real-world applications, such as autonomous driving (CARLA [2]), robotics (Robosuite [3]), and indoor navigation in novel rooms (iGibson [4]). Please consider using these environments that are more realistic and challenging, or other benchmarks of similar calibre. 

> Reference
* [1]  SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies. ICML 2021.
* [2] CARLA: An Open Urban Driving Simulator. 2017.
* [3] robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. 2020.
* [4] IGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes. 2021.
questions: As stated in the "Weakness" section above, how does the proposed method work on environments that are more challenging and realistic than the toy Distracting DMControl? For example, autonomous driving, robotics, indoor navigation, or other practically meaningful domains? 
limitations: The limitation that the authors listed is that the method is only applicable to image-based RL tasks and cannot incorporate state information or heavily occluded image observations. 

However, I believe that another significant limitation is the applicability of the method to meaningful tasks in the real world, rather than the artificially constructed Distracting DMControl suite. The experimental evidence is insufficient as of the current writing.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work presents a new perspective of data augmentation for policy generalization in image-based RL. As different types of spatial corruptions influence the model robustness toward different frequency ranges, the authors conjecture that the observation’s the observation’
Fourier statistics for the same RL task also vary across different environment distributions. Thus, the authors propose to transform an original observation to the frequency domain, augment a portion of the frequency and inverse-transform the augmented frequency map to the pixel domain for agent training. Through such "frequency augmentation"  is shown to be effective in many generalization environments.
strengths_and_weaknesses: Strengths:
- The motivation is clear and the idea of "frequency augmentation" for RL generalization is novel.
- The experiments and ablation studies are sufficient to show the effectiveness of the proposed method.

Weaknesses:
- Related Work is not very sufficient/detailed.
questions: Questions:
1. Will SRM be robust to camera view?
2. What if use the commonly-used encoder of three convolutional layers instead of the very deep 11-conv-layers encoder used in the paper? I expect to see the experiment.

Suggestions:
1. Figure 5 suggests that Erasing is very effective. In other words, directly masking on the frequency map is very effective. It would be better if the authors discuss the relationship between recently popular masked image/frequency modeling [1-5] and the proposed SRM.
2. Better study other mask form such as fan-shaped mask and frequency-band mask, instead of only ring-shaped mask.

[1] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Xie, Zhenda, et al. "Simmim: A simple framework for masked image modeling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Xie, Jiahao, et al. "Masked Frequency Modeling for Self-Supervised Visual Pre-Training." arXiv preprint arXiv:2206.07706 (2022).

[4] Yu T, Zhang Z, Lan C, et al. Mask-based Latent Reconstruction for Reinforcement Learning[J]. arXiv preprint arXiv:2201.12096, 2022.

[5] Seo Y, Hafner D, Liu H, et al. Masked World Models for Visual Control[J]. arXiv preprint arXiv:2206.14244, 2022.
limitations: The authors have claimed some limitations.
review_title: null

Meta-review: metareview: The authors have introduced a method of data augmentation for image-based reinforcement learning that performs masking in the frequency domain, combined with techniques for stabilizing Q-learning, to achieve improved performance on a number of DMControl Generalization Benchmark tasks.

There was agreement among the reviewers that this work is novel and technically sound, and their concerns were mainly related to the breadth of tasks explored in the initial submission. During the review process the authors have gone to considerable effort to introduce new tasks (e.g. DrawerWorld, Robosuite and CARLA) and the improved performance of their method appears to generalize well. I believe that this work will be of broad interest to the RL community and recommend it for acceptance.
confidence: Certain
Decision: Accept