Spectrum Random Masking for Generalization in Image-based Reinforcement Learning
Authors: Yangru Huang
Peixi Peng
Yifan Zhao
Guangyao Chen
Yonghong Tian
Venue: NeurIPS 2022 Accept (2022)
Keywords: Data augmentation
Image-based Reinforcement Learning
Specturm

Abstract: Generalization in image-based reinforcement learning (RL) aims to learn a robust policy that could be applied directly on unseen visual environments, which is a challenging task since agents usually tend to overfit to their training environment. To handle this problem, a natural approach is to increase the data diversity by image based augmentations. However, different with most vision tasks such as classification and detection, RL tasks are not always invariant to spatial based augmentations due to the entanglement of environment dynamics and visual appearance.  In this paper, we argue with two principles for augmentations in RL: First, the augmented observations should facilitate learning a universal policy, which is robust to various distribution shifts. Second, the augmented data should be invariant to the learning signals such as action and reward. Following these rules, we revisit image-based RL tasks from the view of frequency domain and propose a novel augmentation method, namely Spectrum Random Masking (SRM),which is able to help agents to learn the whole frequency spectrum of observation for coping with various distributions and compatible with the pre-collected action and reward corresponding to original observation. Extensive experiments conducted on DMControl Generalization Benchmark   demonstrate the proposed SRM achieves the state-of-the-art performance with strong generalization potentials.

Spectrum Random Masking for Generalization in Image-based Reinforcement Learning
Yangru Huang', Peixi Peng 2 Yifan Zhao', Guangyao Chen', Yonghong Tian
Schooo of Computer Science, Peking University, Beijing, China Peng Cheng Laboratory, ShenZhen, China yrhuangdstu pku. edu (pxpeng, zhaoyf, gychen, yhtian edu
Abstract
Generalization in image-based reinforcement learning (RL) aims learn robust policy that could be applied directly unseen visual environments, which is a challenging task since agents usually tend to overfit to their training environment. To handle this problem, natural approach is to increase the data diversity by image based augmentations. However, different with most vision tasks such as classification and detection, RL tasks not always invariant to spatial based augmentations due to the entanglement of environment dynamics and visual ap- pearance. In this paper, argue with two principles for augmentations in RL: First, the augmented observations should facilitate learning universal policy, which is robust to various distribution shifts. Second, the augmented data should be invariant to the learning signals such as action and reward. Following these rules, we revisit image-basee based RL tasks from the view of frequency domain and propose a novel augmentation method, namely Spectrum Random Masking (SRM).which which is able to help agents learn the whole frequency spectrum of observation for coping with various distributions and compatible with the pre-collected action and reward corresponding original observation. Extensive experiments conducted on DMControl Generalization Benchmark demonstrate the proposed SRM achieves the state-of-the-arl performance with strong generalization potentials.
Introduction
Reinforcement Learning (RL) from image- image-based observations task where representation and Decision-making are jointly learned from visual signals. Although this type of RL methods has made significant progresses due to the development deep Convolution Neural Networks (CNNs). most of them suffer from drastic performance decline previously unseen environments [24 44 4]
especially for image-based inputs 44
27 The reason is that classical RL methods assume the environments of training and testing are identical [17]. However, there usually exists significant distributions divergences between them in real-world scenarios. The high dimensionality visual inputs coupled with the shifted distributions of environment make the generalization of RL tasks even more challenging 33).
32
To handle this challenge, one natural way is to decrease the distribution gap of training and testing data by simulating the diversity of test euvironments in training. Recent studies show that the appropriate data augmentation could facilitate the generalization of the policy network [4]] Most of these data
augmentation methods have been carried out directly in the spatial domain [19] such as flipping and
rotation. However, the dynamic of environment has the disadvantages of lacking sensitivity to such spatial-based data augmentation
28 An example illustrated in Fig. when horizontal flipping
Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Input
Invariant Peature
Ingot
Irvarfant Feature
Policy
Outpet
Envirnmment
Outppt
Encoder
Cartpole of
Eacoder
night
swingup
Original
Original
right
stop
X
Cartgole
V
left
Flipped Ô
Flipped Ó
Computer vision
Retnforcement Leaminng
Figure 1: For image recognition task (left), the features of original images and flipped images ô should be invariant for classification. In contrast, for cartpole swingup task (right) in RL whose goal swing up an unactuated pole, such invariant feature will guide policy network output same actions for o and ö. However, for o, the better action is to apply reversed forces to this cart base to keep swingup. Thus the pre-collected action and reward for are not accurate for â any more.
applied observations, the corresponding left and right actions should be reversed, thus collected rewards could not reflect the correct response.
Considering above problem, instead of working in the spatial domain, we aim to study the gen- eralization of image-based RL from frequency domain perspective. Our motivation stems from the recent studies Fourier analysis in computer vision 42: : different types of spatial corruptions influence the model robustness toward different frequency ranges. For example, the high frequency data augmentations such as random Gaussian noise bias the model towards utilizing low frequency information in the input, and improve robustness to corruptions with high frequencies. Moreover, single spatial- -ased data augmentation rarely improves model robustness of corruption types with different frequency characteristics. Inspired by this phenomenon, we conjecture that the observation's Fourier statistics for the same RL task also vary across different environment distributions. In Fig. 2 we visualize the observation's 's spectrum magnitude of the the original environment as well as averaged delta between 4 shifted environment distributions shifted environments show distinct spectrum difference with the original environment, with each one varying at different frequency regions. Knowing that a deep model often tends to bias on certain frequency band [39], we hypothesis that the diverse spectrum pattern across environment distributions is the key challenge of image- RL generalization.
DeepMind Control suite 34 The
Based on the above analyses, we focus on the Frequency-baid augmentation instead of the spatial- based one by introducing a Spectrum Random Masking (SRM) regularization, shown in Fig. 3 Our proposed SRM can be easily compatible for most existing RL benchmarks. To prevent the model from focusing on a certain frequency range input images, SRM randomly discards partial frequency of observations at training phase, and forces the policy to select the appropriate action with remaining information. In this way, we can increase the robustness to different corruptions (high, middle or low frequency characteristics) by considering the entire frequency distribution. Specifically, an input observation is operated by three steps: (1) the Fast Fourier transformation (FFT), (2) Spectrum random masking, and (3) the inverse Fourier transformation (IFFT), which enhances the diversity of training observations while maintaining the main content.
To summarize, this paper has the following contributions: (1) We provide new frequency perspective for data augmentation in RL. In contrast to previous spatial-based methods, the proposed SRM are performed consistently for task-agnostic RL environments and could deal with diverse distribution by focusing on the whole frequency information. (2) The proposed SRM is plug-and-pla, method that does not require any extra parameters or any additional observations from the environment. Moreover, it is complementary to existing data augmentation and regularization approaches. (3) We conduct comprehensive experiments to demonstrate SRM could bring performance gains on various image-based environments with high generalization potentials.
2 Background
Reinforcement learning. A RL. problem typically described as a Markov Decision Process (MDP) M =< S,A.P,R, >, where S and A denotes the state space and action space respectively [33

---

Oniiial
Color eeey
Color
caey
Video
Figure 2: Spectrum magnitude of the original environment observation [[]] and the obser- vation spectrum magnitude difference [[ - D[ of four shifted environments. F is the 2D Fourier transform and D is the environment shift function. Sample observation images for each environment are placed on the left of corresponding spectrum map. A total of 500K frames on the cartpole swingup task in DeepMind Control suite are used for calculation and the results are averaged.
P(s ae) is the state transition, R(s at) is the reward function, and y E [0. 1) is the discount factor. Here consider Partially Observable MDP (POMDP) (2) M =< O, R, for the continuous control task because S is often not directly obtained from raw images, where O is the high-dimensional observation space of images. Following the convention (25), the POMDP is converted into an MDP by stacking several consecutive environment frames {ot, 0t,1, ((-2) into a single state 8, O.
The goal of typical RL to find an optimal policy zh which maximizes the expected cumulative reward over the entire distribution of MDPs [3]:
(x)
Σ (eonae) [r(s,, at)],
where (st, 0e) is the state-action marginal of the trajectory distribution. In this work, rather than learning an optimal policy on a single MDP, we consider a generalizable тT. which is able to obtain high discounted return on a set of MDPs M =< 0, A. R >, where the observations in O are unseen training.
Soft Ector-Critic Soft Actor-Critic (SAC) 10 a popular off-policy method for continuous
control tasks. It learns a state-action value function Q(s, a) and stochastic policy (a)ss) to find the optimal policy based on the maximum entropy RL framework:
J ( B(sr,a))>p (r(s), a.) + aH(s(-)))))l
(2)
where a is the temperature parameter which determines the relative importance of the entropy term against the reward.
0 degenerates SAC into conventional RL. H(()) provides substantial improvement in exploration and robustness by enabling the probability of action output to be diverse as much as possible, rather than being concentrated one action. Specially, Q(s, a) is approximated by minimizing the soft Bellman residual:
JQ ( a,) (ra + (S(++
(3)
where D denotes the replay buffer, and V s2+1 is the soft target value network approximated as:
V Ba Q ((s+11,
a log ((c
(4)
where is the target Q function, and the weights can be exponentially moving average of the weights in Q. In the policy improvement step, the policy is updated by minimizing the divergence between the policy and the exponential of the soft function:
Jn = E,,ND| (DKL(T((s))] eex((((s))) ))
})]
(5)
Z(s:)
Methodology
3.1 Spectrum Random Masking
We firstly introduce how to generate new samples by spectrum random masking strategy in this section. Suppose we have grayscale image observation E # the Fourier domain spectrum
For RGB images, apply the same operation separately each color channel.

---

Observations
Spectrums
FFT
SRM
Erasing
Noise
Swapping
r 0.0
12 0.1
r1 0.2
2 0.3
r 044
225
Figure 3: The framework and several examples of SRM. For three consecutive observations, SRM adopted as data augmentation respectively. There are three types of masking strategies including directly erase, disturbed by noise or swapping. During training, the masking strategy, position and ratio could be dynamic changed.
F(o:) can be calculated by fast Fourier transformation 26
#-1 W-1 (h, w)e j2 (2r(hh/H+o///W))
F(o,)(u, = Σ
(6)
4=0 w=0
where u E [0, H - components, and (h, w) is the pixel value at position (h, w). Note the subscript is just the index of the sample and irrelevant with the time step. For better processing and visualization, we use the shifted version of F(o,) with the lowest frequency component placed in the center of the spectrum.
1] and E [0, (0,W-1 1I are horizontal and vertical indexes of frequency domain
After transforming images from spatial to frequency domain, we construct a binary masking matrix M E (0, 1) >> We introduce two radius parameters r1, [0, 0.5) to control the position and ratio of the spectral neighborhood to be masked (r1 < r2). As shown Fig. M is constructed by setting the region in the annulus between two concentric circles (with radius r11 and r2) to O.
Ar = T2 - Ti decides the size of mask regions. The area ratio of masking region is defined as Tp r rf. Hence we propose typical types of masking strategies: erasing, noise and swapping. If the frequency component lies between r1 and r2, will be masked; otherwise, the frequency signals will be kept:
MM + (1- (1- F(Z),
where Z is calculated by the specific masking strategy. For the erasing strategy, we set Z to 0 "xW and only the non-masked frequency components will fully explored follow-up training. The random noise strategy sets Z to a random noise image. The swapping strategy replaces the removed frequency components with the same frequency components from another image by setting which draws inspiration from Cutmix [43] and Mixup 477. Considering that the third-party data I is not always available, during training, we randomly shuffle the training batch and pick I 0, (i * from the shuffled batch, where oj has the same batch index with o,.
I,
The masking strategy also influences the supervised signals since is correlated with o, Suppose that 0g, is the supervised signals for observation o,, where and stands for action and reward, respectively. For erasing and random noise strategy, a; and r; are kept unchanged. For the swapping strategy, the supervised signals depended on the area ratio of masking region rp. If rp 0.5, we keep a; and r; unchanged, else we replace as Ti with ay and rj.
Once the masked Fourier spectrum (o,)(u, v) is obtained, we inverse shift and restore it to the spatial domain by the corresponding inverse Fourier transformation F-1;
W-1 Σ
6, w)
Σ
Flo:)(u, Rrth
HW
Herr assume the diagonal length of the 2D Fourier spectrum 1.0 to deal with different image sizes.

---

Algorithm Spectrum Random Masking
Input:
Sample batch of transitions: for, ag, rs ofli = 1, 1,... .K} B; Zero matrix 0 E "xxW
Random noise image E RHxW; Masking matrix M (0. 1]
Output: Augmented transitions {0;, âz, nr, 0f = ;, ,
Init: r1 f R0 0.5), 72 + Rand(rI, 0.5), Ar = r2 -1, Tp = 12 - rf.
1: for i 1,..., K do
2: Fast Fourier transformation according to Eq. 6 F(o,) t oi
3: if Masking strategies is 'Erasing' then
4: F(o:) M F(o,) + (1 M) F(0), d, ai r;
5: else if Masking strategies is 'Noise' then
6: F(a:) = M F(o:) + (1 - M) F(N), dj + 0j. fi t
7: else if Masking strategies is 'Swapping' then
8:
Random Shuffle Batch: {oj, "j, o,li = 1,... K}
F(o.) M ((o)) + (1 -
9:
if rp S 0.5 then
à -- 0g. Pi 6 Pr
11:
else
12:
à; + 0g, Pi 4- end if
13:
14:
end if
15: Inverse Fourier transformation according to Eq. 8 o, +- F(o.)
16: Execute Steps (25~(15) with of to get 65
17:
end for
For observation o,, its next observation 0 also goes through the same SRM operation as o,. Algo- rithm details how to generate new samples by SRM.
3.2 Reinforcement Learning with SRM
In off-policy RL., observations sampled from replay buffer. One intuitive way to data augmentation in RL to perform augmentation before passing them to the agent for training [4] 19 Meanwhile, the augmentation procedure stacked consecutive state frames 8; {or, 0 0:-2 should be consistent to keep temporal information (such as positions and velocities) invariant. Given an augmentation function T(:.v) with parameter E V, the augmented observation & E Ô of the original observation O can be obtained by & T(,,, v). Optimally, T : x should maintain Q-values and policy æ of the agent:
{ Q(5), (c) Q(T(S( a!),
((als))) ((te)T(s,,)))
where E A is the action of However, practically it is unstable to learn value and policy functions even from the original observations, not to mention the augmented data that vary wildly in appearances. As result, only several weak data augmentations such as random shift and random cropping could be employed In contrast, strong data augmentations such random convolution and rotation are always been applicd in a specific way. SVEA [13, both original and augmented images are used to optimize the Q-function, but only the former are adopted compute Q Q-target of the Bellman equation. In SECANT 77, expert policy with weak augmentations guides student policy with strong augmentations for learning robust representations. Attractively, the merit of both weak and strong data augmentations: it can be plugged into any image-based RL methods in plain way, while providing significant performance improvement.
plain way in RL optimization, as shown DrQ [4] and RAD [1).
will show in the experiment section, SRM has
During training, we apply SRM with a probability of 0.5 on each batch of observations. No data augmentation is applied at test time. To evaluate the effectiveness of SRM, we adopt two popular methods DrQ 4 and SVEA [3] as our base algorithms. The former a naive method where data augmentations are plugged into RL algorithms directly, and the corresponding SAC objectives are
5

---

3030
wall, Original
Waller
Walker
822
Expaedde menum
830
6.30
Eare 223
Number framea (x10%)
Kumber Names (xto")
Number frames (x10%))
Number frames (x20%)
Number frmmss (x109)
eway
E0 eetumn
800
Ep-oode (p-sootereeuum return
232
Mpp
Exp saste return
Ep returm
832
90
820
400
400
200
230
200
200
0.1
53
03
B4
01 0. 63
03
62
03
8.1
02
23
(x10%)
kumber
Cartpole extrnuup Original
Cartpole sw noup Color easy
Cartpole swngup Caler hard
Cartpole swingup Video easy
Cartoon swinupp Average
5p-0000 edtum
011 0,2
03
0.4 0.
0.0
11
0.2
03
84
0.5
5.0
03
02
0.3
8.
03
01
0.
03
0.4
25
0.5 n. Ball cup catch, Average
E.1. 022
Number froms (x30%)
Number framees
Number frames (#10
Number frames (x10f)
Ball catch, Drignal
Ball in catch, Calor easy 1006
Ball catch, Coler hard
mhal catch, Video
Ephontd retum
.33
Eublacde vatum
000
Triente ahin
800
500
600
83
600
400
200
200
Number of Trames (410f)
Numbercd of Trames (H32
Number Trames (x10%)
Number of frames (4107)
Number of trames (x30f)
Dro
Rotation
Crouut
DrQ opopuut
DIQ+SRM
Figure 4: Comparisons of SRM with five data augmentation and one regularization methods. The average performance in the last column shows the generalization ability of SRM.
respectively replaced by:
Jo (.) (Q(se, at) + Q(t,, az)) (rx + + V(s++1) + (((++1) )²]
(10)
2
The latter is delicately designed where Q-target is computed strictly from unaugmented data to alleviate the problem of high-variance and its goal is:
Jq = -((eraa))D (Q(s), ar) + Q()., (rz + - (8444)))
(11)
2
By jointly solving the Q-function with policy and value network in Eqs. and (5). the RL systems has the potential to learn from diverse observations for policy learning.
4
Experiments
4.1 Task Setup and Evaluation Protocol
Environments We conduct our experiment on 5 tasks from DeepMind Control Suite (DMCon- trol) (34,, which offers a set of challenging continuous control tasks and has been widely used as a common benchmark for visual RL [I] There are 2 kinds of observations in DMControl: state vector-based and image-based, and the latter used in our experiment. It has been evidenced that the learning of state-based observations is easier than high dimensional image-based observations (18) To evaluate the generalization, we train the agent from DMControl and report results tested on DMControl Generalization Benchmark (DMControl-GB) [14, which includes four distinct test distributions with slight/strong color/video attack: color-easy, color-hard, video-casy and video-hard..
Implementation Details For a fair comparison, we implement all methods following [3). where the same hyperparameters and network architecture are adopted. We use a 11-layer forwadd convolution network as the shared encoder, which is followed by independent linear projections for the actor and critic. During training, the masking ratio and position of SRM are randomly chosen, and the ranges of and Ar are set as [0, 0.5) and [0, 0.05) for each batch of observations, respectively. Note SRM a data augmentation method and needs to combine with other learning baselines. To validate the effectiveness, 3 ffte-of-theeart basclines are employed: DrQ 41). SVEA-C [13] and

---

Table 1: Comparison with state-of-the-ar methods color-easy, color-hard, video-easy and video- hard benchmarks. C and 0 denotes random convolution and overlay respectively. The best results are
in bold.
Color, Easy
CURL
RAD
PAD
SODA-C
SODA-0
DrQ
DrQ +SRM
SVEA-C
SVEA-C +SRM
SVEA-D
SVEA-D
+SRM
walker,
645
636
687
811
826
912
758
941
868
910
walk
+55
+33
119
* 41
10
+ 21
+22
20
18
26
walker,
866
807
894
960
929
967
832
978
974
980
stand
46
= 39
=
40
=
= 1
=
caripole.
668
763
812
865
862
867
swingup
74
29
20
+
1.5
+
+:
+ 4
+ 8
ball in cup.
565
727
775
969
840
972
969
980
976
978
catch
* 168
87
1.99
*
36
+
+ 8
* 1
+ 1
+ 2
finger,
781
789
870
915
861
904
961
963
976
spin
139
160
55
+:40
20
122
14
705
744
808
903
876
935
866
946
929
942
Average
96
75
+ 78
+ 21
+ 42
+ 10
+ 33
+ 10
= 8
+ 10
Color,
CURL
SODA-0 0
SVEA-
SVEA-O
SVEA-
SVEA-D
Hard
+SRM
+SRM
+SRM
walker,
445
400
468
697
692
520
806
760
907
749
836
walk
99
i 66
i
1
:
145
36
61
2.27
walker,
662
644
797
930
893
770
874
942
966
933
965
stand
54
88
46
12
12
71
14
26
11
24
13
cartpole,
454
590
630
831
805
586
802
837
863
832
833
swingup
# 110
+3
# 63
: 21
: 28
+ 52
+ 28
23
15
+ 23
24
ball _in_cup.
231
541
563
892
949
365
796
961
966
959
948
catch
92
29
37
210
7
12
finger,
691
667
803
793
776
889
977
974
972
971
spin
12
154
* 51
128
* 134
* 22
* 5
6.3
+ 6
+ 13
Average
497
568
652
850
826
603
833
895
935
889
911
73
77
56
# 3.
# 51
= 112
+1
27
24
25
Video
CURL
RAD
PAD
SODA-C
SODA-0
DrQ
DrQ
SVEA-C
SVEA-C
SVEA-D -
SVEA-D
Easy
+SRM
+SRM
+SRM
walker,
556
606
717
768
G18
836
819
854
walk
133
63
79
38
144
+:40
71
42
walker,
852
745
935
903
955
873
947
795
932
961
966
stand
77
# 146
# 20
: 565
# 13
+ 83
1
+
: 10
= 8
+ 42
cartpole,
404
373
521
474
758
485
740
606
808
782
812
swingup
66
72
143
*
*
+
85
16
cup.
316
481
436
339
875
318
651
639
882
871
924
catch
* 119
26
55
111
56
+ 157
+ 19
# 110
= 100
106
+ 35
linger,
502
400
691
363
695
533
607
764
816
808
925
spin
19
80
185
: 97
: 119
+ 92
86
93
# 33
526
583
810
688
855
848
896
Average
+38
+:74
62
109
* 53
111
99
+.53
49
28
Video
CURL
RAD
PAD
SODA-C
SODA-D
DrQ
DrQ
SVEA- C
SVEA-C
SVEA-D 0
SVEA-D
Hard
+SRM
+SRM
+SRM
walker,
58
56
93
381
104
225
224
364
377
535
walk
18
+
29
* 72
22
* 29
+7
63
93
35
walker,
45
231
278
771
289
325
525
729
834
863
stand
=5
39
72
=
49
+
+39
#:29
46
57
caripole,
114
110
123
492
213
425
393
523
swingup
15
16
24
+ 64
9
+ 58
#26
5
4545
23
ball_ in cup.
115
97
66
327
92
237
413
339
403
S66
catch
33
+9
61
100
23
+
+57
1.
174
135
finger,
27
34
56
302
71
430
335
335
419
spin
+11
+11
188
41
45
+44
21
58
32
72
106
123
455
139
227
361
438
468
581
Average
+ 18
21
441
+ 72
+ 30
4 4440
135
2.27
+ 83
56
SVEA-O [13), denoted as "DrQ+SRM", "SVEA-C+SRM" and "SVEA-O+SRM" respectively. O and C represent random convolution and random overlay respectively, which are as the default augmentation used in SVEA 13 For DrQ, SVEA-C and SVEA-O, the better performances between the reported ones of the original paper and our re-run ones using the source codes are reported in our experiment. We provide 3 candidate masking strategies erasing, noise and swapping in Algorithm [ all of them can achieve higher performance than baseline as shown in Table Since erasing performs better than others, it is chosen as the default masking strategy in the following experiments if not otherwise mentioned. and report the averaged results.
methods are trained for 500K frames. We run times with random seeds

---

3000
Walter Original
Waller, walk, Color
Walker walk, Color
Walker
Valker
Epioode retum
800
Sphadde ratums
-000
200
100
120
Epsside
800
700
200
0.0
0.1
0.2
0.3
0.4
0.
00
01
02 03 04 of
0.
6.1
6.2
2-3
6.4
6.5
60
6.1
0.2
0.2
0.4
oz
6.0
0.1
0.2
0.2
944
05
Number frames (x10*)
Number of frames (x10")
Number of fremes (x30*)
Number of fromes (x10
Number of fromess (x10")
Cartpow vetngup, Of qraal
Cartonle swrrgge, Coler easy
Caripow vaingup. Color hard
Cartpole saingun. Video, eavy
Caripo www.gup. Average
100
12trode ret."
100
600
400
500
400
Eptepdt
220
200
120
03
0.2
0.3
0.4
0.1
0.3
0.3
03
0.3
C4
03
(x10")
(x
DrQ
DrQ Noise
+wappingg
Erasingg
DrQ
Mix
Figure 5: The results of different masking strategies. Either way, disturbed in frequency domain is useful way to improvement performance.
4.2 Comparisons
Comparisons with state-of-the-arts The comparisons are exhibited in Table [ For the direct comparison, "DrQ+SRM", "SVEA-C+SRM" and "SVEA-O+SRM" outperform DrQ, SVEA-C and SVEA-O with a clear margin, respectively. In addition, the usage of SRM reduces the standard deviation and shows the strong leaning stability. It indicates that performing augmentation in the frequency domain is effective for image-based RL tasks. In addition, the performance of other methods such as CURL 18 PAD [12], RAD [99 and SODA [14] are also reported. Since our method is implemented with different learning baselines with them, the results are just provided for reference. Overall, it is evident that SRM provides new of-te-of-the-art performance for the task.
Comparisons with spatial-based data augmentation Here we compare SRM with 5 spatial-based data augmentation methods and I removal regularization strategy, including random convolution 20) random rotation R random cutout [5], random flip 19] and dropout BII DrQ employed as the baseline learning method. When utilizing each data augmentation method, 50% of frames are remain un-augmented. This is because we find that directly using all kinds of data augmentation is harmful to policy stability. The dropout layer is added after every nonlinearity layer in the MLP of actor and critic networks and the dropout probability set to 0.1.
Since the video hard task is extremely hard and the learning of DrQ is oscillating, we show learning curves of the different data augmentations in the benchmarks of color-easy, color-hard and video-easy. As shown in Fig. 4 the spatial data augmentation methods are less effective and even degrade perfor- mance, especially for flip and rotation. The reason is that these augmentations inevitably changes the environment dynamics, leading to inaccurate state transitions. In contrast, SRM outperforms the baseline in most scenarios and achieve comparable results in the rest scenarios. It demonstrates that the generalization ability of frequency based SRM is stronger than its spatial spatial-based based counterparts. Note that the "random cov" performs well on color hard and is less effective than SRM in other situations. The reason is that applying random convolution mainly changes the low frequency components (i.e., colors and textures) of an image, which makes the agent robust to color changes. However, when facing environments that differs significantly on higher frequency components, random convolution becomes less effective. Compared with "random cov", SRM works more generally to different scenarios.
Fig. 4 also tells that, although the removal based methods dropout and cutout has been proved to be able to improve generalization for discriminative models, they decrease the performance of RL tasks. For cutout, randomly cutting& region of an input image is prone to information loss, especially when the target region (e.g., the cartpole in cartpole swingup task) is removed. The performance decrease of dropout can be attributed to the chaotic nature of the RL training when facing with random dropped hidden units. In contrast, each frequency component in the Fourier spectrum has the global vision of the whole image, thus SRM does not remove critical image regions. Meanwhile, as an augmentation method, SRM also does not influence hidden units of networks.
8

---

Carpolle_sehgggg.
Swiingu
Cortpol:,
Epro0d eetum
Egea retur
993
Eptoodio rcture
Epesodie retur
600
893
203
200
1200
6.1
01
0.
0.5
0.3
0.3
3.3
(4.30P)
11-0.0/2=01
12-0.1,2-022
22-02,
2--33/2-04
01-0.,,2-0055
w/o SRM
SRM
Figure 6: The influence of masking position. Masking frequency random band is better than on a fixed band.
Chrgad
Color say
VId6o way
Anasa
100
400
02
5P6
03
900
02
81
622
055
04
5000
Figure 7: The influence of masking ratio. Masking frequency within a random ratio is better than within a fixed ratio.
4.3 Ablation Study
Spectrum Masking Strategies There are four different frequency masking strategies including erasing, noise, swapping and a mixture of the first three. As shown in Fig. [5] we study the choice of masking value on the performance of RI. and have the following observations: First, all of these masking strategies are able improve the average performance. This benefits from the mechanism of random masking which attempt to strengthen the whole frequency learning while suppress overlitting to certain frequency range. This also demonstrates that the agent may underfit to certain frequency range and overfit to other range when directly training with original observations. Second, with the increasing complexity of frequency disturbance, the performance of mixture is sometimes much worse than solely using erasing or swapping. This phenomenon can be attributed to introduce more variance by mixture.
Spectrum Masking Position To explore how the low and high frequency affect policy training, we set L1 to 0.1, 0.2, 0.3, 0.4 respectively, and Ar to 0. 1. According to Frequency Principle (F-Principle), DNN tends to captures the frequency from low to high components in ascending complexity order. This is consistent with our results as shown in Fig. 6 At the early training stage, masking low frequency lead to slower improvement, while masking high frequency leads to a similar improvement with the non-SRM baseline. In the later period, remaining high frequency is able achieve higher results than remaining low frequency. Moreover, SRM could get the best results in most benchmarks than fixed masking position. It demonstrates that learning frequency from low to high uniformly in random way is beneficial.
Spectrum Masking Ratio To better control the masking ratio, we set Ar to 0.05, 0.1,0.2,0.3 and 0.4 respectively. The results are shown in Fig. 7 Notably, SRM with random masking ratio consistently outperforms the DrQ baseline under all parameters setting, especially for harder environment. In addition, we can observe that masking ratio has little influence on original environment, which can be attributed to the remaining of decent percentage of un-augmented frames.
5 Related works
Generalization in RI. draws increasing attention recent years. Existing methods can be roughly categorized into two groups: (1) increasing distribution similarity between training and testing environments by augmentation methods (such as data augmentation and environment generation) [16] and (2) handling differences between the training and testing MDPs (such
20 18 19 14
28
as regulariss regularisation) but works in the frequency domain, which is ignored by most current RL methods. To explain the generalization behaviors of CNN, recent works in computer vision also provide new insights from the frequency domain aspect domain shift between fixed datasets on discrimination tasks (e.g., classification and segmentation). In contrast, we consider the dynamic environment RI. as well as the joint representation learning and pitecisson-maaingg process. Moreover, CNN tendssince CNN tends to rely either low frequency
15
46 6 [1 [1] 36, 23 29 37 Our method belongs to the first group
35
38 3 However, they mainly consider the

---

or high frequency components for fixed tasks during training [39 existing methods rarely consider regularizing the model on all frequency ranges. Differently, our method considers the broad frequency difference hetween different RI. environment distributions and regularizes the model on all frequency ranges uniformly.
Conclusions
In this work, we investigated the generalization of image-based RL from frequency domain perspective. Motivated by the distinct spectral impacts of different environment distribution shifts, we develop Spectrum Random Masking (SRM), simple yet effective frequency domain augmentation methodwhich randomly discards certain frequency bands the observation. Experimental results show that SRM improves model robustness under various distribution shifts without violating the environment dynamics. The effectiveness of SRM highlights a promising future research direction for researchers to take into account the frequency domain intrinsic mechanism of RL models. As RL tasks are essentially different from conventional computer vision tasks (e.g.. image recognition), we believe that different conclusions will be made to provide further intuition and benefit the model performance.
Limitations: As an image-basee basaee augmentation method, our proposed SRM resort to enhancing the diversity of image-base observations while maintaining the crucial reward-related content, which facing challenges when image observations are incomplete or only state information can be observed.
7 Acknowledgments and Disclosure of Funding
This work is partially supported by grants from the Key-Area Research and Development Program of Guangdong Province under contact No. 2020B010138000 and grants from the National Natural Science Foundation of China under contract No. 62027804, No. 61825101 and No. 62088 102. The computing resources of Pengcheng Cloudbrain are used in this research.
References
[1] Rishabh Agarwal, Marlos Machado, Pablo Samuel Castro, and Marc Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021.
[2] Karl Johan Aström. Optimal control of markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174-22) 1965.
[3] Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, and Yonghong Tian. Amplitude-phan recombination: Rethinking robustness of convolutional neural networks in frequency domain. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 458-467,
2021.
[4] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. International Conference on Machine Learning, pages 2048 2056. PMLR, 2020.
[5] Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. International Conference on Machine Learning, pages 1289. PMLR, 2019.
[6] Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient rl with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1674.1674. PMLR, 2019.
[7] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In International Conference on Machine Learning. 2021.
[8] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1oXivv1803.0777 2018.
10

---

[9] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
[10] Tuomas Haarmoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.55905 2018.
[11] Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[12] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations, 2021.
[13] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in Neural Information Processing Systems, 34, 2021.
[14] Nicklas Hansen and Xiaolong Wang- Generalization reinforcement learning by soft data augmentation. In 202/ IEEE International Conference on Robotics and Automation, pages 13611-13617. IEEE, 2021.
[15] Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian T'sciatschek,, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. Advances in neural information processing systems, 32, 2019.
[16] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to- sim: Data-efficient robotic grasping via randomized-o-ccaonicall adaptation networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12627-12637, 2019.
[17] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation deep reinforcement learning, arXiv preprini
2021.
[18] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Leurning, pages 5639-5650. PMLR, 2020.
[19] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33:19884-199888 2020.
[20] Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization deep reinforcement learning. In International Conference on Machine Learning, 2020.
[21] Guoqing Liu, Chuheng Zhang, Zhao, Tao Qin, Jinhua Zhu, Jian Li, Nenghai Yu, and Tie- Yan Liu. Return-based contrastive representation learning for reinforcement learning. arXiv preprint arXiv:2102.10966. 2021.
[22] Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimization an empirical study on continuous control. arXiv preprint arXiv:1910.09191, 2019.
[23] Chaochao Lu, Yuhuai Wu, José Miguel Hermández-Lobtte, and Bernhard Schölkopf. Invariant causal representation learning for out-o--distribution generalization. International Conference on Learning Representations, 2022.
[24] Dhruv Malik, Yuanzhi Li, and Pradeep Ravikumar. When is generalizable reinforcement learning tractable? Advances in Neural Information Processing Systems, 34, 2021.
11

---

[25] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13/2.560, 2013.
[26] Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms, pages 80-111. Springer, 1981.
[27] Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810. 12282, 2018.
[28] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, llya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[29] Anoopkumar Sonar, Vincent Pacelli, and Anirudha Majumdar. Invariant policy optimization: Towards stronger generalization in reinforcement learning. In Learning for Dynamics and Control, pages 21-33. PMLR, 2021.
[30] Xingyou Song. Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. In International Conference on Machine Learning, 2020.
[31] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, (5(1):1929-1958 2014.
[32] Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting con- trol suite- challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722, 2021.
[33] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[34] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801. 00090, 2018
[35] Haohan Wang, Xindi Wu, Zeyi Huang. and Eric Xing. High- freeufreyynnc component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[36] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. Advances in Neural Information Processing Systems, 33:7968-799 7978, 2020.
[37] Xudong Wang. Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for reinforcement learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6677-6687, 2021.
[38] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian, A fourier- framework for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14383-14392, 2021.
[39] Zhi- thi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In International Conference on Neural Information Processing, pages 264-274. Springer, 2019.
[40] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4085-4095.... 2020.
[41] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2020.
12

---

[42] Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. Advances in Neural Information Processing Systems, 32, 2019.
[43] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2019.
[44] Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:/806.077 2018.
[45] Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos, Marta Kwiatkowska, Joelle Pineau, Yarin Gal, and Doina Precup. Invariant causal prediction for block mdps. In International Conference on Machine Learning, pages 11214-1 1224. PMLR, 2020.
[46] Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine, Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006.1077 10742, 2020.
[47] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.09412, 2017.
[48] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin robosuite: modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12993 2020.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See conclusions.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4..
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Tab. []
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1 We public environments with proper citations.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The data used in this paper does not contains offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "oW2fCqEDIz3", "reviewer": "Reviewer_JBru", "summary": "This work presents a new perspective of data augmentation for policy generalization in image-based RL. As different types of spatial corruptions influence the model robustness toward different frequency ranges, the authors conjecture that the observation’s the observation’\nFourier statistics for the same RL task also vary across different environment distributions. Thus, the authors propose to transform an original observation to the frequency domain, augment a portion of the frequency and inverse-transform the augmented frequency map to the pixel domain for agent training. Through such \"frequency augmentation\"  is shown to be effective in many generalization environments.", "questions": "Questions:\n1. Will SRM be robust to camera view?\n2. What if use the commonly-used encoder of three convolutional layers instead of the very deep 11-conv-layers encoder used in the paper? I expect to see the experiment.\n\nSuggestions:\n1. Figure 5 suggests that Erasing is very effective. In other words, directly masking on the frequency map is very effective. It would be better if the authors discuss the relationship between recently popular masked image/frequency modeling [1-5] and the proposed SRM.\n2. Better study other mask form such as fan-shaped mask and frequency-band mask, instead of only ring-shaped mask.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Xie, Zhenda, et al. \"Simmim: A simple framework for masked image modeling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Xie, Jiahao, et al. \"Masked Frequency Modeling for Self-Supervised Visual Pre-Training.\" arXiv preprint arXiv:2206.07706 (2022).\n\n[4] Yu T, Zhang Z, Lan C, et al. Mask-based Latent Reconstruction for Reinforcement Learning[J]. arXiv preprint arXiv:2201.12096, 2022.\n\n[5] Seo Y, Hafner D, Liu H, et al. Masked World Models for Visual Control[J]. arXiv preprint arXiv:2206.14244, 2022.", "limitations": "The authors have claimed some limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation is clear and the idea of \"frequency augmentation\" for RL generalization is novel.\n- The experiments and ablation studies are sufficient to show the effectiveness of the proposed method.\n\nWeaknesses:\n- Related Work is not very sufficient/detailed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "b3l7ZENwTyu", "reviewer": "Reviewer_UTXd", "summary": "This paper proposes a new data augmentation technique that masks out components in the frequency space: the Spectrum Random Masking (SRM). To adapt with SRM, the authors propose new techniques to stabilize the Q value after augmentation. The proposed method achieves higher performance than SVEA on video background generalization tasks.", "questions": "1. Do you have a in-depth answer about when the SRM technique should be applied? If I am an RL practitioner, my application wouldn't distinguish whether it is color or video background.\n2. Can you test your method on one more task other than DMC-GB? For example, you can use DrawerWorld.", "limitations": "The authors mention that when the input is not full image, the proposed method would fail. However, the authors did not mention the limitation even when the inputs are images. But we appreciate the authors for at least mentioning that.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is clear and easy to understand. \n2. The proposed method is straightforward and easy to implement.\n3. The performance is better than SVEA on video background environments.\n4. This explores new possiblities about using strong and useful augmentations for RL.\n\nWeaknesses:\n1. The proposed method is applicable only when the observation has a wide spectrum. For example, in the color background, the proposed method cannot outperform previous method. This largely compromise the generality of the augmentation and its potential to be applied on real-world tasks.\n2. This paper only test on DMC-GB. However, this benchmark is far from enough to demonstrate the generalization ability. For example, all the videos are copy-pasted to the background.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "avA7VbHLLO", "reviewer": "Reviewer_m2cb", "summary": "This paper focuses on zero-shot policy generalization to new environments in vision-based deep reinforcement learning. It proposes a new data augmentation method by random masking in the frequency domain, dubbed \"Spectrum Random Masking (SRM)\". SRM can be plugged into existing policy generalization algorithms like DrQ and SVEA. Experimental results are demonstrated on the Distracting DMControl benchmark, where the environment background is replaced by random color or distracting videos.", "questions": "As stated in the \"Weakness\" section above, how does the proposed method work on environments that are more challenging and realistic than the toy Distracting DMControl? For example, autonomous driving, robotics, indoor navigation, or other practically meaningful domains? ", "limitations": "The limitation that the authors listed is that the method is only applicable to image-based RL tasks and cannot incorporate state information or heavily occluded image observations. \n\nHowever, I believe that another significant limitation is the applicability of the method to meaningful tasks in the real world, rather than the artificially constructed Distracting DMControl suite. The experimental evidence is insufficient as of the current writing. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. This paper is quite easy to follow. Fig. 2 and Fig. 3 are very good visualizations to show the intuition of frequency-domain augmentation for policy learning.\n2. The proposed Spectrum Rnadom Masking method is conceptually simple and easy to understand and adopt. It is also a plug and play data augmentation module that can be integrated into many policy learning methods. The paper demonstrates experiments on DrQ+SRM and SVEA+SRM. \n\n\n# Weaknesses\n\nMy primary concern with the paper is the experiment section. The only benchmark demonstrated in the paper is Distracting DMControl, which is a set of toy environments with artificial distractions, such as random color and noisy video background. The tasks (such as Half-Cheetah, Walker-run, etc.) are not grounded in any useful real-life scenarios, and the visual distractions are not from naturally occurring sources. Furthermore, Distracting DMControl is already quite saturated, as the baselines methods can achieve close to 100% score in a significant number of tasks. \n\nSECANT [1] includes a number of visual policy generalization benchmarks that feature test-time visual appearance shifts that are representative of real-world applications, such as autonomous driving (CARLA [2]), robotics (Robosuite [3]), and indoor navigation in novel rooms (iGibson [4]). Please consider using these environments that are more realistic and challenging, or other benchmarks of similar calibre. \n\n> Reference\n* [1]  SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies. ICML 2021.\n* [2] CARLA: An Open Urban Driving Simulator. 2017.\n* [3] robosuite: A Modular Simulation Framework and Benchmark for Robot Learning. 2020.\n* [4] IGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5kZaJKefPRE", "reviewer": "Reviewer_T6K6", "summary": "This paper proposes a novel way of performing data augmentation in image-based reinforcement learning by perturbing regions in the frequency domain of the observed images. The method improves upon state-of-the-art data augmentation methods both during training and in generalization.", "questions": "- Simple data augmentation methods like cropping and flipping add almost no computational overhead in the RL training loop. How demanding is SRM in computation power?", "limitations": "- Typo in Figure 4: average performance is the last column.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The explanation and visualization of the Spectrum Random Masking method are excellent.\n- The proposed approach is a novel and effective addition to the existing data augmentation methods.\n- Ablation studies justify the specific implementation choices in the default SRM.\n- The experiments are comprehensive.\n\nWeaknesses:\n- I would love to see a real-world experiment where data augmentation plays a more important role.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept