S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning in Robotics
Authors: Samarth Sinha
Ajay Mandlekar
Animesh Garg
Venue: CoRL2021 Poster (2021)
Keywords: Offline Reinforcement Learning
Data Augmentation
Self-Supervised Learning

Abstract: Offline reinforcement learning proposes to learn policies from large collected datasets without interacting with the physical environment.  These algorithms have made it possible to learn useful skills from data that can then be deployed in the environment in real-world settings where interactions may be costly or dangerous, such as autonomous driving or factories.  However, offline agents are unable to access the environment to collect new data, and therefore are trained on a static dataset.  In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms. We then combine the best data performing augmentation scheme with a state-of-the-art Q-learning technique, and improve the function approximation of the Q-networks by smoothening out the learned state-action space. We experimentally show that using this Surprisingly Simple Self-Supervision technique in RL (S4RL), we significantly improve over the current state-of-the-art algorithms on offline robot learning environments such as MetaWorld [1] and RoboSuite [2,3], and benchmark datasets such as D4RL [4].

S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning in Robotics
Samarth Sinha Ajay Mandlekar", Animesh Garg2. Facebook Al Research, University of Toronto, Vector Institute, Stanford University, ANvidia
Abstract:
Offline reinforcement learning proposes to learn policies from large collected datasets without interacting with the physical environment. These al- gorithms have made possible learn useful skills from data that can then be deployed in the environment in real-world settings where interactions may be costly or dangerous, such as autonomous driving or factories. However, offline agents are unable to access the environment to collect new data, and therefore are trained on a static dataset. In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms. We then combine the best data performing augmentation scheme with state-of-the-art Q-learning technique, and improve the function approximation of the Q-networks by smoothening out the learned state-action space. We experimentally show that using this Surprisingly Simple Self- sufe-vupevviioo technique in RL (S4RL), we significantly improve over the current state-of of-the-art art algorithms offline robot learning environments such as MetaWorld [1] and RoboSuite [2, 3]. and benchmark datasets such as D4RI. [4].
Keywords: Offline Reinforcement Learning, Data Augmentation, Self-Supervised Learning
Introduction
In reinforcement learning (RL), agent is trained to interact with its environment and learn useful skills that help with solving the given task. However, interacting with the environment may be costly, unsafe to do from scratch in real-world scenarios such self-driving or industrial robotics. Unlike directly interacting with the environment, it is much cheaper to collect and store data such that an agent can be trained on it and learn from the past experience. In direct contrast to reinforcement learning, offline RL (or batch RL) [5] allows setup where a behavioral policy is used to collect and store the data such that a target policy can then be used to train the data, without any further interactions with the environments. The behavioral policy can range from human demonstrations. random policy in the given environment, or near-optimal policies for the task. However, learning from such demonstrations is difficult since the data does not cover the full state-action space, and naive behavioral policies will not cover the state-action distribution for an optimal policy for a task.
Current offline model-free RL algorithms include learning a Q-function [6] where a parameterized neural network is trained to learn the state-action values from data [7-12). This family of algorithms suffer from overestimating the true state-action values of data that is not in the same distribution as the offline dataset used. Along with overestimation error, another source of error is with the function approximation of the neural networks that are typically used to parameterize the Q-functions. This form of function approximation error may be of many forms, and recent methods have employed self- supervised learning, causal inference [13] and data augmentations to the image observations during training, resulting in souteoo--th-ae results for normal reinforcement learning [14-16]. However, it is unclear to how perform such augmentations from proprioceptive information and the rolc
Work primarily done when SS was at University of Toronto.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Data Augmentation
Reinforvement L Learning
Offline dataset
T
q-network
observation,s S,
8-ball
(
q-values
T.(s)
target values
target network
Figure 1: Overview of the proposed augmentation scheme. Here Ti (s.) and (2(+:) represent different augmentation schemes. We simply perform k- data augmentations to the ohservations 8t, and use that the augmented states to calculate the q-values and target values for better function approximation of the Q-network by ensuring that local perturbations in the state space have similar Q-value estimates.
of augmentation in offline RL. In this paper we investigate the role of data augmentations from proprioceptive observations focused offline RL. Data augmentation also help perform better function approximation for Q-networks since it allows to smoothen out the state space by "visiting" the local regions and ensuring the learned Q-values similar throughout, since small perturbations in the given observation should not lead to drastically different Q-values.
Our main contribution is Surprisingly Simple Self-Supervised offline RL (S4RL) algorithm that combines studying data augmentations with a simple Q-learning method to significantly improve the performance of offline RL algorithm. The proposed framework is general and can be added to number of off-the-shelf offline/batch RL algorithms for robot learning. We present comparisons our model other comparable self-supervised learning strategies and state-of-the-an offline RL methods on the D4RL benchmark dataset [4], which consists of continuous control, navigation and robotic control tasks collected from suboptimal and human agents. We further experiment with two dexterous robot manipulation environments: World [1] and RoboSuite [2, 3]. Across all evaluations, we find that our proposed state-based data augmentation for Q-learning significantly outperforms base offline RL. algorithm as well as various competitive data augmentation baselines.
2 Related Work
Offline reinforcement learning: In offline RL, static dataset of demonstrations is collected, and used train an agent for a given environment [5]. Popular actor-critic algorithms, such as Soft Actor Critic (SAC) [17], tend to perform poorly on learning from offline datasets, because they are unable to generalize to -fit--i--itiittionn (OOD) data because of an overestimation bias: where the critic overestimates the values of state-action pairs not previously encountered from the collected data which leads to a brittle policy.
The convergence properties of value functions have been proven in simple settings [18]; however learning policy using the learned value functions remains active challenge in offline RL. Simple algorithms like fitted Q-iterations have been able to learn state-action value functions, but do not work well with sub-optimal data [19, 20J. To help solve the generalization and extrapolation problem, different offline RL algorithms have proposed to use constrained policy optimization between the target policy and the behaviour policy that was used to collect the data. These constraints include using KL-divergence [10, 21), MMD [8] or f-divergence [22]. Some algorithms have also proposed to learn state- state-conditioned VAES to minimize the extrapolation error [7, 8, 23]. Recent work by Buckman et al. [24] investigated using worst-case analysis to train policy, denoted as pessinism. Conservative Q-Learning (CQL) explicitly deals with the problem of overestimation for OOD state- action distribution by pushing down the value of randomly sampled state-action distributions in the critic objective [25]. Learning a dynamies model from offline data has been promising [26. 27]. Recent work has also shown great promise using offline RL for real robotics tasks by learning policies

---

that can directly be deployed on real robots to perform complex tasks [28-30). Similar to before, such work relies on collecting large scale robotic datasets [3, 31, 321, and learning policies that can then be safely deployed after a few steps of finetuning.
Representation learning in RL: Recent work in state and pixel-based RL suggests the need to learn better representations from data usig better network architectures [33] or self-supervision [14]. Self-supervised representation learning has recently been applied learn RL agents from pixel- data. By using data augmentations and contrastive learning, Laskin et al. [14] showed significant improvements on learning from pixel data. The need for contrastive learning was simplified by RL with Augmented Data (RAD) [16] and Data- aata-Reeularized Q-learning (DrQ) [15] as they provide simpler alternative that used data augmentations without a contrastive objective. More recently, Self-Predictive Representations (SPR) optionally uses data augmentations for self-supervision [34]. The closest work to S4RL include DrQ [15] and RAD [16] in that we utilize their method for Q-learning over augmentations, but in contrast we focus specifically on offline RL for robotic tasks and propose augmentations from states; we also benchmark different self-supervision techniques from states. There has also been recent works showing the effectiveness of simple data augmentation in imitation learning [35, 36].
3 Preliminaries
In reinforcement learning, an agent interacts with an environment to learn an optimal policy. This typically framed as Markov Decision Process (MDP) which can be represented by tuple of the form (S, .., P. 7/ Po), where S is the state space, A is the action space, p( [st, (z) is the transition function given the current state and action pair, r(st) is the reward model given the state, ? is the discount factor where ? [0, 1) and po is the initial state distribution of the MDP.
policy, represented by क S + A is trained to maximize the expected cumulative discounted reward in the MDP. This can simply be formalized as: E E 7'r(s;, ((a)(s)))
Furthermore, a state-action value function, (ss ae). is the value performing a given action a, given the state st The Q-function is trained by minimizing the Bellman Error over in step called policy evaluation
Qi+1 + arg re (5444114444)) Q (1(s,,
(I)
Q
where Q, is the i-th step of policy evaluation. The policy is then trained to maximize the state-action values of performing an action a, given in step called policy improvement:
1i+1 + arg max E. [[Q(s, ((((ss)(()a)s))))
(2)
where is the i-th step of policy improvement.
Unlike traditional reinforcement learning, in offline RL, the goal is not to learn an optimal policy for the MDP, but rather to learn an optimal policy given the dataset. A behaviour policy is used to collect static dataset D which is then used to train a target policy æ. The policy improvement can now be stated as
T++1 +- arg max BEIND [(s ((a
(3)
=
where 'D is the static dataset. Since the nature of the behaviour policy y is unknown, and can be composed of one (or more) sub-optimal policies, the RL task becomes challenging. Since offline RL algorithms tend to generalize poorly to OOD data, the nature of and the optimality of is important. As shown Fu et al. [4], typical offline RL methods perform poorly when mixture of suboptimal policies or an untrained random policy is used to collect the dataset D. In deep reinforcement learning, both the policy and the value functions are parameterized using neural networks and trained using gradient descent.
3

---

5,
$
real transition
aug. transition
T,
T
e+1
maX
max
Tmay
TTSS
T
(s,,,)
mTT (s
diagram illustrating the difference between Bellman backup for offline RL versus offline RL with augmentations. Using data augmentations, are able smoothen out local state-space around observations in the dataset $t.
Figure 2: A
4 Method
In this section we will first consider a simple Q-eearnin technique based on DrQ [15], which helps combine data augmentations and learning better value functions during training by encouraging local points around the state to have similar Q-values. Then, we introduce 7 different data augmentations strategies and apply them the otherwise static dataset D improve function approximation by smoothening the state-space for the Q-network.
4.1 Self-Supervision in Offline RL
We denote a data augmentation transformation as: (()))s;) where S, ~ D, S is the augmented state, and T is stochastic transformation. After producing & ~ (s)), we need a method to incorporate the augmentation to encourage smoothness and consis- tency. For example, [14] uses contrastive learning "pull" the representations of two augmentations of the same state towards each other, while "pushing" two different state representations apart from each other. We follow similar scheme DrQ [15], where we perform multiple augmentations of a given state, and we simple average the state-action values and target values over the different augmetations of the state.
multiple augmentations of given state
1 Σ Q(T((5))s)) an) rt + 77 Σ Q(T(((++1112)))) (44)) -
min
E Re
(4)
The main difference between our proposed objective and the normal Q-learning objective is the mean over different augmentations in the second term of the equation. We augment the Bellman error in Equation 1 to simply be the mean eтror over the i different augmentations of the same state. Intuitively, this will help improve the consistency of the Q-value within some perturbation field of the current state since the Bellman backups are taken over i-different views of the same state
By simply averaging the state-action values and target values, assume that the reward function is locally smooth to small perturbations in the state. Utilizing multiple augmentations further allows the networks learn different variations of the data; we explore the role of increasing the number of different augmentations empirically in Section 5.1. We use the trained Q-networks to perform policy improvement using the objective in Equation 3, without augmentations. The benefits of the augmentations are distilled to the policy since the value functions are used to train the policy. By combining such local perturbations the states with self-supervision, we are able to learn a more robust policy that can generalize better to unseen data when deployed on robots. An overview of the proposed method is available in Figure where we disentangle the data augmentation and reinforcement learning steps to learn g-values and target values over different augmentations of the original state. Without loss of generality, to show local perturbations, we draw an e-ball over the current state st, however different versions of augmentations possible, will be discussed.

---

4.2 Data Augmentations
In computer vision research, augmentation are commonly used as a way to obtain the same datapoint from multiple viewpoints. Transformations such rotation, translations, color jitters, etc. commonly used to train neural networks. Such transformations preserve the semantics of the image after the transformation since for example: image of cat rotated 15" remains image of cat. However, when working from only proprioceptive information of an agent (for example: the joint angles and velocity information of an industrial robot), such transformations are semantically meaningless.
In offline RL, since an agent is unable to interact with the environment to collect new data samples, data augmentations serve as a simple technique that can allow the agent to do view different augmen- tations of the trajectories in the dataset. Data augmentations from states, assumes that the output of small transformation to an input state results in a physically realizable state in the environment. If the augmented state is in the domain of the state space, & E S, then we consider it valid transformation. By performing valid augmentation stratgies, we are able to artificially increase the amount of data available during training.
Even if a transformation is physically realizable in the environment, data augmentation strategy that is too aggressive may end up hurting the RL agent since the reward for the original state may not coincide with the reward obtained from the augmented state. That is r(s:) A r(s)). Therefore, one key assumption made when performing data augmentations on the states is that the reward model r(s:) smooth, that is r(s;) ss r())). Therefore the choice of ($t needs to be a local transformation to perturb the state without changing the semantics. The choice of ((5c [5]()) is more important in offline RL than in traditional RL, since in traditional RL the agent may be able to self-correct for poor choices of transformations by being able to visit those states and correct the value functions using Bellman backups. A comparison is presented in Figure 2, where we see that compared to normal offline RL, adding augmentation allows for multiple paths in the policy evaluation step using Bellman backups. By using data augmentation, we seek to exploit this smoothness property of the reward model in the MDP, which enables us artificially visit states that may be in the state space, but not in the dataset: 8t, St E S but &, & DD
Following the constraints, consider the following transformations T()((s)):
zero-mean Gaussian noise the state: & < 8t + e, where E (N(0, and a is an important hyperparameter, which we will later discuss.
2. zero-mean Uniform noise the state: &t f St + c, where e E
and a is an
important hyperparameter.
3. random amplitude scaling as in RAD [16]: s; 9t c, where e e U(a, B), which preserves the signs of the states.
4. dimension dropout which simply zeros one random dimension in the state: 8 <- 8, where is a vector of Is with one 0 randomly sampled from Bernoulli distribution. This transformation preserves all-but-one intrinsic values of the current state.
5. state-switch where we flip the value of 2 randomly selected dimensions in the state. This naive transformation will likely break the "physical realizability" assumption, since it is possible that the two randomly selected samples are semantically dissimilar properties of a robot (such as joint angle and joint velocity). To overcome this we hardcode the pairs of dimensions that can be switched for each environment (details in Appendix F).
6. state mix-up where use mixup [37] between the current and the next state: $, < As + (1 -)) where ~ Beta(o, a)) with a = 0.4 as in [37]. The next state 8c+-1 remains unchanged.
7. adversarial state training which we take the gradient with respect to the value function to obtain new state. This can be written as 5, 8, + vV,,Jo(=(ss)) where Ug is simply the policy evaluation update as in Equation 2, and e constraints the size of the gradient. The intuition behind this is choose the direction within e-ball of 8 where the Q-value deviates the most. This augmentation is intuitively the most useful in state spaces with high dimensionality where random noise may be less effective [38].
5

---

Figure 3: Illustration of the robotic environments used in the experiments. The environments range from robotic arms for difficult manipulation tasks such as opening door or placing can into a bin or dexterous manipulation of a pen to a desired configuration using robot hand.
5 Experiments
In this section, we will first describe the dataset (D4RL [4]), and some hyperparameters and informa- tion about the experimental setup. Then we will use S4RL to investigate the effect of different data augmentation strategies on the OpenAl Gym subset of D4RL tasks to investigate which augmentations useful for learning. Using the best data augmentation techniques found, compare the algorithm to other ceffssueervisio techniques that have been proposed for pixel- based RL, namely Contrastive Unsupervised Reinforcement Learning (CURL) [14], and BAC+Auto we feed the augmentations to the AutoErc to learn generalizable representations [39]. We add
(SAC+AE) where
S4RL framework two state-of-the-ard offline RI algorithms: Conservative Q-learning (CQL.) [25] and Behaviour Regularized Actor Critic (BRAC) [10] to show the effectiveness over both baseline agents to show the generality of S4RL. We add the CQL regularization to each self-supervised baseline considered, since the self-supervised baselines were considered for online RL and CQL adds a regularization term help with overestimation. We first test each baseline on the full suite of D4RL tasks [4], which includes environments such Maze environment where an ant agent must learn locomotion and navigation skills reach an environment goal which requires hierarchical control, Adroit domain which requires learning dextrous object manipulation for fine ume-ggrined control of a robotic hand [40], Kitchen environment which simulates a Franka Panda robot to perform kitchen tasks. We additionally experiment with more difficult robotic environments: Meta World [1] which uses a simulated Sawyer robot and RoboSuite [2] which uses a Franka Panda robotic arm for manipulation. We further benchmark each of the baselines on learning with limütd data in Appendix A, to investigate the effect of self-supervision when the agent has significantly less data to train from. Finally, we experiment with different ways to regularize Qneetwrkks typically used in neural network training such as Dropout and Weight Decay Appendix D and include ablation experiments investigating the number of augmentations and the role of hyperparameters in Appendix E.
Hyperparameters: For all experiments, we do not perform any hyperparamter tuning to the base CQL agent; all agents are trained using the original hyperparameters. Similar to DrQ, we use 2 augmentations for obtaining the state-action values and the target values. We use a and a value of 3 x 10-4 for the zero-mean Gaussian and Uniform noise augmentation variant, respectively. We use a value of 10-4 for e for adversarial training; an ablation on a and is available Appendix
5.1 Results
Role of Data Augmentations We first investigate different data augmentation schemes and its performance on the OpenAl Gym subset of the D4RL tasks. We compare a base CQL agent with and without diffcrent forms of augmentation strategies, and tabulate the mean normalized performance, average normalized performance over all the tasks and relative rankings in Table 1. We see that using zero-mean Gaussian noise (N), zero-mean Uniform noise (L4), state mix-up and adversarial state training consistently outperform the the baseline CQL agent, as well as different data augmentation variants. The wverage-ranking of S4RL-Ady and S4RL-N are 1.83 and 1.92 respectively, suggesting their effectiveness over wide range of task and data distributions. We also that CQL+S4RL agent is able to learn useful policies given data collected from a random policy as evidenced by the performance in "walker-random" where the base CQL agent unable to learn any meaningful policy.
We also see that the S4RL agent able to significantly outperform the baseline CQL agent on complex data distributions such as "-medium-replay" where the data collected is from all the data

---

COL
+S4RL
+S4RL
+S4RL (Amp-Scale)
+S4RI (Dim- (Dim-Drop)
+S4RL (State (iste-Switch)
+S4RI. (MixUp)
+S4RI
Task Name
Normal
(N)
(4)
(Adv)
cheetah-random
35.4
52.3
50.3
46.4
45.5
40.3
45.2
53.9
cheetah-medium
44.4
48.8
47.0
42.5
45.6
41.2
46.2
48.6
cheetah medium replay
51.4
46.2
cheetah-mediuee -endimmddddpeeee
62.4
79.0
78.5
71.2
66.1
68.3
73.1
78.1
popper-randon
10.8
10.8
10.7
10.8
9.3
9.5
11.0
10.7
hopper-medium
58.0
78.9
80.6
60.3
45.3
$4.6
79.2
81.3
hopper-medium-ree hupplly
29.5
35.4
35.6
36.8
hopper medium expert
111.0
115.2
112.3
102.6
98.3
108.6
113.5
117.9
walker-random
7.0
24.9
20.5
6.9
3.2
9.9
10.2
25.1
walker-medium
79.2
93.6
93,9
81.6
65.3
65.3
88.6
93.1
walker-medium--re medium-replay
21.1
30.3
27.6
35.0
walker medium--eeeeee expert
98.7
112.2
104.1
100.2
86.8
103.8
104.3
107.1
average-score
49.96
60.57
39.65
51.60
45.31
47.73
56.73
61.6
average-ranking
1.92
6.42
3.17
Table I: Investigating the importance of different augmentation schemes in offline RI. We train baseline CQL agent, and sulf-super CQL SQL+S4RI agents using different augmentation schemes, and report the mean normalized scores over random seeds. We report the baseline CQL results directly from Fu et al. [4]. To make the table easier to read we color the best algorithm on a task in blue, the second best brown, and the third best in green. We also report the average-score and average-ranking found using that augmentation scheme.
CQL
BRAC-V
+CURL
+S4RL (N)
+S4RL (Mix-Up)
Domain
Task Name
Normal
(N)
(V)
(Mix-Up)
(Adv)
Normal
(N)
(Adv)
unimaze-umaze
nntmaae-unte -inere--iveeeee
84.0
81.4
85.
87.8
86.9
88.0
70.0
81.3
65.4
80,9
antmaze medium: play
61.2
60.9
62.1
61.9
61,3
61.6
0.0
0.0
0.0
0.0
45.4
untmaze large play
15.8
12.3
15.2
24.4
16.7
25.1
0.0
0.0
0.0
0.0
untmare- mimaae-llngg-dee linge-diverse
14.9
8.3
17.3
27.0
15.9
26.2
0.0
0.0
0.0
0.0
chectabrcandomm
35.4
43.1
35.7
52.3
-15.2
53.9
31.2
35.6
34.3
36.1
cheetab medium
44.4
44.8
45.6
48.8
46.2
48.6
46.3
46.1
46.0
medlul--mdium--epplay
62.4
65.6
70.7
79.0
75.1
78.
41.9
$2.1
46.7
53.4
hopper-random
10.8
10.8
10.8
10.8
11.0
10.7
12.2
12.9
10.9
12.1
hopper-medium
Gym
hopper-medium medium-repppe meddim--iiumyylla
29.5
30.1
34.8
35.4
35.6
36.8
0.6
L.1
0.9
0.4
Popper-medium-ee
111.0
109.7
114.3
113.5
112.3
117.9
0.8
$9.0
49.5
55.3
Tallerr-anddoe
7.0
6.6
24.9
10.2
25.1
2.4
14.1
walker-medium
79.2
81.3
80.8
93.6
88.6
81.1
80.9
86.7
walkerr-mddimme medium-replay
21.1
24.5
26.4
30.3
27.6
35.0
0.9
1.1
0.9
1.1
walker medium expert
98.7
103.0
99.6
112.2
104.3
107.1
90.4
89.7
94.5
pen-human
37.5
33.4
39.7
44.4
41.6
$1.2
0.6
11.3
0.8
14.1
pen-cloned
hammer-human
6.1
5.9
6.3
0.2
3.3
5.4
Adroit
doce hoce-human
hammer-cloced
2.1
10.3
1.9
14.3
2.1
27.0
2,7
16.7
2.4
35.3
2,9
0.3
0.1
0.2
0.3
1.2
4.2
door-clened
0.4
2.1
0.3
-0.1
0.5
relocate- human
0.2
0.2
0.2
0.2
0.2
-0.3
0.1
0.1
0.2
sclocste-cloned
-0.1
-0.1
-0.1
-0.1
-0.3
-0.1
-0,1
-0,1
Franka
Nitchen-complete
43.8
54.4
42.1
77.1
60.4
88.1
0.0
0.0
0.0
12.3
kitchen-purtial
$9.6
74,8
59.0
13.9
Table 2: Full set of experiments on the D4RL suite of tasks. We perform experiments over different domains and many different environments and data distributions for each environment. We compare two S4RL variants (N and Mix-Up) with different popular unsupervised representation learning methods and the base CQL offline RL agent. train the CURL and VAE variants with Gaussian additive noise variant that was empirically the most effective. We report the mean normalized performance over random seeds.
collected while training policy in the environment. Therefore the data split consists of data that ranges from a completely untrained policy (random policy), to a "medium" trained policy.
It is also important to look at the different augmentation techniques that do not help policy learning. Techniques that hurt the performance and perform worse than the baseline CQL agent include Dntension-Droponee and State-Switch. Both techniques are inspired by popular computer vision data augmentation algorithms, mainly MixMatch [41] and CutMix [42]. Since both techniques perform element- wiien operations, it is likely that they omit important information about the state of the robot such as joint velocity. Without such information. reason about the environment which results in poor value estimates.
possible that the value function unable to
7

---

push
COL+VAE
door-close
CDL+ SOL+SARL-N
COL S4RL-AdV
pick-place-can
COL-S4
CQL CQU-S4ILLAd
sweep-into
reach-wall
block-lift
reach
20
40
60
80
10
20
30
40
50
% Goal Reached
% Goal Reached
(b) RoboSuite Environments
(a) MetaWorld Environments
Figure 4: Results on Dexterous Manipulation Environments from MetaWorld [1] and RoboSuite [2, 3J. We report the fraction (as %) of goals reached in each of the environments during evaluation. Notably with fixed amount data, S4RL augmentation improves policy success rate over base CQL by approx. 20% across different tasks.
Comparison to other self-supervision techniques Following insights from Table 1, we choose three data augmentation approaches, namely S4RI.N, S4RI--MixUp and S4RL--Adv, to compare to other self-supervision techniques. We present the results over all the different tasks in D4RL in Table 2 using base CQL [25] and BRAC agents [10]. Despite its simplicity, we continuously observe that +S4RL-N and +S4RL-adv agents are able significantly outperform the baselines almost all tasks, and is comparable to the best in the others for both offline RL algorithms. We specifically note that +S4RL-Adv significantly outperforms all other baselines when the stte--dimensionaliyy is high, such as the dexterous manipulation environment of Adriot and on Franka robot. challenging environments that require hierarchical control such the AntMaze environments, S4RL agent continues to be the best performing agent. Interestingly, on certain environments namely "pen-cloned", Open-human"", "hammer-human" and idoor-human" the BRAC+S4RI. agent is able to learn some useful skills, whereas the base CQL agent performs poorly. This further highlights the usefulness and generality of performing state-based augmentations for offline reinforcement learning.
that the
Dexterous Manipulation Environments Finally, we perform more experiments in the robotic domain using the MetaWorld [1] and RoboSuite environments on difficult manipulation tasks. For MetaWorld, we train a SAC agent [17] for 1M steps and collect 1000 trajectories of 200 episode length. For RoboSuite, we collect data at 3 different instances during training make the data distribution more complex. More details on the data collection is available in Appendix I. The results are presented in Figure 4. Similar before we see significant improvements over the baseline in each of the 7 environments over both domains. The proposed S4RL-Adv agent is the only agent that able to outperform the trained behaviour policy on of the World tasks, while continuing to significantly outperform the baselines by as much as 3 times, as in the case of "push-v1" and ppick-and-place place-can". Full numerical results can be found in Appendix
Conclusion
In this paper, we present S4RL: Surprisingly Simple Self-Supervised offline RL method that uses data augmentations to improve the function approximation for Q-learning algorithms offline RL. S4RL offers simplicity and ease of implementation, and can be added to any offline agent that requries Q-learning. We first compare and benchmark the effectiveness of different data augmentation strategies from states in offline RL and then use the insights to compare against different self-supervised representation learning algorithms that have been proposed for pixel-based online RL. We observe significant performance gains over the benchmark D4RL dataset [4] and on more dexterous robot manipulation environments. We observe significant performance gains which make offline policy learning increasingly more competitive and help robot learning from only past data. Interesting future extension of this work can seek to use the suggested augmentation schemes to build better supervised learning algorithms.

---

Acknowledgments
Animesh Garg is supported by CIFAR AI Chair, NSERC Discovery Grant EGPIN-2021-044 and DGECR-202 (GEECR2021-00068, University of Toronto Engineering Xseed Grants. Ajay Mandlekar acknowl- edges the support of the Department of Defense (DoD) through the NDSEG program. The authors would also thank Vector Institute for computing support. The authors would also like to thank Aviral Kumar for his help setting up the CQL experiments and for helpful discussions regarding the paper.
References
[1] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv: 10897, 2019.
[2] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martin, robosuite: modular simulation framework and benchmark for robot learning. arXiv preprint 2tt::20009.12993. 2020.
[3] A. Mandlekar, Y. Zhu, A. Garg. J. Booher, M. Spero, A. Tung. J. Gao, J. Emmons, A. Gupta, E. Orbay, S. Savarese, and L. Fei-Fei. ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation. In Conference on Rohot learning (CoRLJ, 2018.
[4] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint urXiv:2004.072) 2020.
[5] S. Levine. A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspec- tives on open problems, arXiv preprint arXiv:2005.01643 2020.
[6] C. J. Watkins and P. Dayam. Q-learning. Machine learning, ((3-4):279-297 1992.
[7] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062. PMI.R, 2019.
[8] A. Kumar, Fu, G. Tucker, and Levine. Stabilizing off- off-policy q-learning via bootstrapping emror reduction. arXiv preprint 19X,099906.00999 2019.
[9] C. Gulcehre, Z. Wang, A. Novikov, T. L. Paine, S. G. Colmenarejo. K. Zolna, R. Agarwal, J. Merel, D. Mankowitz, C. Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning, arXiv preprint arXv::2006. 13888, 2020.
[10] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning, arXiv preprint arXiv: 191111911..1361,, 2019.
[11] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[12] A. Mandlckar, Ramos, B. Boots, S. Savarese, L. Fei-Fei, A. Garg, and D. Fox. Iris: Implicit reinforcement without intcraction at scale for learning control from offline robot manipulation data. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
[13] S. Pitis, E. Creager, and A. Garg. Counterfactual data augmentation using locally factored dynamics. In Advances in Neural Information Processing Systems (NeurIPS). volume 33, 2020.
[14] M. Laskin, A. Srinivas, and P. Abbeel, Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference Machine Learning, pages
PMLR, 2020.
[15] I. Kostrikow, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement lcarning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
[16] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data, arXiv preprint 20:4 2020.
[17] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint 18Xiv:180..0/240 2018.
[18] D. Ormoneit and S. Sen. Kernel-based reinforcement learning. Muchine learning, 19(2):161-178, 2002.
[19] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503 0003-55,222 2005.

---

[20] M. Riedmiller. Neural fitted q iteration-firs first experiences with a data efficient neural reinforcement learning method. In European Conference on Machine Learning, pages 317-328. Springer, 2005.
[21] N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard. Way off-policy hatch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: / 1907.0046,, 2019.
[22] S. Sinha, J. Song, A. Garg, and S. Ermon. Experience replay with likelihood-free importance weights. arXiv preprint arXiv:2006. 13,69, 2020.
[23] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offiine reinforcement learning. arXiv preprint arXiv:2010. 13666 2020.
[24] J. Buckman, C. Gelada, and M. G. Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.0677 2020.
[25] A. Kumar, A. Zhou, G. Tucker, and Levine. Conservative q g-learning for offline reinforcement learning. arXiv preprint arXiv:2006 04779, 2020.
[26] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
[27] R. Kidambi, A. Rajeswaran, P. Netrapalli, and Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05553 2020.
[28] A. Singh, Yu, J. Yang. Zhang. A. Kumar, and S. Levine. Cog: Counecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010. 145000 2020.
[29] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zoina, Y. Aytar, D. Budden, Vecerik, et al. A framework for data-driven robotics. arXiv preprint 1rXiv:1909.12230 2019.
[30] J. Jin, D. Graves, C. Haigh, J. Luo, and M. Jagersand. Offline learning of counterfactual perception as prediction for real-world robotic reinforeement learning. arXiv preprint arXiv:201 1.008577, 2020.
[31] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg. S. Savarese, and L. Fei-Fei. Scaling robot supervision hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In IEEE International Conference on Intelligent Robots and Systems (IROS), 2019.
[32] S. Dasari, F. Ebert, S. Tian, Nair, B. Bucher, K. Schmeckpeper, Singh, S. Levine, and Finn. Robonet: Large-scale multi-robot learning. arXiv preprint 1rXiv:1910.12215. 2019.
[33] S. Sinha, H. Bharadhwaj, A. Srinivas, and A. Garg. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163. 2020.
[34] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and Bachman. Data-efficient reinforcement learning with momentum predictive representations. arXiv preprint arXiv:2007.0592 2020.
[35] P. Florence, L. Manuelli, and R. Tedrake. Self supervup correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-499, 2019.
[36] M. Laskey, J. Lee, R. Fox, A. Dragan, and K. Goldberg. Dart: Noise injection for robust imitation learning. In Conference robot learning, pages 143-156. PMLR, 2017.
[37] H. Zhang, M. Cisse, N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 171000/9 2017.
[38] A. Mandlekar, Y. Zhu, A. Garg, L. Fei-Fei, and S. Savarese. Adversarially robust policy learning: Active construction of plysicall perturbations. In 20/7 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pages 1032-39399 IEEE, 2017.
[39] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pincau, and R. Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint 191001191, 2019.
[40] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint 1709, 2017.
[41] D. Berthelot, N. Carlini, 1. Goodfellow, N. Papernot, A. Oliver, and C. Raffel. Mixmatch: A holistic approach semi-super learning. arXiv preprint 105244990, 2019.
10

---

[42] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6023-6032, 2019.
[43] N. Srivastava, G. Hinton, A. Krizhevsky. Sutskever, and R. Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955)) 2014.
[44] T. Haarnoja, A. Zhou, Abbeel, Levine. Soft cctor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, arXiv preprint arXiv:1801.01290, 2018.


Reviews:
[{"review_id": "W5aVUf0FGq_", "reviewer": "Reviewer_TxK8", "summary": "This paper addresses the problem of improving the performance of\noffline reinforcement learning (RL) through data augmentation. In\nparticular, the authors discuss ways of adding random perturbations to\nthe dataset in a manner that preserves the physical meaning of the\ndata (e.g., realistic action-reward pairs). The authors discuss a\nnumber of data augmentation methods and present extensive evaluation\nover multiple datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Offline RL is an important problem, especially in sim-to-real tasks\nwhere performing online RL might be difficult and time-consuming. As\nsuch, addressing the generalization issues with offline RL is\ncertainly an interesting and important research direction.\n\nThe paper is well written, and the problem is clear. The idea of using\ndata augmentation is a nice and simple way to improve the performance\nof modern classifiers, especially in settings where training data is\nnot overly abundant.\n\nThe results look promising. It would be interesting to see if the\nproposed method can be used on benchmarks where the observations are\nimages or other high-dimensional sensor data. Performing physically\nconsistent data augmentation in such settings would be more\nchallenging but potentially more useful as well.\n\nOne comment I have is that authors mention in a number of places in\nthe paper that the proposed approach reduces overfitting, yet this\nclaim is not properly evaluated. Overfitting is indeed a major concern\nin (offline) RL since the learned policy is very much dependent on the\ntraining set (presumably collected using other predefined\npolicies). One way to evaluate generalization error would be to add\nprocess/measurement noise to the various RL environments and perform a\ncomparison in that setting as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper tackles a challenging problem, and the results look promising. The proposed data augmentation method makes sense, as it has been applied successfully in supervised learning settings.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "N/A", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "VTsjF8bd-Kc", "reviewer": "Reviewer_vBpT", "summary": "This paper aims to tackle the problem of offline reinforcement learning and presents an extension to existing methods via data augmentations. The method, S4RL: a Surprisingly Simple Self-Supervised offline RL method, assumes that the reward function is locally smooth and employs a range of data augmentation mechanisms to expand the offline dataset. The authors hope to improve the generalization ability of the RL agent in regions not well-supported by the original offline data.\n\nThe authors compare and benchmark the effectiveness of seven different data augmentation strategies from states in offline RL and then use the best-performing ones to compare against different self-supervised representation learning algorithms that have been proposed for pixel-based online RL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strength\n\nThe proposed method, S4RL, is simple and easy to implement, making it compatible with many offline reinforcement learning algorithms that require Q-learning.\n\nThe authors show that the augmentation techniques can improve the performance over existing baselines.\n\n\n=== Weakness\n\nThis paper is not particularly novel. Augmentation is a widely-used technique in a range of applications. It is not surprising that applying data augmentation to offline RL can improve the performance in some scenarios. It would improve the significance of this paper if the authors can show some observations that are specific to offline RL. For example, what are some of offline RL's properties that make it more suitable for some data augmentation techniques and not the others?\n\nI also have concerns regarding the applicability of the proposed method. The authors make an assumption that the reward has to be smooth. However, many real-world tasks can involve discontinuous events like contacts. Considering grasping or pushing an object, the reward can be sharply different when the end effector is off by just a little bit. In cases where the end effector is close to the table, data augmentation could also lead to physically-infeasible states and incorrectly assign high value to these states. The authors may consider providing more theoretical justifications on where and when we should include the augmentation and to which extent does it help.\n\nI also have questions regarding the comparison with the baselines. CURL [1] and CQL [2] were designed or at least contain experimental results where the inputs to the agent are visual observations. What are the inputs to S4RL in each experimenting environment? Are they images or state information? The authors should include more detailed descriptions of the testing environments for the readers to better understand the context of the reported performance scores.\n\nRelated to my previous point, if the inputs to S4RL contain high-dimensional visual observations, why do the authors only apply augmentations to the state? What about the wide range of augmentation techniques that can be applied to images? If the inputs to S4RL are low-dimensional states, comparing it with techniques like CURL and CQL does not seem particularly convincing to me, as they are designed and have shown results in more challenging scenarios than this paper.\n\nTypo: the symbol for the dataset in equation 3 is not consistent with the text description.\n\n[1] Srinivas et al., CURL: Contrastive Unsupervised Representations for Reinforcement Learning\n[2] Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning\n\n========\n\nPost-rebuttal\n\nI have read the reviews from other reviewers, the rebuttal from the authors, and the discussions. I appreciate the authors' efforts in providing detailed responses and the additional experiments, which have addressed my major concerns. The detailed comments from Reviewer oh9V and the revision from the authors also helped make the statements in the paper more precise at describing the contributions and related concepts.\n\nI'm now supportive of this paper to present at the conference and have updated my score from Weak Reject to Weak Accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I have concerns regarding the novelty and the applicability of the underlying assumptions of this paper. I also find the comparison with the baselines not particularly fair, as the baselines are designed to process high-dimensional visual observations, and this paper, S4RL, only seems to apply augmentation to the low-dimensional states.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see the weakness section for detailed descriptions of the issues.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "OoqQQyMfQ0b", "reviewer": "Reviewer_bi2U", "summary": "This paper takes inspiration from recent improvements on image-base RL through augmentations and proposes to apply augmentations to state input as a form of self-supervision. The paper analyzes a range of different augmentations for offline RL by combining them with SAC (a state of the art RL method).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written, presents an idea that makes a lot of sense and appears simple in hindsight (I mean this as a big plus). The creative and original contributions are the augmentations considered here (Section 4.2).\n\nThe weaknesses of the paper are a few missing clarifications (see below) especially regarding the experiments and the limitation of the experiments to the offline setting – it would be interesting to see if the method produces similar improvements in online RL, or if it particularly fits the challenges of offline RL as suggested (but never shown) in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper appears correct and contains results that should be significant to the robot learning community. The paper is generally very well written and structured. If the issues regarding clarity and the experimental section are resolved, this can make a good CoRL paper.\n\nEDIT: Issues addressed in rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "L41,43,...: “proprioceptive observations” is not really true for the inputs used in all experiments as some of them will also provide the poses of objects to be manipulated (see Fig. 7).\n\nL207: “we compare the algorithm to other self-supervision techniques that have been proposed for pixel-based RL”. This statement confuses me as it suggests that the proposed method works from pixel input where I had assumed that it is working from state input. Now, if the proposed method is actually working from pixel input (maybe applying an encoder to produce a low-dim. state and the augmenting that as described), this needs to be explained better. And if this is not the case and the proposed method actually works on state input, then comparing to pixel-based baselines should be done very very carefully as this is generally not a fair comparison.\n\nTable 2: Here the question is now if “Normal” refers to learning from state or from pixels. And then +CURL/VAE or +S4RL cannot both be extensions on top of “normal” if one works from pixels and one works from state input. In this case, we need two “normal” versions and it should be made very clear in the table which results are based on pixel or state input (which is currently not indicated at all).\n\nTable 1 & 2: In both tables, I have trouble finding definitions for the meaning of the different suffixes to the task names, e.g. “-medium”, “-medium-expert”, “-unmaze”, “-unmaze-diverse”, “medium-play”, “-human”, “cloned”. I would suggest to clearly explain all of these in the captions of these tables and potentially to split up the task name column into “task” and “data generation”.\n\nRight now, the method is only evaluated in an offline setting and that is understandable if the authors are interested in that setting. But it would make this paper much stronger if the interactions of the online / offline setting and the advantage of using the proposed augmentations were evaluated. This way, we could test whether S4RL is especially suited for the offline setting as suggested e.g in L121-124."}}, {"review_id": "IBI2P3J7i4a", "reviewer": "Reviewer_oh9V", "summary": "This paper studies the effect of data augmentation on offline RL algorithms.  Essentially it applies the DrQ data augmentation technique to an offline-RL, proprioceptive-only-state setting.  Specifically, it measures the effect of up to 7 different data augmentations, across 3 different simulated task domains (D4RL, MetaWorld, and RoboSuite), and to two different offline-RL algorithms: CQL and BRAC.  The primary empirical results are that the noisy data augmentation can improve closed-loop evaluations of the policies, with the two most effective methods being either adding noise from (i) a zero-mean gaussian or (ii) adversarially using the Q-function gradients.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To best organize my feedback and analysis, I’ve created three subsections: (i) a summary of strengths, weaknesses, and what I learned, (ii) a list of objective strengths, and (iii) a list objective critiques and room for improvement. \n\n(i) *Summary of strengths, weaknesses, and what I learned*:\n\nMy summary of the strengths and weaknesses are that there seems to be a main strength, and several main weaknesses.  In terms of strengths, the paper’s main strength is that the quantitative improvement shown by the proposed method can be large.  This includes all three of the MetaWorld, RoboSuite, and D4RL results.  In terms of weaknesses, there are several: significant claims are not supported by data, in some cases the data supports the opposite conclusions arrived at by the authors, the scope of novelty is limited, and the consideration of real-world robotics impact isn’t considered.  My subjective recommendation for the paper will be discussed in the Summary of Recommendation, but in the next two subsections, I will try to keep the analysis objective.\n\nI would also like to emphasize to the authors that although I’ll enumerate a large list of critiques, they are on to some interesting results, and I applaud their work so far!  Their work has shown the high potential that simple data augmentation can have on making the most of offline robotics data.  In particular their use of the adversarial augmentation is a nice connection between a couple different works, and opens interesting questions for further work. I hope my feedback encourages them to revise their work further, and I do think this line of work could have an impact on the field, even if it needs significant revision first.\n\n(ii) *Objective strengths*\n\n1. Within the scope of the results reported within this paper, the direct comparisons show that the data augmentation methods (particularly Gaussian noise and adversarial training) improve the closed-loop evaluation results of both CQL and BRAC on the evaluated simulated tasks.\n2. Comparing as well against numbers from other papers, it seems that for many of the D4RL tasks these “CQL+S4RL” numbers would be considered state-of-the-art quantitative numbers on D4RL which is a commonly used offline RL simulated benchmark.  \n\n(ii) *Objective critiques and room for improvement in revision*\n\n1. A central claim of the paper is that the presented method reduces overfitting, but there is insufficient data to support this claim.   “Preventing overfitting” is a specific thing, which means that the model fits the training data much better than the test data. This was not measured in the presented experiments.  No analysis of fitting the training data was presented, so overfitting can’t be established.  Since the evaluations all use the metric of closed-loop evaluations, the performance increases that are seen could actually be from a phenomenon besides overfitting, for example having better closed-loop stability.  The models with the data augmentations could even overfit *more*, but have better closed-loop stability.  The authors could distinguish these effects with more experimentation.  The results of such experiments might end up changing their story considerably.\n2. To clarify, this overfitting claim is a central claim of the paper that is used both to describe a limitation of prior work  (in abstract: “However, current algorithms overfit to the dataset they are trained on”; in Introduction “neural networks may overfit the data, further resulting in poor generalization…”) and as the key effect of the presented method (end of intro Method paragraph, Section 4: “Then we introduce 7 data augmentations… to reduce overfitting and improve function approximation by smoothening the state-space.”; second paragraph Section 4.2: “by performing valid augmentation… thereby reducing chances of the overfitting to the data”.)\n3. The data potentially supports the opposite of the overfitting claims: Appendix C shows that Dropout makes the results worse.  If Dropout, which is well known to prevent overfitting, makes the results worse, this may support the conclusion that the phenomenon does not have to do with overfitting.  This is not discussed.  Instead, the reader is left to assume that the presented data augmentation strategy can reduce overfitting, but Dropout can not.\n4. A key statement about novelty is incorrect, and reduces the scope of the novelty claim: “however, it is unclear to how perform [sic] such augmentation from proprioceptive information”. This statement is incorrect because one of the cited references, specifically [15], “Reinforcement Learning with Augmented Data”, does show both methods and results for proprioceptive inputs: see the end of their Section 4 “Extension to state-based inputs” and the results on OpenAI Gym benchmark.\n5. Some missing related work discussion would be around noise/augmentation in imitation methods, including http://proceedings.mlr.press/v78/laskey17a/laskey17a.pdf (online adding noise to the demonstrator actions) and https://arxiv.org/pdf/1909.06933.pdf, (Section IVC, offline adding gaussian noise to state).\n6. More on novelty claims, the paper says “The main difference between our proposed objective and the normal Q-learning objective is the mean over i different augmentations in the second term of the equation.”  But this makes it seem as though averaging Bellman updates over various augmentations is a contribution of the paper.  This was a contribution of DrQ, see for example Equation 1 and/or Algorithm 1 in [14].\n7. “Self-supervision”, which is a central word in the title and in the manuscript overall, is a misnomer and is not an accurate characterization. The presented method is a data augmentation method, not a self-supervision method.  This distinction is understood in the prior works, including both [14] “Image augmentation is all you need” and [15] “Reinforcement learning with augmented data”, which both accurately characterize their methods as data augmentation methods.  This is in contrast with [13] “Contrastive Unsupervised Representations for Reinforcement Learning”, which can be called an unsupervised/self-supervised method, since the formulation involves an actual self-supervised representation.  There is a relationship between data augmentation and many self-supervision methods (i.e. many of the self-supervised methods in computer vision such as SimCLR use data augmentation as a key component), but those other methods have the goal of recovering some particular self-supervised representation.  Instead, the presented paper is strictly a data augmentation method applied to the supervised offline RL objective, with no self-supervised representations used.\n8. There are also various incorrect statements discussing the central topic of the paper, which is data augmentation and its effect in the studied setting:\n    - The end of the second-to-last paragraph in the Introduction states: “since small perturbations in the given observation should not lead to drastically different Q-values.”  This is not in general a true statement.  It is possible for world dynamics to be sharp or discontinuous such that slightly different states of the world may lead to vastly different Q-values for an optimal policy.  What the authors could say instead is: “Our approach makes the assumption that small perturbations in the given observation do not lead to drastically different Q-values.  While this is not generally true, our results show...\"\n    - The start of Section 4.2 states, “In computer vision research, augmentations are commonly used as a way to obtain the same datapoint from multiple viewpoints.”  This is not true… obtaining the same datapoint from multiple viewpoints might be called some type of data fusion, but not augmentation?\n    - Also near the start of Section 4.2: “Such transformations preserve the semantics of the image after the transformation.” This is not generally true, and depends on which semantics are being discussed.  For example one can apply a left/right flip (mirror) augmentation often when doing classification computer vision, but one would not usually do this for training a robot policy that for example might want to learn to put object A to the left of object B.\n    - Also near the start of Section 4.2: “However when working from only proprioceptive information of an agent…, such transformations are semantically meaningless.”  Since semantics are a human-defined concept up to interpretation, one cannot generally say that transformations do not change semantics.  In fact, the authors agree with this, two paragraphs later.\n    - Second paragraph of Section 4.2: “In offline RL, data augmentations serve as a simple technique that can allow the agent to do local exploration from the trajectories in the dataset.”  Although we can interpret the augmentations as something similar to imagining local explorations, it’s not true to say that the agent actually does local exploration.  Exploration would imply receiving new observations/rewards from the environment.  The authors have more accurate language to describe this phenomenon later, saying “...enables us to artificially visit states that may be in the state space.”\n    - Third paragraph of Section 4.2: “Therefore the choice of \\mathcal{T}() needs to be a local transformation to perturb the state without changing the semantics.”  Two paragraphs ago it was stated that transformations are semantically meaningless, but now it says that only small transformations avoid changing the semantics.  This is inconsistent.  Also, it may be helpful in revision for the authors to define their definition of semantics, or perhaps consider removing the phrase and substituting something else that is defined.\n9. Although the presented method shows increased performance on the simulated benchmarks, it is not discussed whether the discussed data augmentations would in practice improve real-world robotics results.  As shown in Table 5 in Appendix D, the preferred Gaussian noise is on the order of 1e-4.  The scale of this value is not discussed, but in general the D4RL environments have observations and actions that are normalized to [-1,1] range.  So 1e-4 is 1/20,000th of the observation range.  This might be on the order of natural noise observed by real-world robot sensors, for example the accuracy of a cartesian end-effector state estimate estimated by forward kinematics from 6 noisy encoders.  Point being that perhaps natural noise on real-world robots already provides the benefits observed by the method in simulated scenarios.  This question might be investigated further in simulation, through simulating proprioceptive sensor noise.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think it is clear that the paper needs considerable revision, but the question is whether that amount of revision is small enough such that it could be accepted to this conference, or if it is considerably large enough such that it would be best for the work to be revised, more experiments to be run, and submitted to a future conference.  In general I can be very supportive of papers that have a strong empirical simulated result, even if there is no theory or real-world result.  Usually though, one would expect a paper which focuses on empirical results to provide an accurate interpretation and framing of those results.   In my opinion, the scope of revisions and open questions (see my list in previous section) is large enough such that it seems best to revise and submit to a future conference.\n\n*Update after rebuttal:*\nI think the paper is more clear now.  I will assume the authors will be willing to make the two edits below, which I think are required for truthfulness.\n\n#1, Line 46, key moment of intro:\n- current: \"Our main contribution is a Surprisingly Simple Self-Supervised offline RL (S4RL) algorithm that combines studying data augmentations with a simple Q-learning method to significantly improve the performance of offline RL algorithm. \"\n- critiques of the current statement:\n    - I don't think they contributed anything that can be called an algorithm? As I pointed out in my initial review, Equation 4 is directly from DrQ.\n    - \"Combining data augmentations with a simple Q-learning method\" somewhat implies that they have a new Q-learning method, which they don't.\n- suggested instead: \"Our main contribution is in showing the effectiveness that simple data augmentations can have in improving existing offline RL methods.\"\n\n#2, Line 88, key discussion of related work:\n\n- current: \"The closest work to S4RL include DrQ [14] and RAD [15] in that we utilize their method for Q-learning over augmentations, but in contrast we focus specifically on offline RL for robotic tasks and propose augmentations from states.\"\n- critical error of the current statement:\n    - RAD also proposed augmentations on states.\n- suggested instead: \"\"The closest works to S4RL include DrQ [14] and RAD [15] in that we utilize their method for Q-learning over augmentations for online RL, but in contrast we focus specifically on offline RL for robotic tasks, and <insert some statement about the thoroughness of results>\"\n\nIn contrast I think these statements at other key moments are more truthful and clear:\n\n- Abstract: \"In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms.\"\n- Conclusion: \"In this paper, we present S4RL: a Surprisingly Simple Self-Supervised offline RL method that uses data augmentations to improve the function approximation for Q-learning algorithms in offline RL.\"\n    - nit: they haven't actually shown that they improve function approximation. They've just shown that the methods work better. But I consider this to be a minor grievance compared to #1 and #2 above.\n\nIf they could correct those statements I would be supportive of having the paper at the conference.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See Strengths and Weaknesses, I put a list in the *(iii) Objective critiques and room for improvement in revision.*", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After an extensive discussion, the reviewers changed their scores to indicate that the paper should be published at CoRL. I would like to encourage the authors to carefully analyze the reviews and address the reviewers' points (especially regarding some of the wording pointed out by Reviewer oh9V) in the final version of the paper.

Decision: Accept (Poster)