Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper aims to tackle the problem of offline reinforcement learning and presents an extension to existing methods via data augmentations. The method, S4RL: a Surprisingly Simple Self-Supervised offline RL method, assumes that the reward function is locally smooth and employs a range of data augmentation mechanisms to expand the offline dataset. The authors hope to improve the generalization ability of the RL agent in regions not well-supported by the original offline data.

The authors compare and benchmark the effectiveness of seven different data augmentation strategies from states in offline RL and then use the best-performing ones to compare against different self-supervised representation learning algorithms that have been proposed for pixel-based online RL.
strengths_and_weaknesses: === Strength

The proposed method, S4RL, is simple and easy to implement, making it compatible with many offline reinforcement learning algorithms that require Q-learning.

The authors show that the augmentation techniques can improve the performance over existing baselines.


=== Weakness

This paper is not particularly novel. Augmentation is a widely-used technique in a range of applications. It is not surprising that applying data augmentation to offline RL can improve the performance in some scenarios. It would improve the significance of this paper if the authors can show some observations that are specific to offline RL. For example, what are some of offline RL's properties that make it more suitable for some data augmentation techniques and not the others?

I also have concerns regarding the applicability of the proposed method. The authors make an assumption that the reward has to be smooth. However, many real-world tasks can involve discontinuous events like contacts. Considering grasping or pushing an object, the reward can be sharply different when the end effector is off by just a little bit. In cases where the end effector is close to the table, data augmentation could also lead to physically-infeasible states and incorrectly assign high value to these states. The authors may consider providing more theoretical justifications on where and when we should include the augmentation and to which extent does it help.

I also have questions regarding the comparison with the baselines. CURL [1] and CQL [2] were designed or at least contain experimental results where the inputs to the agent are visual observations. What are the inputs to S4RL in each experimenting environment? Are they images or state information? The authors should include more detailed descriptions of the testing environments for the readers to better understand the context of the reported performance scores.

Related to my previous point, if the inputs to S4RL contain high-dimensional visual observations, why do the authors only apply augmentations to the state? What about the wide range of augmentation techniques that can be applied to images? If the inputs to S4RL are low-dimensional states, comparing it with techniques like CURL and CQL does not seem particularly convincing to me, as they are designed and have shown results in more challenging scenarios than this paper.

Typo: the symbol for the dataset in equation 3 is not consistent with the text description.

[1] Srinivas et al., CURL: Contrastive Unsupervised Representations for Reinforcement Learning
[2] Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning

========

Post-rebuttal

I have read the reviews from other reviewers, the rebuttal from the authors, and the discussions. I appreciate the authors' efforts in providing detailed responses and the additional experiments, which have addressed my major concerns. The detailed comments from Reviewer oh9V and the revision from the authors also helped make the statements in the paper more precise at describing the contributions and related concepts.

I'm now supportive of this paper to present at the conference and have updated my score from Weak Reject to Weak Accept.
summary_of_recommendation: I have concerns regarding the novelty and the applicability of the underlying assumptions of this paper. I also find the comparison with the baselines not particularly fair, as the baselines are designed to process high-dimensional visual observations, and this paper, S4RL, only seems to apply augmentation to the low-dimensional states.
issues: Please see the weakness section for detailed descriptions of the issues.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper takes inspiration from recent improvements on image-base RL through augmentations and proposes to apply augmentations to state input as a form of self-supervision. The paper analyzes a range of different augmentations for offline RL by combining them with SAC (a state of the art RL method).
strengths_and_weaknesses: The paper is clearly written, presents an idea that makes a lot of sense and appears simple in hindsight (I mean this as a big plus). The creative and original contributions are the augmentations considered here (Section 4.2).

The weaknesses of the paper are a few missing clarifications (see below) especially regarding the experiments and the limitation of the experiments to the offline setting – it would be interesting to see if the method produces similar improvements in online RL, or if it particularly fits the challenges of offline RL as suggested (but never shown) in the paper.
summary_of_recommendation: This paper appears correct and contains results that should be significant to the robot learning community. The paper is generally very well written and structured. If the issues regarding clarity and the experimental section are resolved, this can make a good CoRL paper.

EDIT: Issues addressed in rebuttal.
issues: L41,43,...: “proprioceptive observations” is not really true for the inputs used in all experiments as some of them will also provide the poses of objects to be manipulated (see Fig. 7).

L207: “we compare the algorithm to other self-supervision techniques that have been proposed for pixel-based RL”. This statement confuses me as it suggests that the proposed method works from pixel input where I had assumed that it is working from state input. Now, if the proposed method is actually working from pixel input (maybe applying an encoder to produce a low-dim. state and the augmenting that as described), this needs to be explained better. And if this is not the case and the proposed method actually works on state input, then comparing to pixel-based baselines should be done very very carefully as this is generally not a fair comparison.

Table 2: Here the question is now if “Normal” refers to learning from state or from pixels. And then +CURL/VAE or +S4RL cannot both be extensions on top of “normal” if one works from pixels and one works from state input. In this case, we need two “normal” versions and it should be made very clear in the table which results are based on pixel or state input (which is currently not indicated at all).

Table 1 & 2: In both tables, I have trouble finding definitions for the meaning of the different suffixes to the task names, e.g. “-medium”, “-medium-expert”, “-unmaze”, “-unmaze-diverse”, “medium-play”, “-human”, “cloned”. I would suggest to clearly explain all of these in the captions of these tables and potentially to split up the task name column into “task” and “data generation”.

Right now, the method is only evaluated in an offline setting and that is understandable if the authors are interested in that setting. But it would make this paper much stronger if the interactions of the online / offline setting and the advantage of using the proposed augmentations were evaluated. This way, we could test whether S4RL is especially suited for the offline setting as suggested e.g in L121-124.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies the effect of data augmentation on offline RL algorithms.  Essentially it applies the DrQ data augmentation technique to an offline-RL, proprioceptive-only-state setting.  Specifically, it measures the effect of up to 7 different data augmentations, across 3 different simulated task domains (D4RL, MetaWorld, and RoboSuite), and to two different offline-RL algorithms: CQL and BRAC.  The primary empirical results are that the noisy data augmentation can improve closed-loop evaluations of the policies, with the two most effective methods being either adding noise from (i) a zero-mean gaussian or (ii) adversarially using the Q-function gradients.
strengths_and_weaknesses: To best organize my feedback and analysis, I’ve created three subsections: (i) a summary of strengths, weaknesses, and what I learned, (ii) a list of objective strengths, and (iii) a list objective critiques and room for improvement. 

(i) *Summary of strengths, weaknesses, and what I learned*:

My summary of the strengths and weaknesses are that there seems to be a main strength, and several main weaknesses.  In terms of strengths, the paper’s main strength is that the quantitative improvement shown by the proposed method can be large.  This includes all three of the MetaWorld, RoboSuite, and D4RL results.  In terms of weaknesses, there are several: significant claims are not supported by data, in some cases the data supports the opposite conclusions arrived at by the authors, the scope of novelty is limited, and the consideration of real-world robotics impact isn’t considered.  My subjective recommendation for the paper will be discussed in the Summary of Recommendation, but in the next two subsections, I will try to keep the analysis objective.

I would also like to emphasize to the authors that although I’ll enumerate a large list of critiques, they are on to some interesting results, and I applaud their work so far!  Their work has shown the high potential that simple data augmentation can have on making the most of offline robotics data.  In particular their use of the adversarial augmentation is a nice connection between a couple different works, and opens interesting questions for further work. I hope my feedback encourages them to revise their work further, and I do think this line of work could have an impact on the field, even if it needs significant revision first.

(ii) *Objective strengths*

1. Within the scope of the results reported within this paper, the direct comparisons show that the data augmentation methods (particularly Gaussian noise and adversarial training) improve the closed-loop evaluation results of both CQL and BRAC on the evaluated simulated tasks.
2. Comparing as well against numbers from other papers, it seems that for many of the D4RL tasks these “CQL+S4RL” numbers would be considered state-of-the-art quantitative numbers on D4RL which is a commonly used offline RL simulated benchmark.  

(ii) *Objective critiques and room for improvement in revision*

1. A central claim of the paper is that the presented method reduces overfitting, but there is insufficient data to support this claim.   “Preventing overfitting” is a specific thing, which means that the model fits the training data much better than the test data. This was not measured in the presented experiments.  No analysis of fitting the training data was presented, so overfitting can’t be established.  Since the evaluations all use the metric of closed-loop evaluations, the performance increases that are seen could actually be from a phenomenon besides overfitting, for example having better closed-loop stability.  The models with the data augmentations could even overfit *more*, but have better closed-loop stability.  The authors could distinguish these effects with more experimentation.  The results of such experiments might end up changing their story considerably.
2. To clarify, this overfitting claim is a central claim of the paper that is used both to describe a limitation of prior work  (in abstract: “However, current algorithms overfit to the dataset they are trained on”; in Introduction “neural networks may overfit the data, further resulting in poor generalization…”) and as the key effect of the presented method (end of intro Method paragraph, Section 4: “Then we introduce 7 data augmentations… to reduce overfitting and improve function approximation by smoothening the state-space.”; second paragraph Section 4.2: “by performing valid augmentation… thereby reducing chances of the overfitting to the data”.)
3. The data potentially supports the opposite of the overfitting claims: Appendix C shows that Dropout makes the results worse.  If Dropout, which is well known to prevent overfitting, makes the results worse, this may support the conclusion that the phenomenon does not have to do with overfitting.  This is not discussed.  Instead, the reader is left to assume that the presented data augmentation strategy can reduce overfitting, but Dropout can not.
4. A key statement about novelty is incorrect, and reduces the scope of the novelty claim: “however, it is unclear to how perform [sic] such augmentation from proprioceptive information”. This statement is incorrect because one of the cited references, specifically [15], “Reinforcement Learning with Augmented Data”, does show both methods and results for proprioceptive inputs: see the end of their Section 4 “Extension to state-based inputs” and the results on OpenAI Gym benchmark.
5. Some missing related work discussion would be around noise/augmentation in imitation methods, including http://proceedings.mlr.press/v78/laskey17a/laskey17a.pdf (online adding noise to the demonstrator actions) and https://arxiv.org/pdf/1909.06933.pdf, (Section IVC, offline adding gaussian noise to state).
6. More on novelty claims, the paper says “The main difference between our proposed objective and the normal Q-learning objective is the mean over i different augmentations in the second term of the equation.”  But this makes it seem as though averaging Bellman updates over various augmentations is a contribution of the paper.  This was a contribution of DrQ, see for example Equation 1 and/or Algorithm 1 in [14].
7. “Self-supervision”, which is a central word in the title and in the manuscript overall, is a misnomer and is not an accurate characterization. The presented method is a data augmentation method, not a self-supervision method.  This distinction is understood in the prior works, including both [14] “Image augmentation is all you need” and [15] “Reinforcement learning with augmented data”, which both accurately characterize their methods as data augmentation methods.  This is in contrast with [13] “Contrastive Unsupervised Representations for Reinforcement Learning”, which can be called an unsupervised/self-supervised method, since the formulation involves an actual self-supervised representation.  There is a relationship between data augmentation and many self-supervision methods (i.e. many of the self-supervised methods in computer vision such as SimCLR use data augmentation as a key component), but those other methods have the goal of recovering some particular self-supervised representation.  Instead, the presented paper is strictly a data augmentation method applied to the supervised offline RL objective, with no self-supervised representations used.
8. There are also various incorrect statements discussing the central topic of the paper, which is data augmentation and its effect in the studied setting:
    - The end of the second-to-last paragraph in the Introduction states: “since small perturbations in the given observation should not lead to drastically different Q-values.”  This is not in general a true statement.  It is possible for world dynamics to be sharp or discontinuous such that slightly different states of the world may lead to vastly different Q-values for an optimal policy.  What the authors could say instead is: “Our approach makes the assumption that small perturbations in the given observation do not lead to drastically different Q-values.  While this is not generally true, our results show..."
    - The start of Section 4.2 states, “In computer vision research, augmentations are commonly used as a way to obtain the same datapoint from multiple viewpoints.”  This is not true… obtaining the same datapoint from multiple viewpoints might be called some type of data fusion, but not augmentation?
    - Also near the start of Section 4.2: “Such transformations preserve the semantics of the image after the transformation.” This is not generally true, and depends on which semantics are being discussed.  For example one can apply a left/right flip (mirror) augmentation often when doing classification computer vision, but one would not usually do this for training a robot policy that for example might want to learn to put object A to the left of object B.
    - Also near the start of Section 4.2: “However when working from only proprioceptive information of an agent…, such transformations are semantically meaningless.”  Since semantics are a human-defined concept up to interpretation, one cannot generally say that transformations do not change semantics.  In fact, the authors agree with this, two paragraphs later.
    - Second paragraph of Section 4.2: “In offline RL, data augmentations serve as a simple technique that can allow the agent to do local exploration from the trajectories in the dataset.”  Although we can interpret the augmentations as something similar to imagining local explorations, it’s not true to say that the agent actually does local exploration.  Exploration would imply receiving new observations/rewards from the environment.  The authors have more accurate language to describe this phenomenon later, saying “...enables us to artificially visit states that may be in the state space.”
    - Third paragraph of Section 4.2: “Therefore the choice of \mathcal{T}() needs to be a local transformation to perturb the state without changing the semantics.”  Two paragraphs ago it was stated that transformations are semantically meaningless, but now it says that only small transformations avoid changing the semantics.  This is inconsistent.  Also, it may be helpful in revision for the authors to define their definition of semantics, or perhaps consider removing the phrase and substituting something else that is defined.
9. Although the presented method shows increased performance on the simulated benchmarks, it is not discussed whether the discussed data augmentations would in practice improve real-world robotics results.  As shown in Table 5 in Appendix D, the preferred Gaussian noise is on the order of 1e-4.  The scale of this value is not discussed, but in general the D4RL environments have observations and actions that are normalized to [-1,1] range.  So 1e-4 is 1/20,000th of the observation range.  This might be on the order of natural noise observed by real-world robot sensors, for example the accuracy of a cartesian end-effector state estimate estimated by forward kinematics from 6 noisy encoders.  Point being that perhaps natural noise on real-world robots already provides the benefits observed by the method in simulated scenarios.  This question might be investigated further in simulation, through simulating proprioceptive sensor noise.
summary_of_recommendation: I think it is clear that the paper needs considerable revision, but the question is whether that amount of revision is small enough such that it could be accepted to this conference, or if it is considerably large enough such that it would be best for the work to be revised, more experiments to be run, and submitted to a future conference.  In general I can be very supportive of papers that have a strong empirical simulated result, even if there is no theory or real-world result.  Usually though, one would expect a paper which focuses on empirical results to provide an accurate interpretation and framing of those results.   In my opinion, the scope of revisions and open questions (see my list in previous section) is large enough such that it seems best to revise and submit to a future conference.

*Update after rebuttal:*
I think the paper is more clear now.  I will assume the authors will be willing to make the two edits below, which I think are required for truthfulness.

#1, Line 46, key moment of intro:
- current: "Our main contribution is a Surprisingly Simple Self-Supervised offline RL (S4RL) algorithm that combines studying data augmentations with a simple Q-learning method to significantly improve the performance of offline RL algorithm. "
- critiques of the current statement:
    - I don't think they contributed anything that can be called an algorithm? As I pointed out in my initial review, Equation 4 is directly from DrQ.
    - "Combining data augmentations with a simple Q-learning method" somewhat implies that they have a new Q-learning method, which they don't.
- suggested instead: "Our main contribution is in showing the effectiveness that simple data augmentations can have in improving existing offline RL methods."

#2, Line 88, key discussion of related work:

- current: "The closest work to S4RL include DrQ [14] and RAD [15] in that we utilize their method for Q-learning over augmentations, but in contrast we focus specifically on offline RL for robotic tasks and propose augmentations from states."
- critical error of the current statement:
    - RAD also proposed augmentations on states.
- suggested instead: ""The closest works to S4RL include DrQ [14] and RAD [15] in that we utilize their method for Q-learning over augmentations for online RL, but in contrast we focus specifically on offline RL for robotic tasks, and <insert some statement about the thoroughness of results>"

In contrast I think these statements at other key moments are more truthful and clear:

- Abstract: "In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms."
- Conclusion: "In this paper, we present S4RL: a Surprisingly Simple Self-Supervised offline RL method that uses data augmentations to improve the function approximation for Q-learning algorithms in offline RL."
    - nit: they haven't actually shown that they improve function approximation. They've just shown that the methods work better. But I consider this to be a minor grievance compared to #1 and #2 above.

If they could correct those statements I would be supportive of having the paper at the conference.
issues: See Strengths and Weaknesses, I put a list in the *(iii) Objective critiques and room for improvement in revision.*
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of improving the performance of
offline reinforcement learning (RL) through data augmentation. In
particular, the authors discuss ways of adding random perturbations to
the dataset in a manner that preserves the physical meaning of the
data (e.g., realistic action-reward pairs). The authors discuss a
number of data augmentation methods and present extensive evaluation
over multiple datasets.
strengths_and_weaknesses: Offline RL is an important problem, especially in sim-to-real tasks
where performing online RL might be difficult and time-consuming. As
such, addressing the generalization issues with offline RL is
certainly an interesting and important research direction.

The paper is well written, and the problem is clear. The idea of using
data augmentation is a nice and simple way to improve the performance
of modern classifiers, especially in settings where training data is
not overly abundant.

The results look promising. It would be interesting to see if the
proposed method can be used on benchmarks where the observations are
images or other high-dimensional sensor data. Performing physically
consistent data augmentation in such settings would be more
challenging but potentially more useful as well.

One comment I have is that authors mention in a number of places in
the paper that the proposed approach reduces overfitting, yet this
claim is not properly evaluated. Overfitting is indeed a major concern
in (offline) RL since the learned policy is very much dependent on the
training set (presumably collected using other predefined
policies). One way to evaluate generalization error would be to add
process/measurement noise to the various RL environments and perform a
comparison in that setting as well.
summary_of_recommendation: This paper tackles a challenging problem, and the results look promising. The proposed data augmentation method makes sense, as it has been applied successfully in supervised learning settings.
issues: N/A
review_title: null

Meta-review: After an extensive discussion, the reviewers changed their scores to indicate that the paper should be published at CoRL. I would like to encourage the authors to carefully analyze the reviews and address the reviewers' points (especially regarding some of the wording pointed out by Reviewer oh9V) in the final version of the paper.

Decision: Accept (Poster)