Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a conservative smoothing technique by adding perturbation to the states to improve the robustness of learned policy on offline RL. Theoretically, they claim their work enjoys a tighter suboptimality bound in linear MDPs.
strengths_and_weaknesses: **Strengths:**
Prior offline RL methods mainly focus on the OOD actions instead of states. This paper targets the robustness issue for Q value via perturbing the states. This paper is well-written and easy to follow.

**Weaknesses:**
The algorithm employs the ensemble Q functions. It is well known that the ensemble technique can bring robustness to learning. However, it is unclear whether or not the ensemble technique brings the main contribution to the robustness. To this end, comparing single agent methods, such as CQL and BC, is unfair. Moreover, on the results of the experiments, the proposed RORL is also very close to the ensemble baseline EDAC.
questions: More ablation study is necessary, especially for clarifying the contribution of the ensemble technique.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the problem of training robust RL agents with offline datasets. The paper claims that regularizing policy and value networks to have similar values against adversarial perturbations and applying this technique to PBRL can achieve state-of-the-art performance on the D4RL benchmarks in both standard and adversarial settings. The paper also provides a theoretical analysis of the sub-optimality gap of the proposed algorithm in linear MDPs.
strengths_and_weaknesses: Strengths:
- The paper thoroughly analyses the proposed algorithm both empirically and theoretically. 
- The proposed method achieves state-of-the-art performance in the standard offline RL benchmark, while the performance improvement is quite marginal.

Weaknesses:
- The paper does not provide experimental results on expert or near-expert datasets.
- The proposed algorithm requires computing an adversarial example $\hat{s}$, which is computationally expensive.
questions: **OOD penalization**

As mentioned in line 147, the difference between the proposed method and PBRL in OOD penalization is that the proposed method penalizes both OOD states and actions. However, assuming that $\hat{a}$ is close to $a$, the proposed method has a similar effect as PBRL since the Q-functions are trained to have similar values for $s$ and $\hat{s}$. Could you please conduct an ablation study that only penalizes OOD actions like PBRL?

**Expert datasets**

Is there any reason to exclude *-expert-v2 or *-medium-expert-v2 datasets from the experiments?

**Comparison with S4RL**

I think the paper is closely related to S4RL [1], which utilizes adversarial examples for smoothness. It would be better to provide the results of CQL + S4RL or PBRL + S4RL.

[1] Sinha, et al., S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning in Robotics, 2021.
limitations: I think the proposed method is much slower than other offline RL algorithms since it has to solve mini-max problems. Please report the wall-clock time of the proposed method in the main text.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an approach to offline reinforcement learning that is robust to small perturbations in the observation space such that the changes are not detrimental to the performance of the final policy. The achieve this by encouraging the value estimator network to be smooth over the state space while being conservative on out of distribution samples. Moreover the learned policy is also constrained to change less with these perturbations.  Experimental results show that the proposed algorithm is able to perform competitively with current Robust / Baseline approaches, and enjoy increased robustness over adversarial attacks. 
strengths_and_weaknesses: ### Strengths
S1: Overall the motivations for the approach are intuitive and easy to follow. The paper is well written and clear. (excluding some minor comments in the questions section.) 

S2: The proposed attack metrics are diverse and well defined under the current framework. The experimental results show that RORL is more robust towards the proposed attacks as compared to other methods while being able to perform competitively under normal conditions.

S3: Theoretically, the proposed framework (RORL) enjoys a tighter suboptimality bound that PBRL.  

### Weakness

C1: My main complain towards the paper is that while the authors specifically tackle the tradeoff of conservatism and robustness in offline RL no clear metric has been defined to quantify the robustness of an approach. Hence the reader is forced to trust their eyes over the evaluation curves to judge the robustness of the approach. It may be insightful to invest some thoughts on quantifying the robustness by for example measuring the area under the curve on the performance under attack curve where the scale is normalized over the overall variation and dimensionality of the given dataset. 

C2: While this might come off as a knee jerk comment, It would be interesting to see the same set of experiments on more challenging benchmark such as ant-maze, especially as the proposed approach is claiming to better generalization ability and improve the overall robustness of the policy. Will the generalization ability result in significant improvements in a more challenging domain ? 

questions: Q1: I could not find a definition for the distance metric d used to calculate the epsilon (line 90). Is this a L2 distance metric ?
Q2: Can the epsilon be normalized so that it is takes into account the overall variation of the state space as well as the dimensionality. ?  
Q3: What parts are highlighted in the table 1, is this discounting the scores that are not statistically significant during ranking? 
Q4: Should the score for scale 0 attack at figure 4(b) be same for all attacks and match the evaluation score at table 1? If not what is the effect of scale zero on the final policy. 
limitations: While the authors clearly mention the adversarial state sampling as their main limitation, they do not quantify this slow down for different state sampling approaches. It would be interesting to see the actual effect on the compute time as a percentage of the total training time.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors handled my major concerns on approximation and experiments by providing additional responses and adding more experiments. I'd like to improve my score as boarderline accept.

===

Training robust RL agent from offline datasets is an important yet challenging problem. This paper proposes RORL: offline RL algorithms with conservative smoothing. The main idea is to add smooth constraints (forcing agent to generate outputs on (adversarially) perturbed inputs) to offline RL algorithm. Here, to avoid overestimation issues, the authors also utilize uncertainties from Q-function as a penalty term. The proposed method not only achieved strong results on mujoco tasks from D4RL datasets but also showed that learned agents are more robust to perturbation.
strengths_and_weaknesses: # Strength

* Motivation is clear and the proposed method sounds reasonable. 

# Weakness

* Lack of evaluation on challenging domains. Even though mujoco tasks are standard benchmark in offline RL, it would be nice if the authors can evaluate on more challenging tasks such as AntMaze or Atari. Also, it would be nice if the author can consider the combination with more state-of-the-art offline RL algorithm [1].

# Overall

* I think this paper studies an important research question and proposes a reasonable solution. Also, the authors showed the gains from the proposed method very clearly on standard offline RL benchmarks. However, at the same time, there are several concerns (i.e., more evaluation on challenging tasks, combination with more stated-of-the-art offline RL algorithms and so on) about the draft. Because of that, I'd like to suggest "weak reject" but I'm also willing to change my score based on other reviews and author responses.

[1] Kostrikov, I., Nair, A. and Levine, S., 2021. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.

questions: Q1. I like the examples for motivating the paper in Section 3 but have the following minor questions:

* To obtain a perturbed state, the authors utilize action diff. However, generating perturbed state with respect to Q-function would be more natural choice for this example. Because the authors considered the perturbation with respect to Q-function in main algorithm and experiments, I wonder if there is a special reason to generate perturbed state using action diff here.

* The author mentioned that they choose one of the reduced dimensions (by PCA) in Figure 3. It would be nice if the authors can clarify how to choose this dimension (for reproducibility).

Q2. I have the following questions about the method:

* What is the definition of perturbation set? 

* According to Algorithm 1, the authors mentioned that $\hat s$ is sampled from the perturbation set but description in line 133 sounds like they found the worst case state which maximizes eq (2). This part is bit confusing. 

* Finding worst case state, which maximizes eq (2), is computationally expensive. Could the author clarify this part? 

limitations: As pointed out in Section 8, the overhead induced by the proposed method can slow down the training. Even though this can be handled later, it would be nice if the authors also can clarify the training overhead from the proposed method (e.g. comparing training time of RORL and other offline RL algorithms).
review_title: null

Meta-review: metareview: All reviewers agree that the author's response has addressed their primary concerns. Reviewer frMM had two reservations that resulted in a borderline rating 1) concerns about how the adversarial samples were generated and 2) a request for evaluation on AntMaze. The author's followup response and further experiments address 1 and partially 2. It would be great to see RORL results on AntMaze in the final version.

Overall, the performance of RORL is competitive with state-of-the-art methods on Mujoco and Adroit tasks with fewer ensemble elements needed. The main benefit is on improved performance against adversarial attack, where RORL significantly improves over existing methods. I think the paper makes a nice contribution that the community will find valuable.

I encourage the authors to think carefully about how to integrate the additional experiments into the paper to resolve the questions raised by reviewers.



confidence: Less certain
Decision: Accept