Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies offline reinforcement learning by proposing the Implicit Value Regularization (IVR) framework. In IVR general regularizers are added to the policy learning objective. Algorithms from prior work are analyzed in this framework showing some of their weaknesses. From these theoretical insights, a practical algorithm is derived that achieves strong results on the D4RL benchmark.
strength_and_weaknesses: The paper is well-written and does a good job in explaining relevant concepts. The theoretical analysis gives interesting insights and shows also how prior algorithms (IQL and CQL) fit into this framework. Further relevant work is nicely summarized in the context of the paper and put in comparison to the proposed method. 
The proposed method Sparse Q-learning (SQL) is derived from the theoretical framework under reasonable and well-explained assumptions.
SQL achieve SOTA performance on the D4RL benchmark and also strong results for the noisy and the small data regime.

Some typos:
p.2 "implicit value regularization" -> "implicit value regularization"
p.2 "IQL remains much close to the..." -> "IQL remains much closer to the..."
p.2 "...similarities to IQL However..." "...similarities to IQL. However..."
p.3 In the last equation defining the policy evaluation operator for V: There should be a plus instead of a minus
p.4 "... adds a entropy..." -> "... adds an entropy..."
p.5 the sentence " U∗(s) can be uniquely solved from the equation obtained by ..." appears twice in two consecutive paragraphs.
p.9 "This simulates the situation where the dataset is fewer and has limited state coverage near the target location because the data generation policies maybe not be satisfied and are more determined when they get closer to the target location"  what is meant here??
clarity,_quality,_novelty_and_reproducibility: The paper is written well and clearly. It gives both new theoretical insights and a practical algorithm that achieve strong performance.
summary_of_the_review: This is an interesting paper both in terms of theory and practical results. Hence, I vote for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces Implicit Value Regularization (IVR) framework, provides some interesting theoretical results that might help better understand offline methods such as CQL and IQL, and then under this framework, a new offline RL method called Sparse Q-learning (SQL) is proposed. 

Empirical results from D4RL benchmark shows a consistent superior performance over competing algorithms. Authors also provided some other analysis on SQL and also compared to IQL on noisy data regime and CQL on small data regime (mixed datasets made from D4RL datasets). 
strength_and_weaknesses: Strengths:
- Overall good writing quality and clarity. 
- Interesting theoretical results that might help better understand offline methods that rely on some form of regularization terms. 
- Empirical results on D4RL and other 2 comparisons seem to show SQL has stronger performance than alternative methods, and some of these methods are recent methods with good performance. 
- Technical details provided for reproducibility. 
- a lot of discussion on related works, easy to understand how it relates to prior works

Weaknesses:
- Thank you for providing technical details including your hyperparameter search range. However, I am a bit concerned that it seems you performed a hyperparameter search on each dataset individually, and then select the best-performing one. This makes sense, but for the methods you compare to, are they using the same hyperparameter for all tasks, or do they also search hyperparameter on each individual dataset and report the best one? 
- Similarly, in the case when you compare SQL to CQL and IQL on the noisy and small data regime settings, did you also do a thorough hyperparameter search for IQL and CQL? 
- Maybe I missed sth but have you provided empirical evidence and figures to demonstrate that your method has a more "sparse" policy than the other competitive methods? 
- What is the final hyperparameter you selected for each individual dataset? (You mentioned the best is selected for each dataset, would be good if you also report the actual numbers)
- What about computation efficiency? How fast in wall-clock speed does your method compare to others? 
- Hyperparameter sensitivity: for example can you provide some results on how performance changes with different alpha values? This will help other researchers understand how hard it is to tune your method on new tasks. 

Not really important: 
- some minor typos

Other Questions: 
- Will your code be open sourced? 
clarity,_quality,_novelty_and_reproducibility: - Clarity: Overall writing is clear and presentation is good. 
- Quality: Overall paper quality is good. 
- Novelty: The new method is not that different from existing methods, however, the design of the new method together with the theoretical contributions can be considered novel. 
- Reproducibility: Overall the method is not too complex, technical details provided and some discussion on hyperparameters provided. 
summary_of_the_review: Overall seems a solid paper, some interesting results and theoretical analysis. Overall writing is good. 

My main concern: 
- Additional technical details and whether other competing methods also had thorough hyperparameter tuning on each task should be provided, also the computation efficiency comparison, and final hyperparameters for each dataset would be good to have in the paper. 
- I'm looking for a bit more analysis on your method, for example on the sparsity thing

I'm willing to increase my score if concerns are fully addressed and if anything is wrong in my comments please point it out. 

=============== post rebuttal ===============
Sorry for the late reply, I thank the authors for their effort in addressing the concerns and providing the additional results. The new results seem to be more convincing. I have also checked other reviewers' comments and it seems most of the major concerns are addressed by the rebuttal. Based on that I increase my score to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper theoretically derives an in-sample offline RL algorithm, Sparse Q-Learning (SQL), using a behavior regularized MDP for offline reinforcement learning. SQL has interesting connections to existing offline RL algorithms including CQL and the in-sample IQL algorithm. SQL performs similarly to prior methods on MuJoCo and Antmaze tasks and better on Kitchen tasks in D4RL. SQL is shown to be superior on custom smaller datasets than CQL as well as more stable than IQL on mixture datasets containing expert and random data.
strength_and_weaknesses: Strengths:

- The paper is well written, especially the theoretical section.
- The proposed SQL method is theoretically well-motivated. The theory also points out interesting connections to existing methods including CQL and IQL. Furthermore, SQL only requires one hyperparameter as opposed to two in IQL.
- The implicit value framework derived in the framework seems like a general framework for instantiating in-sample Q-learning methods based on $\alpha$-divergences (although most methods might turn out to be intractable).

Weaknesses

- The improvements on the D4RL benchmark do not seem to be statistically significant -- Reported results are likely within 1 standard deviation of baseline results and thus the SOTA claim may not be valid.
   - Error bars are not reported for any of the baselines in Table 1 and 2 -- however, the standard deviation is known to be large for D4RL datasets. See https://openreview.net/pdf?id=Y4cs1Z3HnqL for a paper that reports standard deviation for some of the baseline methods.
   -  The proposed method seems more complex to implement than existing methods such as IQL, while resulting only in marginal gains. 
-  Possibly Unfair Comparisons: It seems that per-task hyperparameter tuning is done for SQL while the baseline methods' results seem to be copied from prior papers which used the same hyperparameter for a given domain (Kitchen / Antmaze / MuJoCo).
- It's not clear whether "sparsity" plays an important role in performance of SQL -- this is not explored much in the paper despite the title.
-  

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clear and easy to read.

Quality: Fair. The paper could do a better job of demonstrating the benefits of SQL over IQL.

Originality: Fair. The theoretical results are novel and interesting, however the empirical method is highly motivated by the in-sample Q-learning (IQL) method. 

Reproducibility: There are not enough details in the appendix to easily replicate the results on D4RL -- This possibly can be fixed if the code is open-sourced. 
summary_of_the_review: Overall, the paper presents theoretically well-motivated method for addressing the ad hoc choices in IQL. However, the method doesn't seem to offer much empirical gains over IQL on the standard benchmarks. However, other than robustness results on mixture datasets (noisy data regime in Section 5.1), it is unclear whether SQL offers any benefits over IQL in terms of practical benefits. 

Questions / Suggestions:
- Can you provide empirical evidence for whether SQL provide any benefits over IQL in the small-data regime in Section 5.2?
- In-line with the best practices for evaluation, I'd recommend the use of [rliable](https://github.com/google-research/rliable) library [1] to validate the claim for SOTA results. Furthermore, standard deviation be reported for all methods in Table 1.
- Does the method scale to high-dimensional image-based datasets such as the Atari datasets in RL Unplugged [2]? 
- Kumar et. al (2022) showed degradation in CQL performance with prolonged training (including on the Antmaze datasets) -- would SQL  provide any benefit compared over CQL / IQL due to being a principled approach for in-sample learning?
- Are there any guidelines to set the $\alpha$ parameter -- given that SQL is theoretically motivated, it would be nice if the hyperparameter tuning is easier / more intuitive than existing methods like CQL / IQL.

Minor:
- Can you clarify how Jensen's inequality is applied in Equation 2?
- The generality of the IVR framework would be seen if another value of $\alpha$ was instantiated to derive in-sample version of another existing method or deriving a new method altogether.


References:

[1] Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., & Bellemare, M. (2021). Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 29304-29320.

[2] Gulcehre, C., Wang, Z., Novikov, A., Paine, T., Gómez, S., Zolna, K., ... & de Freitas, N. (2020). Rl unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Processing Systems, 33, 7248-7259.

[3] DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization (2022). ICLR. https://openreview.net/forum?id=POvMvLi91f


-------
----------  Post rebuttal ---------- 

Based on authors's response as well as the updated paper, I am now in favor of acceptance and updated the score to 8 (accept) from 5 (below accept).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes the Implicit Value Regularization framework for offline reinforcement learning. This framework includes existing works such as the in-sample learning method, IQL, and value regularized method, CQL. The authors further propose a new practical algorithm, Sparse Q-learning that show SOTA performance on the D4RL benchmark.

Strengths:
- Theoretical analysis provides useful insight to show the weakness of existing algorithms
- A new practical algorithm that introduces sparsity in learning the value function, achieves better results and faster convergence.
- The authors also provide a section to analyze the sparsity characteristic of SQL in appendix B following the suggestion of a reviewer.

Weaknesses:
- The additional hyperparameter needs to be tuned per dataset, though the authors show the choice of hyperparameter is relatively robust.

The major concerns of the reviewer have been addressed after the rebuttal and all agree that this paper presents good contributions to offline RL.
justification_for_why_not_lower_score: The reviewers reach unanimous agreement that this paper presents both a general framework for offline RL with interesting theoretical insights and a practical algorithm that achieves SOTA performance.
Decision: Accept: notable-top-5%