Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose an Implicit Reinforcement Learning via Supervised Learning (RvS) methods by leveraging the implicit model---the composition of $\arg\min$ with a general function approximator $f_\theta$ to represent the policy ($\hat{a} = \arg\min_a f_\theta (s, a)$) [1]---instead of the traditional explicit model ($a = f_\theta (s)$) in RvS methods to solve offline RL problems. Moreover, the authors provide empirical results to show the superiority of the implicit model. Experiments demonstrate the performance improvement of the implicit model in RvS methods for offline RL.

[1] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Proceedings of the 5th Conference on Robot Learning. PMLR, 2022.
strength_and_weaknesses: Strength: 
1. The authors leverage the exponential tilt density [1] to learn policies that can head toward the largest rewards from offline datasets collected by policies with different expertise levels.

Weaknesses:
1. The authors claim that they bridge an important gap between IBC [2] and RvS by modeling the dependencies between the state, action, and return with an implicit model on Page 6. However, noticing that IBC proposes to use the implicit model to model the dependencies between the state and action, I think the contribution of this paper is to introduce the return from RvS to the implicit model. Thus, the proposed method looks like a combination of IBC and RvS.

2. The authors conduct experiments in Section 5.1 to show the advantages of the implicit model. However, such advantages are similar to IBC, which could hurt the novelty of this paper. The authors may want to highlight the novelty of the proposed method against IBC.

3. The discussions of the empirical results in Sections 5.1 and 5.2.2 are missing. The authors may want to explain: 1) why the RvS method fails to reach either goal and converges to the purple point in Figure 4(b); 2) why the explicit methods perform better than implicit methods on the locomotion tasks.

4. The pseudo-code of the proposed method is missing.

[1] Søren Asmussen and Peter W Glynn. Stochastic simulation: algorithms and analysis, volume 57. Springer, 2007.

[2] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Proceedings of the 5th Conference on Robot Learning. PMLR, 2022.
clarity,_quality,_novelty_and_reproducibility: Clarity:
This paper has several typos as follows.
1. $\eta^{-1} = 5.0$ in the caption of Figure 2(b) is inconsistent with the value in the sentence “In Figure 2b, we study the impact of increasing $\eta^{-1}$ to 3” on Page 4.
2. In Extrapolation of Section 5.1, “Figure 4c a)” should be “Figure 4c i).”

Novelty:
The novelty of this paper is marginally significant.

Reproducibility:
The authors provide the code for reproducibility.

summary_of_the_review: I think the proposed method is a simple combination of IBC [1] and RvS. The claimed major novelty of this paper is the same as that proposed by IBC.

[1] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Proceedings of the 5th Conference on Robot Learning. PMLR, 2022.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes using implicit models for offline reinforcement learning. More specifically, this paper proposes using an implicit model to model the distribution of actions and returns conditioned on observations, and to bias sampling towards actions with high returns. The paper evaluates using implicit models against state of the art offline reinforcement learning methods on the ADROIT, Mujoco, and FrankaKitchen settings.
strength_and_weaknesses: Strengths:
- The paper is well-written and clear to follow.
- The proposed method is interesting and seems to be pretty simple and elegant. 
- The toy examples and plots are nice in providing some intuition about the approach.

Weaknesses:
- It's not clear why IRvS is better than the naive way of incorporating return information by combining IBC with RWR. Furthermore, IRvS only does slightly better than IBC w/ RWR, and the performance is similar on most tasks. IRvS with the best eta does perform better, but requires tuning for eta. Much more thorough analysis is needed here, as this is the main contribution of the paper.
- Training EBMs with Langevin Dynamics can be difficult and requires many hyperparameters. 
- Sometimes reward information is unavailable or difficult to get, particularly in real robotics, and the method may struggle in these cases.

clarity,_quality,_novelty_and_reproducibility: Clarity: paper is well-written.
Quality: See strengths and weaknesses above. In the experiments, the proposed method does not convincingly work better than other approaches for robot tasks. 
Novelty: Technical novelty is somewhat limited, as much of the proposed method does not introduce many novel concepts. 
Reproducibility: The authors would need to provide better documented code, and with all environments, in order to reproduce this work. 
summary_of_the_review: I think this is an interesting direction, but due to the limitations in the performance of the method and drawbacks of the method itself, I think the paper is not ready for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to incorporate the return information in implicit behaviour cloning. Empirical study confirms the efficacy of the proposed algorithms in some domains
strength_and_weaknesses: Strength:
The paper is well written and easy to follow. The idea is simple but proves to be effective in some domains. 

Weakness:
1) The improvements in adroit and kitchen are impressive. However, the reason for the failure in Mujoco is not clear. Since the main contribution of this work is empirical, I feel the authors should dig into Mujoco more deeply to understand this failure.
2) I think some important baselines are missing, e.g., the authors should include comparison with offline RL methods with return-conditioned policy with explicit model, mentioned in Section 4.2. 
3) Table 1 results from only 3 seeds without metrics on variation. So it is not as informative as it should be.
clarity,_quality,_novelty_and_reproducibility: The clarity and quality are good. The algorithm seems easy to reproduce. The proposed algorithm seems novel to me.
summary_of_the_review: see above
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) Summary
The paper combines implicit behavioral cloning with Reinforcement Learning via Supervised Learning. The method is evaluated both on toy benchmarks and (simulated) robotics benchmarks and compared to baselines.

(b) Strengths
- Well written paper on an interesting topic
- Elegant method
- Good variety of experimental domains
- Impressive results on adroit and kitchen

(c) Weaknesses
- The novelty of the proposed method and its significance still remains unclear after the rebuttal
- The promised results with more seeds were not delivered. The statistical significance of the results is still questionable, as are the improvements on MuJoCo
justification_for_why_not_higher_score: None of the reviewers is arguing for acceptance due to the weaknesses outlined above.
Decision: Reject