Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: While the title of the paper suggests that it leverages techniques from the vast literature on numerical continuation, the proposed approach is much more specific. The main idea consists in annealing the temperature parameter of the soft Bellman operator, and to warm start the the corresponding series of problems across time. In the language of numerical continuation, this approach fits under the umbrella of "natural parameter continuation", which I would describe succintly as "warm starting". 

From the perspective of addressing the challenging optimization landscape presupposed in many problems, the combination of "continuation" + smooth approximation to the optimal equations makes sense. However, for me the narrative doesn't hold when the authors motivate their method in the context of offline batch policy gradient methods. As the authors point out, the main challenge associated with offline data is the inability to sample new data. This is problematic in the policy gradient setting because our derivative estimator only holds under the distribution under which the samples have been collected. As soon as the policy parameters are updated, the distributional shift should be addressed via an appropriate change of measure (or via a model). To me, this is the main challenge in the off-policy setting and the proposed continuation-based solution does not address this issue.  I view this as a Monte Carlo estimation problem first; not one pertaining to the optimization landscape. 

Perhaps the paper should have been named differently because the remaining theoretical contributions in the paper do not pertain to "continuation" per se. Theorem 1 provides a bound on the policy gradient methods with a softmax policy (this is different from the "soft" optimality equations). Theorem 2 and 3 follow from the results in Rust (1994, 1996) and in econometrics where the smooth (soft) Bellman operator has been widely used. Theorem 2 follows from the perspective of policy iteration as an application Newton-Kantorovich to the smooth Bellman optimality equations. Theorem 3 follows from Dini's theorem where $\lim_{\tau \to 0} T^\star_\tau v = T^\star v$ where $T_\tau$ would be the smooth (soft) Bellman operator and $T^\star$ the usual Bellman operator. See Rust 1996 "Numerical Dynamic Programming in Economics", section 4, more specifically equations 4.2 and 4.4

review_title: A continuation-like method

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of the paper:
This paper proposed a new batch RL algorithm based on the continuation method in numerical optimization. The proposed method makes use of KL regularization in the offline policy optimization process, with a decreasing temperature parameter -- which comes from the continuation method of optimization. The paper shows that the policy learning with the KL regularized value yields a faster convergence rate, as the motivation of using KL regularization, and a decreasing temperature will ensure the final convergence to the optimal policy. Experiments on Mojoco, atari, and a recommender data-set shows that the proposed algorithm is effective.

Justification for the score:
The proposed algorithm is a natural improvement over the current behavior regularized policy optimization methods. The intuition from the continuation method provides a justification for using a decreasing temperature. I think the main merit of this paper is the solid experiment, in simulation tasks with discrete actions and continuous actions, and in a real data-set. This contribution seems solid, but I have some concerns about Theorem 1 and important related work that is missed.

Detailed comments:
Pro:
1. The intuition behind the algorithmic change is clear enough. Section 3.1 gives the motivation of using a KL regularization and section 3.3 gives an illustrative example of why the continuation method can be better than a constant threshold.
2. The experiment section covers the standard RL benchmark in both continuous and discrete action settings. It additionally studied the performance of the proposed algorithm in a real dataset. Since while most batch RL work only run experiments in simulation tasks, this is a good step to bring the algorithm close to the motivation of doing batch RL.

Cons:
1. Theorem 1 seems to be a bit disconnected from the main contribution of the paper and it's hard to understand its role for me. Later theoretical analysis and practical approximation are all based on the policy iteration algorithm, but Theorem 1 seems to be based on policy gradient. Could the convergence rate improvement be shown in the policy iteration case? Additionally, Theorem 1 says "maximizing $\widetilde{V}^{\pi,\tau}$", does that mean theorem 1 consider the on-policy setting? In such a case what does $\beta$ mean?
2. The proposed algorithm seems to be very related to the BRAC framework and the algorithm BRAC with the KL value penalty. Why is it not mentioned at all? The BRAC paper may not directly use such a decreasing temperature or link it to the continuation method, but as prior work in batch RL, it also considered KL regularization and even studied using an adaptive regularization coefficient. 
3. More recent batch RL algorithms like BRAC, CQL, etc also reported their result in mujoco and atari (CQL did) domains and seems to be better than BCQ, BEAR, and REM. An open dataset D4RL has the reported performance of those more recent baselines. I think in general it will be better to compare with these more recent baselines (especially BRAC since it's very related.).


review_title: New batch RL algorithm with natural intuition and solid empirical study

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary
-------------
The paper extends soft actor-critic (SAC) to the batch RL setting, replacing the policy entropy in the objective function with the KL divergence from the behavioral policy. The temperature parameter tau weighting the reward agains the KL term is annealed towards zero during the optimization process, which corresponds to starting with behavioral cloning for high values of tau and ending up with the standard reward maximization RL objective for tau=0. Theoretical analysis and experiments confirm the advantages of the proposed method.

Decision
-----------
I vote for accepting the paper. The idea of annealing the KL constraint is simple and elegant. Although it is very similar to other constrained policy update methods discussed in the Related Work section, the evaluation in the batch RL setting and demonstration of the improved convergence properties is novel. The execution is of high quality, with evaluations on tabular problems, MuJoCo, Atari, and a contextual bandit problem for movie recommendation.

Questions
--------------
1. As pointed out in Sec. 3.3, when the policy deviates too much from the behavioral policy, the value estimate becomes erroneous. Therefore, a criterion based on the ensemble variance of the Q-function estimates is proposed. Is there a way to derive such a criterion from first principles?
2. Can you relate your work to [1]?
3. Does your method work when the behavior policy is not known but only the dataset is available?
4. Can you quantify how far the optimal policy is allowed to be from the behavioral policy? For example, on a pendulum swing-up task, if the behavioral policy is taking random actions and the pendulum always jitters at the bottom, inferring the optimal policy from this data appears quite challenging. Can one give some criteria when the method is expected to work well?

References
---------------
[1] Nachum, O., & Dai, B. (2020). Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01866.
review_title: Good paper, accept

final_score: rating: 7: Good paper, accept
review_content: The authors propose a KL regularized approach for batch RL, where the importance of the KL term is reduced during learning. Theoretical guarantees are provided in the tabular domain. The algorithm is tested in several domains (MuJoCo, Atari, and a recommender task). 

Strengths:
- Although the theoretical results are mostly straightforward extensions of existing results, they provide a solid backing to the method. In particular, Theorem 1 is an interesting result which motivates KL regularization in this setting. 
- The experimental section is very thorough. I would prefer more seeds but the coverage over many domains and behavior policies is convincing evidence for the empirical success of the method.
- The appendix is very comprehensive. Although code is missing, the method and experiments are reproducible with the provided descriptions.
- The writing was clear, and the method was well-motivated. Overall, the paper feels fairly complete and well polished. 

Weaknesses:
- There is one very glaring weakness to this paper- the proposed KL regularized approach for offline RL already exists (Wu et al., 2019) & related (Jacques, et al., 2019). This is additionally problematic as both methods are not cited or discussed in the paper. To the best of my knowledge, the continuation aspect is still novel as well as the theoretical contributions. However, a discussion on (Wu et al., 2019) is necessary.
- My first impression was that the variance solution to checkpointing $\tau$ in Section 3.3 was somewhat "hacky". On second thought, however, as suggested by the authors in the introduction, gradually reducing $\tau$ provides a mechanism for searching for the "optimal" value which trades between the constraint and learning on top of the proposed benefits of continuation for optimization. I think this is an interesting component of the method. On the plus side I think the checkpointing solves an important problem for batch RL, but on the downside, I think measuring the variance of the ensemble is not as well motivated as the rest of the method. I think the paper could benefit from additional discussion, or experiments which examine this aspect further. 
- While the method is the "best" over a wide range of domains, the performance benefit seems fairly incremental. Consequently, for most users it's unclear if the benefits are sufficiently significant to warrant the additional complexity.

Recommendation:

So firstly, the novelty concerns absolutely need to be addressed and the mentioned papers cited/discussed in the paper. Regardless, I do feel like there is a meaningful contribution that builds on this prior work at both a theoretical and empirical level. As a result, I'm leaning on the side of accept.

References
- Wu, Yifan, et al. "Behavior regularized offline reinforcement learning." 2019.
- Jaques, Natasha, et al. "Way off-policy batch deep reinforcement learning of implicit human preferences in dialog." 2019.

**Post-Rebuttal

The authors have addressed most of my concerns. I have increased my score. Although the additional experiments on the variance/checkpointing are helpful I would still like to see more discussion in the paper itself. 

review_title: Solid paper, but important novelty concerns

Meta-review: The paper got a quite high disagreement in the scores from the reviewers. R2 voted for rejecting the paper as he did not see the connection of the algorithm to the continuation method and also that the continuation method does not address the distributional shift, which is one of the main problems for offlline RL. Yet, these concerns have been properly answered in the rebuttal of the authors and the distributional shift is also addressed by the continuation method by reducing the error in policy evaluation. Further concerns from the reviewers were raised in terms of related work to a similar algorithm (BRAC), which is also addressed in the revision of the paper. 

The reviewers also identified the following strong points of the paper:
- The algorithm is a simple and very effective adaptation to SAC 
- The presented results are exhaustive and convincing
- The paper provides strong theoretical results for the presented algorithm
- The authors did a very good job with their revision, adding more comparisons and ablation studies.

I agree that this paper very interesting and recommend acceptance.
Decision: Accept (Poster)