Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The paper proposes an off-policy actor-critic QWR algorithm that extends the AWR from (Peng et al. (2019)). The QWR algorithm estimates the Q-function using parameterized Q-network whereas AWR estimates the V-function. 

I think the paper is not novel enough to guarantee acceptance. The contribution of the paper of the paper is limited given the existing work on AWR. 

The Section 2.2 is very confusing. They prove that AWR convergences to a policy that takes the actions appeared in the replay buffer in state-determines-action assumption holds. I think no-algorithm can learn if the state-determines-action assumption holds. Many existing off-policy papers make assumption on sampling policy (strong ignorability (1)). 

1. Shalit, Uri, Fredrik D. Johansson, and David Sontag. "Estimating individual treatment effect: generalization bounds and algorithms." International Conference on Machine Learning. PMLR, 2017.
review_title: Not novel enough

final_score: rating: 3: Clear rejection
review_content: This paper proposes a variant of AWR with an added Q-function. It is motivated by the *state-determines-action* assumption, which is used to argue that AWR should always return the data-collecting policy if optimized to convergence. Experimental evaluation of QWR focuses on online sample efficiency and offline performance.

I have reviewed this paper for a previous conference. It seems that the issues pointed out during that review process have not been addressed. Most importantly, the main motivating theorem (Theorem 1) for QWR appears to be incorrect: it relies on $\log \pi(a \mid s)$ being nonpositive, but this is not true in general in continuous action spaces. Instead of fixing the theorem, it has been moved to the appendix.

The algorithm presented here is largely the same as that in two recent papers: [advantage-weighted actor critic](https://arxiv.org/abs/2006.09359) and [critic regularized regression](https://arxiv.org/abs/2006.15134). It would be unfair to fault this paper for its similarity to others written in the same timeframe, but it does probably mean that the paper should be judged more on its experimental evaluation and analysis than on its novelty. By the metric of experimental evaluation, it falls short. The offline setting has recently seen more standardization of benchmarks (see, eg, [D4RL](https://arxiv.org/abs/2004.07219) and the [DQN Replay Dataset](https://offline-rl.github.io/)), and evaluating on these would allow for cleaner comparison to other works. (To the paper's credit, it does discuss these prior works and their relation to QWR.)

It is possible this comparison was not performed because the offline setting studied is different than the usual offline setting: the data-collecting sampling policy $\pi(a \mid s)$ is required for the optimization in Equation 3, unlike most offline RL works which use only logged trajectories. It is also not necessarily an issue to study a different setting, but it requires some amount of justification for the modified problem setting. It also makes the evaluation more important, because it becomes unclear whether any gains are due to improving the algorithm or relaxing the problem statement.

If I am mistaken about the main theorem being incorrect, I will happily increase my score.

review_title: Large problems remain unaddressed from previous submission

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on offline policy learning with a limited dataset and proposes a sample efficient algorithm called Q-Value Weighted Regression. Based on the Advantage Weighted Regression algorithm, this algorithm calculates the advantage of the sampling policy \mu by estimated Q-value function. Experiment results show that the QWR algorithm has better performance than the AWR algorithm with limited data.
This paper is well-written and easy to follow.  The main contribution is delivered:  New sample efficient algorithm. However, I still have some concerns about this paper.
First, in the experiment part, the authors compared the performance between three kinds of QWR algorithms and the AWR algorithm. Though QWR has better performance than the AWR algorithm, it seems that the QWR algorithms' output is more unstable than other algorithms.
Second, as the authors mention in related work, several recent works have developed algorithms similar to the QWR algorithm. It is better to show the performance of those algorithms in the experiment. If those developed algorithms have similar performance to the QWR algorithm, the QWR algorithm's importance may be affected.
Third, in the critic part of the algorithm (line 7 to line 15), the Q^*-value function is estimated by sample a’ and the Q_{\phi} value function uses samples (independent sample for two value function). However, in the actor part of the algorithm (line 16 to line 20), both the value function Q_{\phi} and \hat{V} is estimated by samples (same sample for two value function). So, I was wondering why there is a difference between the critic-step and actor-step?
Finally, in the formula (5), it seems strange that there is an expectation for random variable a’ and variable s', but none of them appear in the formula.
review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper presents a Q-value weighted regression (QWR) on top of the advantage weighted regression (AWR) to improve the sample efficiency for offline RL settings. Through the analysis to the AWR, the authors claim that it performs poorly in scenarios with discrete actions, which motivates the development of QWR. Empirically, the authors show that QWR is comparable to SAC in continuous tasks and a variant of Rainbow in discrete tasks. 

I think the topic investigated in this work is critical. Particularly in many real-world applications, which are offline, the sample efficiency stands to be the most important problem. The paper is also easy to follow and includes thorough investigation on the literature survey. However, the authors need to pay attention to a few major issues that discourage me to give a decent score. 

1. Though this work proposes a new offline RL method, the novelty is marginal. As QWR is completely on top of AWR, which seems quite straightforward. This just looks like an incremental step. The authors mention in the paper they theoretically analyze the AWR to motivate the QWR, but after checking the Appendix, no formal theoretical results have been reported, particularly for the proposed QWR. It would be better to see formally how QWR improves on top of AWR, in terms of returns and some constants. 

2. The state-determines-action assumption is problematic. In the paper, the authors say “But in the case of limited data, when only a few trajectories were collected, this assumption may hold, at least for a large subset of the replay buffer, which makes it relevant to the study of sample efficiency.” Such an assumption is quite strong in the paper. Since the authors leverage this assumption to give Theorem 1 for AWR, which reveals shortcomings for AWR. This sets a nice motivation for QWR. However, without this assumption, the motivation becomes weak? How to justify this assumption? I would suggest the authors to show in-depth analysis for AWR and QWR to theoretically show their fundamental distinctions.

3. The experimental results are not convincing. In Table 1, though QWR outperforms both AWR and PPO, but not SAC. Similarly, in Table 2, still QWR cannot outperform completely baselines. Though in the abstract, the authors have mentioned, in the result section, they failed to give any further discussion. For offline RL, AWR completely outperforms the proposed QWR, though they are close. Still, the authors just directly described the results without discussion.

*****************************
After reading carefully the rebuttal from the authors, I raise the score a bit as it somewhat clarifies some confusion. However, the paper still needs improvement, particularly in terms of analytical results. 
review_title: Review of "Q-value weighted regression: reinforcement learning with limited data"

Meta-review: This paper proposed Q-value-weighted regression approach for improving the sample efficiency of DRL. It is related to recent papers on advantage-weighted regression methods for RL. The approach is interesting, intuitive, and bears merits. Developing a simple yet sample-efficient algorithm using weighted regression would be a critical contribution to the field. The work has the potential to make an impact, if it has all the necessary ingredients of a strong paper.

However, reviewers raised a few issues that have to be addressed before the paper can be accepted. As some reviewers pointed out, there seem to be unaddressed major issues from previous submissions. Novelty appears limited, especially because the proposed approach is very similar to recent works (e.g., AWR). The experiment section lacks comparison to recent similar algorithms, and the available comparisons appear to be not strong enough to justify merits of the proposed algorithm. Theorem 1 requires an unrealistic state-determines-action assumption for the replay buffer. Although the authors made an effort to justify this assumption, it remains very problematic and rules out most randomized/exploration algorithms.
Decision: Reject