Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new offline RL algorithm that uses in-sample policy evaluation and advantage-weighted regression during policy improvement. Particularly, it utilizes sarsa-like TD to update Q-function and avoids the query value of unseen actions during training. To evaluate their proposed method, they used the D4RL benchmark to compare with previous offline RL methods. In addition, they show results for fine-tuning of learned offline-policies during online deployment.  
main_review: This paper studies how to avoid updating with out-of-sample or unseen actions during Q-updates which is one of the main challenges in offline RL as unseen actions can generate erroneous values for Q-functions and result in overestimation problems. While this paper looks at right place to address this issue, there are some concerns with novelty and design choices in this paper: 

- [1] originally proposed to use SARSA-like TD update during policy evaluation step that doesn't require querying out-distribution actions either. While [1] applies their method only in the discrete-action space, there is significant overlap between this work and [1]. Importantly, the authors fail to acknowledge and discuss this important related work which also weakens their argument about novelty with regard to the in-sample policy evaluation step.

- There is lots of unknown about why expectile regression was selected for value function updates and it is not clear what is the motivation to go with expectile regression rather than mean regression. Particularly, what would happen if use mean regression, i.e. $L_V(\phi) = E_{(s,a)} [(Q_\hat{\theta}(s,a) - V_\phi(s))^2 ] $,  where it requires no $\tau$? How will performance change?  Can you run experiments by using MSE loss instead? 

- One other idea is to use $V(s') = \frac{1}{N} \sum_{a' \sim \pi(.|s')} (Q_\hat{\theta}(s',a')] $ which is utilized in [4] ( see equation 1 in that paper). In addition, this work seems very similar to [4] as well. Can you discuss this and can you run experiments where $V(s)$ is estimated as mentioned? 

- What happens if $V_\phi(s')$ is evaluated on s' in Eq 6 that is not trained on ( i.e s' is not shown during training of $V$ in Eq 5)?

- Looking at table 1, it seems your proposed method performs very similarly to others in locomotion-v2 and kitchen-v0, but marginally better on antmaze-v0. Since these results are very close, can you do significance testing or include their learning curves to provide a better picture about these results? Finally, can you also add BCQ results to Table 1?

- Can you explain how the learned policy is used during the deployment/test phase? Did you evaluate the learned policy with something like Algorithm 1 in [2]? 

Related works:
 In the related work section, you only discussed offline methods where either policy or Q-function are regularized, not both. However, [3] shows that regularizing both policy and Q-function can be beneficial which is relevant to this paper as this paper also regularizes both Q and policy but implicitly. In addition, [2] also restricts the Q-function updates to [almost] state-action pairs that are in-distribution by learning a density function ( although there is still a chance for OOD action to be sampled by their density function). Both these works are missing and need to be discussed in the related work section.

[1] Regularized Behavior Value Estimation https://arxiv.org/abs/2103.09575

[2] EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL https://arxiv.org/abs/2007.11091

[3] Continuous Doubly Constrained Batch Reinforcement Learning https://arxiv.org/abs/2102.09225

[4] Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning https://arxiv.org/abs/2002.08396


summary_of_the_review: While the general motivation of this paper about offline reinforcement learning is spot on, there are issues with this paper as mentioned above. Happy to hear authors' answers to raised issues.


**update after rebuttal**

Considering authors' responses and comments, I've increased my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes in-sample Q-learning which composed of fitting value function with expectile regression, training Q-function and extracting the policy using advantage-weighted behavioral cloning. The method is validated on a set of D4RL examples and shows promising results on Ant Maze tasks. 
main_review: The paper overall is easy to follow and well written. I have two major concerns / questions:
1. For a task with continuous state space, most state may only have one action in the logging data, which means $\pi_\beta(a|s)$ maybe a Dirac distribution. If this is the case, $E_a Q(s,a)$ would be the same with $\max_{a\in A,  \pi_\beta(a|s)>0} Q(s,a)$. If this is the case, does it mean the SARSA-style policy evaluation and optimal Q-function are not different? Then what's the role of expectile regression under this case?
2. The results of IQL on Gym locomotion tasks are on-par or worse than those from CQL, it would be interesting to add some analysis of this. I found it a little bit counter-intuitive because IQL is more similar to behavior cloning than CQL, but underperforms on tasks with better quality data. 

Minor:

1. Should it be $V_{\tau}(s) = E_{a\sim \pi_\beta (a|s)}[Q_{\tau}(s,a)]$ above Lemma 2?
2. It would be great to bold the best model on Table 1. 
summary_of_the_review: In summary, I think the paper is well written and the idea is interesting. The empirical evaluation is significantly better than the existing baselines on challenging tasks like Ant Maze, and I thereby recommend accepting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an algorithm In-Sample Q-Learning (IQL), which learns a Q- and value function under behavior policy and applies advantage-weighted behavior cloning for policy learning. In one sentence, the main idea of the paper is to learn an approximately optimal Q/value function for better behavior cloning. The expectile regression instead of MSE is applied to find optimal Q-function of behavior policy.  
main_review: The paper is well organized and easy to follow and read. The main contribution apart from prior work is the change in Q-function learning, which involves an additional value function learning and the expectile regression technique. After reading, I have the following question: 
1.	The one-step and multi-step concepts come from Peng et al., 2019. I think in their case, the multi-step means iterative optimization between policy improvement and policy evaluation, so that it is the dynamic programming problem. In this paper, the multi-step of IQL is for learning Q function and value function? 
2.	The learned optimal Q-function is for the learning of policy. Why advantage-weighted behavior cloning based on optimal Q-function could be better? I cannot see direct reasons here. 
3.	From the proof, it is only when expectile tau goes to 1 in limit, so that the learned value function could be the optimal under the data. From the code, I observed that for MuJoCo tau=0.7, Adroit tau=0.7, and Ant-maze tau=0.9, which all do not goes limit to 1, like 0.99. I think there is a gap between the theory and the implementation. 
4.	Given your learned optimal Q-function is pretty good and accurate, why not just optimize a parameterized policy (deterministic or Gaussian policy) based on the learned Q-function, for example maximizing E_(a,s) [Q(s,a)]. Why must use behavior cloning? BC is hard to exceed the best policy provided in the dataset.  
5.	Is the value function really necessay in the framework? Do you try to remove the value function learning by just doing expectile regression over target Q values and Q values? The reason for asking this is that in continuous RL case, each state in the dataset is corresponding to one and only one action, which I think it is hard to train a good value function $V(s)$ with only one action $a$ by your eq(5). 

summary_of_the_review: The idea is simple and straight forward, but more explanations are needed to provide strong insights.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an offline RL algorithm named IQL, which generalizes between Bellman expectation equation and Bellman optimality equation with expectile regression. Expectile regression assigns low weights for low-performing samples and assigns high weights for high-performing samples to learn the expectile. By performing $\tau$-expectile regression on the randomness of action in the dataset (from the sampling process of behavior policy), IQL is able to obtain a value function that is $\tau$ optimal. IQL is generalizing SARSA and Q-learning in the sense that SARSA corresponds to IQL with $\tau=0.5$ and Q-learning corresponds to IQL with $tau=1.0$. It offers very stable learning of a near-optimal value since it is only using in-distribution samples and the effective dataset size can be controlled through $\tau$. In the experiments, the paper has shown that IQL can obtain state-of-the-art performance.
main_review: I believe that the main strength of this paper is the novel idea of using an expectile regression on dataset actions to learn a value function that is based on high-performing actions in the dataset. There have been many previous pieces of research on using quantile regression on RL, but most of them were to learn a distribution of value functions where the randomness comes from the environment, and the quantile regression is usually for the worst-case robustness. This paper, on the other hand, uses expectile regression on the value function where the randomness comes from the action and shows that it generalizes the Bellman expectation equation and Bellman optimality equation. As far as I know, this is the first suggestion of such an algorithm. The trick of using both $V$ and $Q$ functions to enable such learning seems to be very clever as well.

Theoretical analysis also backs up the algorithm well, and empirical performance is also presented well, showing its advantage over other algorithms. It is especially interesting that IQL works well on hard domains (antmaze, dexterous manipulation, etc.). IQL will be a simple and robust but high-performing tool for an offline RL. I also believe that a similar approach can be applied to standard RL algorithms, increasing their robustness. 

It will be more interesting if the paper contains more analysis and experiments, such as the learned value functions for different $\tau$s and more state-of-the-art offline RL algorithms, but I am satisfied with the current form of the submission.
summary_of_the_review: The paper provides a good-performing robust offline algorithm based on a very simple but novel idea. I recommend acceptance of paper based on the novelty of the idea and the clean presentation.
review_title: null

Meta-review: This paper proposes a new paradigm --- called in-sample Q learning --- to tackle offline reinforcement learning. Based on the novel idea of using expectile regression, the proposed algorithm enjoys stable performance by focusing on in-sample actions and avoiding querying the values of unseen actions. The empirical performance of the proposed algorithm is appealing, outperforming existing baselines on several tasks. The paper is also well written.
Decision: Accept (Poster)