Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes In-sample Actor Critic (IAC), a new algorithm for offline reinforcement learning. The main idea is to avoid extrapolation error by explicitly performing in-sample policy evaluation to learn the critic. To implement this, IAC applies sampling-importance resampling which has lower variance than a naive importance sampling implementation. The paper provides empirical studies in Mujoco locomotion control and AntMaze showing that IAC contains competitive performance compared to the state-of-the-art algorithms. 
strength_and_weaknesses: Strength

As discussed in the paper, offline RL algorithms suffer from extrapolation error by bootstrapping from out-of-distribution (OOD) actions. Previous methods mostly rely on using regularization to penalize OOD actions. But such methods are very sensitive to the regularization level. More importantly, when the data collection policy is very bad, the regularization actually prevents the algorithm from finding a good policy. What I really like about this paper is to rethink the problem from the first principle: is it possible to do bootstrapping from the in-sample actions? From this perspective I think this paper makes contribution to a very important problem in offline RL. The paper is also well organized and well written. Due to the high variance of importance sampling (IS), the paper proposes to use sampling-importance resampling (RIS). The authors then compare the variance of RIS and IS, and discuss how RIS is applied to the critic learning objective in offline RL algorithms. Most implementation details are clearly discussed in the paper. 


================================================


Weakness: 

I have the following questions and comments. Please correct me if I missed anything important. I am happy to adjust my score based on how well the authors answer the questions in the rebuttal. 

1. My major concern is on the computation cost of the algorithm. It seems that after updating the current policy, IAC needs to adjust the weights (is ratio) for all (s,a) in the dataset.

2. IAC needs to learn the behavior policy before training. How does it affect the performance if we train $\beta$ as the other networks? Is there an ablation study for this? 

3. The paper suggests that one can simply treat $\beta$ as a constant such that it’s canceled in the weights. In fact, from Table 1, we can see that w/o $\beta$ even produce better results than original IAC. Is there an explanation on this? It’s always interesting to understand why something you try to approximate actually works better than the original algorithm. 

4. Following the previous question, I am wondering if it is because the domains are continuous control problems, where we can only observe one action in one state. Thus the log probability of $\beta$ will be very close to zero. Then I am wondering how does IAC perform in discrete problems? 

5. IAC works very well empirically. But I am wondering why the algorithm performs so well, especially compared to IQL, which also performs in-sample bootstrapping. One limits of IQL in my opinion is that it’s hard to approximate the in-sample maximum with expectile regression when the behavior policy is very sub-optimal. But it seems to me that RIS also has the problem. I think it’s better to provide some evidence, even simple proof-of-concept experiments, to show the advantage of IAC over IQL. 

6. In Section 5.1, the authors suggest that IAC outperforms one-step RL as it has lower bias. But isn’t that the main advantage of IAC compared to one-step RL is to use multi-step DP?  (This doesn’t effect my score, I am just wondering if this discussion is important. )

7. Finally, I am wondering what does OOD action exactly mean? The paper seems to suggest that OOD actions are those actions that are not in the dataset. Despite this is very reasonable, another view is to think of actions that are out of the support of the data collection policy. The difference is that for example an action can be covered by the behavior policy, but due to finite samples we don’t observe it in the dataset. It seems to be the second perspective is more interesting, as it allows generalization. Any comments on this? 

clarity,_quality,_novelty_and_reproducibility: Please see the previous section for more details.
summary_of_the_review: I recommend not accepting as the proposed algorithm might have large computational cost in practice and some claims are not well support by the experiment results. However, I am willing to adjust my score if the authors answer my questions in the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper applied the sampling-importance resampling methods to offline learning to conduct in-sample learning. The methods in this paper fit well into the line of in-sample offline RL learning.


strength_and_weaknesses: 

Strength:

The method is very clear and makes sense for in-sample offline learning.

The paper contains several solid theoretical analyses of the method.

The experiments on Gym locomotion tasks and ANtMaze tasks are sufficient to show the effectiveness of the method.

Weakness:

One limitation of the method is that it needs to know the probability of the in-sample action in the dataset. Although the paper also presented the result by removing the density estimator, this method does not perform well on AntMaze problem. Can you give the reason why the method does not perform well on the AntMaze-UMaze and AntMaze-Large datasets? The paper also didn't report the training time for using a density estimator.

The density estimator only uses Gaussian distribution to model the distribution. This often could limit the capacity as in practice the distribution of the dataset could hardly be a unimodal distribution.



Several Questions:

What's the training time for using a density estimator?

I cannot see why the proposed method outperforms other in-sample offline RL methods. Could the author provide some insights on this? 


clarity,_quality,_novelty_and_reproducibility: 

The presentation of the paper is clear.

The methods in this paper fit well into the line of in-sample offline RL learning.

The implementation of the paper is not open-sourced. Open-source code is not compulsory, however, it could help readers understand some details of the methods.
summary_of_the_review: 

The presentation of the paper is very clear and makes sense for in-sample offline learning. The theoretical analyses of the method are solid and well support the main idea of the paper. The experiments on various tasks are also sufficient to prove the effectiveness of the method.

Although the method proposed in this paper is very interesting and contributes to the community, I cannot see why the superiority of the proposed method compared to existing other in-sample methods.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents In-sample Actor-Critic (IAC), an algorithm for offline RL. The policy evaluation of IAC is done only using in-distribution samples of the dataset, where each sample's (normalized) importance ratio is used for resampling probabilities. Then, TD updates are performed using the resampled samples. For policy improvement, advantage-weighted regression is used to ensure that the learning policy samples actions within dataset support. The SIR is a consistent estimator and has a lower variance compared to the standard importance sampling. Experimental results show that IAC performs competitively with the state-of-the-art methods on Gym-Mujoco locomotion tasks and AntMaze tasks.

strength_and_weaknesses: [Strengths]
1. This paper presents an offline RL algorithm exploiting in-sample learning based on importance resampling, which enables stable learning yet achieves good empirical performance.


[Weaknesses]
1. It seems technical novelty is limited, given that IAC can be understood of a combination of Importance Resampling [1] for policy evaluation and AWAC[2]/CRR[3] for the policy improvement part.
2. The algorithm requires behavior density estimation, which introduces additional computational costs and hyperparameters.
3. In the experiments, comparison with some baselines is missing, e.g. AWAC and CRR[3], which are using similar approaches. Also, it would be great to compare with the method using self-normalized importance sampling instead of SIR.


[Questions and comments]
1. I am wondering if using state-distribution correction $d^\pi(s) / d^\mu(s)$ can further improve the performance.
2. In addition to the OneStepRL and IQL, OptiDICE [4] only requires in-distribution samples $(s,a,r,s')$ of the offline dataset too.
3. I am curious to see the result when using different $\eta$ in Eq (12), e.g. $\{0.1, 0.3, 0.5, 0.7, 0.9\}$. Does $\eta=1$ always perform the best?


[1] Schlegel et al., Importance Resampling for Off-policy Prediction, 2019

[2] Peng et al., Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019

[3] Weng et al., Critic Regularized Regression, 2020

[4] Lee et al., OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation, 2021

clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and easy to follow. It seems that the technical novelty is limited given that IAC is a combination of existing methods for policy evaluation and policy improvement. It seems Theorem 3 of the paper is very similar to Corollary 3.1.1 in [1], but it was not properly mentioned.

[1] Schlegel et al., Importance Resampling for Off-policy Prediction, 2019

summary_of_the_review: I think this paper presents a sound offline RL algorithm of in-sample learning. My main concerns are technical novelty and some missing baselines in the experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the offline Reinforcement Learning setting, and more precisely the problem of computing Q-Values from transitions in the dataset, where bootstrapping from out-of-distribution actions (actions not in the dataset) may lead to over-estimation and poor learning. The proposed solution builds on sampling importance-resampling, and changes how the transitions used for learning are sampled from the dataset, to sample "good" (in-distribution) samples more often. These transitions are used to train a critic (using a somewhat conventional TD loss, with the main contribution being the sampling and not a new loss), and an actor, in the loss of which the estimated behavior policy of the dataset appears.

Experimental results are thorough and show that the proposed method outperforms several baselines on challenging continuous-action tasks.
strength_and_weaknesses: Strengths:

- The problem being considered is important and an active area of research. The proposed solution is elegant and outperforms the baselines. It also seems relatively easy to implement and computationally cheap (except that a SumTree needs to be added to the experience buffer)
- The paper is clear and well-written, and theoretical details are given to motivate the approach, and justify the results.
- Section 4.4, the practical implementation, is very well-written and concisely gives all the details needed for implementing the algorithm

Weaknesses:

- I did not find any big weakness of this work, only minor comments.
- Starting from Equation 4, the notation mixes $\Delta$ and $\nabla$, without a (one-line) explanation of what these symbols mean. It is a bit weird to compute a TD-error from a gradient, so the equation looks like the $\Delta$ is also a gradient (the gradient of the Q update, not the update itself). I find this quite confusing and explanations of the notations would be welcome.

Question:

- For offline RL papers, I'm always curious of what happens if the proposed algorithm is used in an online setting. In the case of this paper, is it possible to, every so often, let $\pi_{\theta}$ execute an episode in the environment, add the resulting transitions to the experience buffer, and continue learning from that?
clarity,_quality,_novelty_and_reproducibility: - Quality: high. The proposed method is well-presented. It is theoretically analyzed in a sound way, and the empirical evaluation is thorough and well-designed (with strong baselines and challenging environments)
- Clarity: medium-high. The paper is easy to follow for someone very familiar with offline RL algorithms, but I have the feeling that a quick reader, or someone less familiar with offline RL, may find the paper less clear. For instance, neither the abstract nor the introduction mention the simple summary of the paper: "we modify how samples are sampled from the experience buffer to correct for out-of-distribution actions". Sentences like "utilizes sampling-importance resampling to execute in-sample policy evaluation" are only understandable by people already familiar with sampling-importance resampling.
- Originality: average. The proposed method seems novel, and sampling-importance resampling is not used often, but it is quite close to Prioritized ER (mentioned by the authors in the paper). This does not lower the value of the paper, though.
summary_of_the_review: The paper considers an important problem, proposes an elegant solution with a theoretical analysis, and empirically shows that the proposed method outperforms many baselines.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ## Summary

Offline RL methods often suffers from OOD state-action pairs and extrapolation errors. Most ORL approaches address this issue by penalizing the OOD state-action pairs and regularize policy to stay close to the behavior policy. This paper proposes an in-sample actor critic (IAC) method to perform in-sample policy evaluation via importance resampling which samples in-distribution transitions more often. The proposed method has lower variance than vanilla importance sampling based approaches. The paper provides empirical results on continuous control tasks like locomotion  and AntMaze domains.

Below I will list some of the strengths and weaknesses as indicated by the reviewers:

## Strengths
- The paper is well-written and easy to follow.
- Attacking an important problem in offline RL.
- The results seem to be strong on the tasks that the IAC was evaluated on.
- Interesting theoretical results and well-supported justification for the method.

## Weaknesses
- Limited technical novelty in terms of proposed algorithm or approach.
- More thorough analysis of the computational cost in the paper is needed.
- Results are only limited to continuous control problems. It is not clear how they would generalize to the discrete actions.
- Some unclear statements such as what is meant by the OOD actions.
- The algorithm relies on a behavior density estimation, which introduces additional computational costs and hyperparameters.

*Decision:* Offline RL is an important problem and this paper is attacking and important issue in offline RL with respect to extrapolation error and OOD actions. The experimental results are encouraging and the paper is very well-written. The authors did a great job addressing the concerns raised by reviewers. Thus I am recommending the paper for acceptance with the following recommended changes:

1. Please add a discussion about the training time and computational complexity of the approach. How much it differs in terms of FLOPS or GPU-hours for instance when compared against a One-Step RL approach. This was also raised by two of the reviewers.

2. Please clarify the unclear statements raised by the reviewers.

3. In the related work please cite [1] which came before (Brandfonbrener et al., 2021) and proposed one-step RL methods for domains with discrete actions. (Brandfonbrener et al., 2021) work extended it to the continuous control domains as they mentioned in their paper. Also [2]   does an in-sample offline actor-critic approach via a hybrid approach by training the value function as a BVE to estimate the behavior value but still uses v-trace to correct the off-policiness in the policy updates. Please discuss [2] in the related works as well.

### References

[1]: Gulcehre, C., Colmenarejo, S.G., Wang, Z., Sygnowski, J., Paine, T., Zolna, K., Chen, Y., Hoffman, M., Pascanu, R. and de Freitas, N., 2021. Regularized behavior value estimation. arXiv preprint arXiv:2103.09575.

[2]: Mathieu, M., Ozair, S., Srinivasan, S., Gulcehre, C., Zhang, S., Jiang, R., Le Paine, T., Zolna, K., Powell, R., Schrittwieser, J. and Choi, D., 2021, October. StarCraft II Unplugged: Large Scale Offline Reinforcement Learning. In Deep RL Workshop NeurIPS 2021.
justification_for_why_not_higher_score: The limited technical novelty and experiments are restricted to only continuous control domains.
justification_for_why_not_lower_score: The paper is very well-written, the method is well-justified with some interesting theoretical and empirical results. The authors did a great job at addressing the concerns raised by the reviewers during the rebuttal. This paper could be an interesting contribution to the ICLR community interested in offline RL.
Decision: Accept: poster