Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to incorporate prior expert (possibly sub-optimal) demonstrations in the learning procedure of any actor-critic algorithms. Furthermore, the authors propose also a modified target estimator for the critic updates, based on the maximum of an MC estimate and the standard estimators of the base AC algorithms. The resulting framework is fairly easy to implement and introduces no additional parametrization or computation at the cost of an initial set of demonstrations from the expert. The empirical evaluation show an improvement over SOTA RL algorithms on a set of continuous domains with sparse rewards.
strengths_and_weaknesses: The paper is clearly written and easy to understand. The authors propose a novel method to for Q-target correction, which is fearly easy to combine with current RL algorithms. The experimental evaluation shows significant performance improvement without additional computational cost. 

Overall I like the paper and I think it should be accepted.  The authors propose a simple method, thoroughly evaluate it and compare with appropriate baselines and discuss the limitations and shortcomings of the method. 

The only minor weakness I would point is the lack of clarity in the presentation of the confidence intervals in the plots. It seems like the confidence intervals are applied around the non-smoothed curves which in practice makes them difficult to understand, I would suggest to add the confidence intervals around the smoothed curves and remove the raw curves from the plot to improve readability.

The main weakness I see, is the requirement of demonstrations in sparse reward tasks. In these kind of tasks, often times (not always), it might be difficult to have demonstrations that achieve a high return.


questions: The main question I have is how important is the presence of the expert demonstrations in the replay buffer. Namely, have you measured the performance of the proposed algorithm only with the application of the MC targets in Eq. 5.4?

Related to this, a study on the effect of the sub-optimality of the demonstrations should be performed. How much is the performance impacted from more sub-optimal demonstrations. This could be studied in a simple environment, by varying the amount of noise added to the expert demonstrations. This is important, since the method, as the authors state, is to be applied exclusively to sparse reward environments, and often times in these environments it might be difficult to have demonstrations with a reward signal (e.g. goal based environments where a human might have difficulties in making a policy that reaches the goal)
limitations: The authors adequately address their limitations in the conclusions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an Actor-Critic method (MCAC) to resolve the sparse reward problem in deep reinforcement learning (single-agent cases). Technically, the algorithm starts from a replay buffer with demonstrations and performs policy evaluation with a modified $Q$-value function. Specifically, the policy evaluation is the maximum of the standard critic function and a Monte-Carlo estimation. The idea behind using two value estimations is that the combination may share the advantages of these two estimations, such as low variance and unbias. The implementation is given under some assumptions (such as "... the last observed reward value will repeat forever ...") to make the algorithm adapt to cases with an infinite horizon. The authors demonstrate experiments and comparisons with previous work on five continuous control tasks, in which the results show that MCAC outperforms all baselines.
strengths_and_weaknesses: ### Update After Rebuttal 

I thank the authors for their in-depth responses also the supplemented experiments. I am mostly convinced by the authors and have updated my score from 5 to 6.

### Strengths

1. this paper is well-organized and easy to follow.
2. this paper concentrates on an interesting problem in deep reinforcement learning and proposes a novel plug-in for Actor-Critic methods.

### Weaknesses

1. **Some typos and referring errors**: e.g., line 209: "... provided in Appendix ??"
2. **The demonstration is not so serious**: the demonstration lacks a comparison or investigation of the effectiveness of value function combinations. As the authors claim in this paper, they aim to bridge the advantages of the standard value function and Monte-Carlo methods to improve the accuracy/efficiency of policy evaluation (or value estimation). However, there is no explanation (at least empirically) for why this method can achieve such a target.
3. **Some technique flaws**:

    - As introduced in this paper, MCAC modifies the estimation of the target value function as Eq-(5.3). However, there may be a potential issue that such a method is biased as $Q^{target}$ and $Q^{target}_{MC-\infty}$ have different scales.
    - The authors approximate the Monte-Carlo estimation of an infinite horizon as Eq-(5.2). However, I am concerned about this approximation since the authors neither refer to some previous work to support the correctness nor give empirical results to explain it.
4. This point is more like a suggestion to highlight your motivation. As the authors claim that their method is the first application of Monte-Carlo estimation of the return-to-go to improve value estimation for online cases, while is no discussion about what will such an estimator bring to online cases. Thus, I think the authors can make a deeper discussion on that.
questions: 1. As the MCAC requires pre-training with offline datasets. So, Is there any requirement on the size of the dataset? I think it is critical to the algorithm's performance.
2. For $Q^{target}_{MC-\infty}$ in the infinite cases, why don't we use the critic to approximate the $\gamma^{T-j+1}\frac{r^i_T}{1-\gamma}$?

limitations: The authors figure out some unresolved limitations in the last section. Such as MCAC is not helpful and even hinders learning by overestimating Q-values in dense-reward cases. As I mentioned in the "weaknesses", the normalization of the maximum operation maybe necessary to handle this problem.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a simple yet effective method MCAC to assimilate the benefits of both TD Q-target and Monte Carlo Q-targets, which can be applied to any actor-critic algorithm to solve the sparse Reinforcement learning problem. The authors empirically prove the effectiveness of MCAC for both standard reinforcement learning and imitation learning.
strengths_and_weaknesses: Pros:
The method proposed is simple and works effectively on various tasks. For some tasks, MCAC achieves a pretty solid performance improvement whereas the baselines work very badly. The authors also provide some intuitive explanations for MCAC which I think is inspiring for future work. 

Cons:
However, my largest concern is that either the current experimental results or the intuitive explanations cannot justify the consistent effectiveness of MCAC, and generally, I think this work does not show enough technical contribution. This method is just a simple extension the idea in [1]. Besides, the writing of this paper is very messy with many missing section references. I see the authors want to give some ablation study in section 6.4 but that part is finally in the appendix.  

[1] Robert Wright, Steven Loscalzo, Philip Dexter, and Lei Yu. Exploiting multi-step sample trajectories
419 for approximate value iteration. In Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and
420 Filip Železný, editors, Machine Learning and Knowledge Discovery in Databases, pages 113–128,
421 Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-40988-2.
questions: For the experimental parts, the authors claim in Figure 2 that when GQE already achieves very strong performance, MCAC may have little impact. I feel confused about the criterion of "strong performance". If strong performance refers to the performance being higher than the demos, we can still see that on the Sequential Pushing, although GQE achieves strong performance, MCAC can still accelerate the learning. 

I also have a similar concern for Figure 3 about OEFD, the criterion for "unable to learn efficiently or have significant instability".

Another stuff is for Figure 3 in the appendix, I cannot see a clear pattern as $\lambda$ increases (e.g., block lifting), I feel the results are still dominated by the randomness rather than the hyperparameters.

My general intuition is that MCAC is not shown to consistently take effect in these 5 environments, I suggest the authors give some theoretical proof and more experimental results on other domains.
limitations: The authors address both societal impacts and limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a modified Actor-Critic algorithm (MCAC) for sparse reward problems. This modification is achieved by initializing the replay buffer with demonstration and computing the modified Q-value by combining temporal distance target and Monte Carlo estimation. The method is evaluated in five environments, and performance is compared with a few existing methods. 
strengths_and_weaknesses: ### Strength:
- Tackle an important problem of learning from sub-optimal demonstration in sparse environments.
- Relevant literature is discussed, and a few baselines are compared in five environments.

### Weakness (Details in Questions):
- Lack of empirical evidence of the advantage of the proposed method
- Some details of the method design and experiments are missing.

questions: Standard RL algorithms usage demonstration (line 297) and is then compared with the MCAC version. However, the standard RL is not equipt to deal with such a demonstration, and having some sub-optimal demonstration might worsen the performance. Thus, RL algorithms without the sub-optimal demonstration should be compared as well.
 
In figure 3, MCAC barely improves performance in all the environments; in most scenarios, performance remains the same as the starting position (initial timesteps); why is that the case, especially AWAC and AWAC+MCAC?

Additionally, the results seem puzzling ( Figure 3); the AWAC results start with a high reward and then fall. In the cases where it recovers (with MCAC) and barely reaches the initial performance again. Thus though MCAC makes training stable, it barely gives any reward-level advantage. This improvement is the core of this paper. Though on OEFD, MCAC is performing better; overall, this method appears to be not as good as AWAC-based methods.

How was the demonstration collected? Was it collected by programming (automatic) or human-collected (e.g., line 239)? Adding zero-mean noise might make it a bit realistic; however, it may be beneficial to have humans collect data, which would be more realistic. 

I understand that the paper focus on suboptimal demonstration (e.g., line 258); however, is the proposed method tied to such an assumption? How does the performance stand with near-to optimal demonstration, which is often the case of expert (human) demonstration?

Looking at Figures 2 and 3, it seems the demonstration-based method performs worse than the standard-baseline variant (e.g., GQE in Figure 2). It seems that the demonstration is not helping in these scenarios. Further discussion on this issue would help clarify the advantage of the proposed method. 

The MCAC has two modifications, namely, replay buffer and Q-value modification. An ablation study on them would further clarify their significance.

There are many missing references (“?”) throughout the paper. A careful pass should fix them. There are too many environmental descriptions in the main paper, which can go to the Appendix.

limitations: Yes. Limitations and potential societal impacts are discussed.
review_title: null

Meta-review: metareview: This paper proposes two modifications of Actor-Critic algorithms for reinforcement learning with sparse reward: use demonstrations to seed the replay buffer, and combine TD Q target and Monte Carlo Q-target. All the reviewers agree that this paper has made good progress in an important research direction. The proposed method is simple-to-implement and seems to significantly boost the performance of the actor-critic methods. Most of the concerns in the original reviews were addressed through the rebuttal and discussions. However, one common concern that were raised by multiple reviewers, the lack of empirical/theoretical evidence of bias & variance reduction, remained unsolved. We understand that rigorous theoretical analysis for Deep RL algorithms, in general, is challenging. However, empirical analysis by conducting numerical experiments on simpler toy problems would still significantly improve the quality of this paper.
confidence: Certain
Decision: Accept