Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method to leverage information from expert traces on an MDP M to help solve a sparse reward task on an MDP M', where M' differs from M only by its reward.  The method involves performing policy evaluation on expert traces on M to calculate a value function V_M, and then leveraging that value function as a potential function for reward shaping on M'.  Experiments are performed both in simulation and on a physical robot for variants of an insertion task, where the sparse-reward variants of the task are shown to be solved faster than with existing methods thanks to the use of information from an expert solving a different instance of a similar task.
strengths_and_weaknesses: This paper is well written and easy to follow, and the ideas are easy to understand.  The experiments are convincing and clearly applicable to a robotics domain.  

I think the paper could be improved with some more pertinent baselines, and a bit more detail on the actual algorithm.  I am also dissapointed that the robot experiments are just a sim2real transfer of the simulator-learnt policy, whereas the whole point of this type of algorithm is to be able to perform sufficiently efficient sparse reward learning such as to be able to train directly on hardware.  Another concern is that this approach requires a dense reward for the original expert demonstrations, which is something that is generally not available in a realistic scenario.

The paper is also missing some import related work:
Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models [https://arxiv.org/pdf/2011.01298.pdf]
Reinforcement Learning from Demonstration through Shaping [https://openreview.net/forum?id=S145CVGuWB]
Deep Reward Shaping from Demonstrations [https://ieeexplore.ieee.org/abstract/document/7965896]

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I think the ideas in the paper, although well-executed, is not particularly novel.  Using a potential function from expert demonstrations to shape the reward for a given task is something that has been looked at on a couple of occasions.  The novelty of the paper is to look in particular at domain shift, and leverage the existence of a well-defined reward during the expert demonstration, 
issues:  I'm not convinced the 'sparse reward signal backward propagation blockage' is exactly what's going on, or if it is that would require further discussion.
I'm not sure I understand the point of using pure imitation algorithms such as GAIL as baselines, as it seems pretty obvious they would not be able to solve a novel task.  Perhaps I missed something here in how the setup is done, but I'm not sure these provide much insight beyond BC, there is no expectation they would work in this setup anyways.
If the authors could convince me of the novelty of the contribution, and the types of scenarios where one would have access to a high-quality reward on the expert demonstrations, that would allow the calculation of V, as well as perhaps baselines that also use reward shaping coming from different potential functions (perhaps inspired from some of the existing literature) this could potentially make for a stronger paper.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a learning from demonstration method to learn a certain task from demonstrations of a different task. Starting from a recent method that is able to learn from off-policy data using a replay buffer, they additionally include a shaped reward function to provide a learning signal even in failed trajectories. A key component of this shaped reward is explicitly setting actions that lead to out of distribution states to 0 reward. In their experiments, they show that their method achieves superior performance in two settings: (1) when environment dynamics differ (in their case changing the shape of a hole for an insertion task) and (2) when both environment dynamics and the reward differs (in their case randomly perturbing the location of the hole from the demonstrations). They also include a detailed ablation study.
strengths_and_weaknesses: Strengths
1. Rigorous experiments considering numerous baselines. The constructed baseline of extending SAC with the method of Vecerik et al. is a nice addition. Ablation studies illustrating the impact of model components is also good. Also show zero-shot sim-to-real transfer.
2. Methods section is clear, and the proposed method is simple and principled. 

Weaknesses
1. While the related work is thorough, I am missing the motivation here. Why not just use inverse RL? The authors state that inverse RL methods are not able to learn from environment rewards, but this is a bit of a misrepresentation, as most inverse RL methods are assuming that such a reward function is unavailable.
2. Explanation of "reward signal backpropagation blockage" is confusing to me. Section could be reworked to better explain the phenomenon. The corresponding figure 2 is also confusing, not sure if it is adding to the understanding.
3. No baselines considering inverse RL methods.
4. Only one type of task (object insertion) is considered. What about multi-step tasks? Navigation? I feel that the method may be more limited when considering more significant shifts between demonstrated and target task as the reward shaping mechanism seems to rely on the original value function learning a complete enough representation to transfer to new tasks, which is a strong assumption. 
5. No mention of how many demonstrations are used. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The experiments are well constructed with nice results, and the method is well-principled and elegant. However, as mentioned before, I do not completely see the motivation behind this method, and one of the main issues addressed, the blocked reward signal backpropagation, is unclear. Also, evaluation on only a block-insertion task is limited. If the authors can provide a better motivation and clarify the blocked reward signal section, I would be willing to change my score.
issues: Section 4.1 and Figure 2: Rewrite/update to better explain phenomenon of blocked reward signal backprop.
Intro/Related Work: Discussion of inverse RL and it's limitations could be improved
Occasional typos
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Learning from Demonstration (LfD) is a mechanism to help guide reinforcement learning (RL) algorithms when in the setting of learning with sparse rewards. This paper explores a problem with using LfD to guide RL when the demonstrators providing the LfD data are performing a different ("mismatched") task. The paper develops Conservative Reward Shaping from Demonstration (CRSfD), which estimates and provides a shaped reward to the RL agent to support intelligent exploration. Empirical results show that the approach outperforms baselines, and the paper provides a robot demonstration.
strengths_and_weaknesses: Strengths:
-The paper evaluates the proposed approach, CRSfD against numerous baselines, including both pure RL methods (SAC), pure LfD mechanisms (BC, GAIL, GAIfO), and hybrid appraoches (POfD, SQIL, and SACfD). 
-The evaluation shows consistently positive results for CRSfD (Figure 3) across a set of custom tasks. The evaluation shows CRSfD can outperform its most challenging competitor, SACfD, when multiple aspects of the task have changed.
-An ablation study is provided to show the relative contributions of various aspects of the algorithmic framework for CRSfD.
-A real robot demonstration is provided (Table 1).
-Code is provided, along with a video.

Weaknesses: 
-It would be helpful if the paper contained more details about the experimental domain (it wasn't until watching the video that it was clear what was going on) and why these tasks are the right tasks for benchmarking. 
-This paper evaluates two forms of task misspecification (i.e., the (1) target location and (2) geometry for an insertion task move). However, 
-The paper states that [22] "enforce[s] the agents to imitate the demonstrations only initially and then gradually discard them as learning progresses." However, in reading [22], it appears the method is grounded in the Preference-based Reinforcement Learning (PbRL) literature and generally does not imitate the demonstrations and gradually discard them. Rather, it takes a fundamentally different approach in which a reward function is learned based upon ranking data from humans, and a policy is optimized against that reward function. Later work by Brown et al. removed the need for rankings (Brown et al., CoRL'20), and Chen et al. then improved upon this method (CoRL'21). I think it would be helpful to benchmark SACfD against these approaches for learning from suboptimal demonstration.

Brown, D.S., Goo, W. and Niekum, S., 2020, May. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning (pp. 330-359). PMLR.

Chen, L., Paleja, R. and Gombolay, M., 2021, October. Learning from Suboptimal Demonstration via Self-Supervised Reward Regression. In Conference on Robot Learning (pp. 1262-1277). PMLR.

-Further limitations could have been discussed other than multi-task learning. What are the limits of the approach? When would it fail? Would this work for pick-and-place or other robot tasks? How does the approach degrade with the quality of the demonstration?
-The paper provides multiple figures (e.g., Figure 1 and Figure 2) that attempt to give intuition to the problem being studied. However, the paper spends very little time actually describing how the approach works (Lines 165-196). Pseudocode is relegated to the appendix. It would have been more helpful had the paper removed Figure 2 and spent more time describing the algorithm itself. The mismatched reward problem is relatively self-explanatory, and the "reward signal backward propagation blockage" doesn't seem to be a novel problem or even a problem that needs to be discussed at length. If a robot fails to accomplish a new task because it has learned a different one, it may not be able to receive any (sparse) rewards, so learning will be difficult. It isn't clear to this reviewer than anything beyond this point is being argued. Is the reviewer missing something? Also, is "reward backward propagation" the same issue than (though with different words -- perhaps a special case of) the canonical credit-assignment problem?
-Prior work has also considered reward shaping that should be considered:

Goyal, P., Niekum, S. and Mooney, R.J., 2020. Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards. arXiv preprint arXiv:2007.15543.

Questions:
-What would happen if one collected demonstrations on '4' but trained on '0' (or any other digit)?
-How could multi-task RL/IRL be helpful here? Consider:

Gleave, A. and Habryka, O., 2018. Multi-task maximum entropy inverse reinforcement learning. arXiv preprint arXiv:1805.08882.

Singh, A., Jang, E., Irpan, A., Kappler, D., Dalal, M., Levinev, S., Khansari, M. and Finn, C., 2020, May. Scalable multi-task imitation learning with autonomous improvement. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2167-2173). IEEE.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper develops an interesting reward-shaping approach to enable an RL algorithm to benefit from demonstrations that were provided for a different task. The results are positive; code and a robot demo are provided. However, the paper leaves out some important baselines (e.g., Chen et al., CoRL'21), does not devote enough attention to describing the technical method, and the limits of the algorithm are not sufficiently discussed/explored.
issues: Please address the issues of related work, clarity in the presentation of the algorithm, and improved discussion/exploration of limitations.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper studies how demonstrations can be used to improve the performance of online reinforcement learning (RL). In particular, the authors study settings where the demonstrated task does not exactly match the tasks which robot will have to execute. This removes the requirement of obtaining expert demonstrations for every new task the robot encounters. The main contribution is a conservative reward shaping algorithm for reinforcement learning with demonstrations from mismatched tasks. The work is evaluated in simulation and real-world experiments, including comparisons against state-of-the-art Learning from Demonstration (LfD) methods.
strengths_and_weaknesses: Overall, the paper is well written, clearly structured and addresses an important challenge in robot learning – using demonstrations from mismatched tasks in RL. The authors identify shortcomings of existing approaches, and propose a novel solution technique. The evaluation in simulation and real-world experiments deliver convincing results.
On the downside, the paper lacks detail in the theoretical part, and the evaluation is limited to only one experiment. Please see detailed comments below:

•	resolved -- The paper identifies a key issue for standard methods combining LfD with RL, called Sparse Reward Signal Backward Propagation Blockage. While the description and illustration are convincing on a high level, I would like to see more technical details on this.

•	resolved -- Related to above comment, the paper lacks a more rigorous description of how the use of reward shaping avoids the backward propagation blockage. Are there any theoretical guarantees on this?

•	resolved -- The limited theoretical discussion also raises questions about the overall contribution of the paper: is the use of reward shaping with demonstrations novel? I don’t think any related work for this was cited, yet it is not entirely clear to me. 

•	Updated -- The second main contribution is addressing the out-of-distribution problem. The approach for this is only described very briefly in lines 176-180, and leaves questions about the details: How does the regression work? How is the discount factor increased? 

Update: While the overall description has improved in the revised paper, the details about how the discount factor is updated are still unclear (including the extended algorithm in the appendix)

•	resolved -- What is the ‘Conservative’ part of the proposed technique? Is there a less or more conservative variant of the algorithm, and how does it compare? Since it appears in the name of the algorithm, it should be elaborated more.

•	Updated -- The parameter \lambda in equation 3 is not explained. 

Update: Similar to the discount factor, I am still missing details on how to choose this p.arameter

•	Updated -- While the evaluation is overall providing detailed insights and delivers promising results, the major weakness is that it only considers slight variations of a peg-into-hole problem. It could be significantly strengthened by being extended to a completely different robot task.

Update: the paper improved greatly by additional experiments with varying mismatch. Thus, the experiments are rich enough for a conference paper.

•	resolved -- An important question left unanswered is how mismatched the tasks can be. While it might be difficult to get specific theoretical results, empirical data would be interesting. I would expect that with increasing mismatch, the advantages of the proposed approach will start to disappear. For large mismatch, using information from the demonstrations might actually be detrimental to learning progress.


Minor comments:
•	resolved -- Typo line 77 Standard should be lower cas: standard 

•	resolved -- Line 156, shouldn’t the function map be  $\Phi : S \rightarrow \mathbb{R}$ not … \rightarrow R ?

Limitation Section:
•	resolved -- The limitation section only discusses joint learning for multiple tasks. While this is an interesting future work direction, it is only partially a limitation of the presented work – multiple tasks is not part of the problem statement. Some discussion of how to find the discount factor, if there are problems where the proposed method is not beneficial at all, or the influence how mismatched demonstrations are would be very interesting


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper discusses an interesting and relevant problem and is well structured. However, the technical presentation lacks some more detailed description and discussion and thus is not entirely convincing. 

Update: The technical description has improved through the revisions, and the experiments were extended which helped to show the robustness of the proposed work.
issues: 1) Resolved -- Add a theoretical discussion on how the proposed algorithm avoids Sparse Reward Signal Backward Propagation Blockage
2) Resolved -- Elaborate the description of the regression and discount factor (lines 176-180), as well as the conservative character of the approach.
3) Resolved -- Extend evaluation to at least one of the following: i) increasingly larger task mismatch, ii) entirely different problem (plots for ii could just be in the appendix).
review_title: null

Meta-review: metareview: This work proposes a learning from demonstration method that utilizes demonstrations from a task that is different from the target task to be learned via reinforcement learning. This removes the requirement of obtaining expert demonstrations for every new task the robot encounters. This is in an interesting problem setting and likely of relevance to the robotics community.  Specifically the authors propose a reward shaping algorithm for reinforcement learning with demonstrations from mismatched tasks. The work is evaluated in simulation and real-world experiments, including comparisons against state-of-the-art Learning from Demonstration (LfD) methods.

The authors have addressed most of the weaknesses outlined by the reviewers, and the reviewers agree that this work proposes a novel (incremental) algorithm, the manuscript is clear and evaluations are thorough (though the core contribution is evaluation in simulation only). Overall I recommend accept.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)