Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
Authors: Yanjiang Guo
Jingyue Gao
Zheng Wu
Chengming Shi
Jianyu Chen
Venue: CoRL 2022 Poster (2022)
Keywords: Sparse Reward Reinforcement Learning
Learn from Demonstration
Task Mismatch

Abstract: Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Learning from demonstration (LfD) is an effective way to eliminate this problem, which leverages collected expert data to aid online learning. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new data for every new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Such setting can be challenging and we found existing LfD methods may encounter a phenomenon called reward signal backward propagation blockages so that the agent cannot be effectively guided by the demonstrations from mismatched task. We propose conservative reward shaping from demonstration (CRSfD), which shapes the sparse rewards using estimated expert value function. To accelerate learning processes, CRSfD guides the agent to conservatively explore around demonstrations. Experimental results of robot manipulation tasks show that our approach outperforms baseline LfD methods when transferring demonstrations collected in a single task to other different but similar tasks.

Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
Yanjiang Guo', Jingyue Gao', Zheng Wu², Chengming Shi', Jianyu Chen 1.3,
Institute for Interdisciplinary Information Sciences, Tsinghua University University of California, Berkeley
Shhnghaii Qizhi Institute [guoyj22, gaojy 19, 19) @ wawiissttingua.euuu www.ua.edduccc. zheng_wo @ berkeley.edu Correspondence to: Jianyu Chen (jaanycchen @ tsinghua.edu.cn)
Reinforcement learning often suffer from the sparse reward issue in real-world robotics problems. Learning from demonstration (LfD) is an effee- tive way to eliminate this problem, which leverages collected expert data aid online learning. Prior works often assume that the learning agent and the expert aim to accomplish the same task, which requires collecting new task. In this paper, we consider the case where the target task is mismatched from but similar with that of the expert. Such setting be challenging and we found existing LfD methods can not effectively guide learning in mismatched new tasks with sparse rewards. We propose conservative reward shaping from demon- stration (CRSID), which shapes the sparse rewards using estimated expert value function. To accelerate learning processes, CRSID guides the agent conserva- tively explore around demonstrations, Experimental results of robot manipulation tasks show that our approach outperforms baseline LfD methods when transfer- ring demonstrations collected in a single task to other different but similar tasks.
Abstract:
data for every
Keywords: Sparse Reward Reinforcement Learning, Learn from Demonstration, Task Mismatch
Introduction
Reinforcement learning has been applied various real-world tasks, including robotic manipulation with large state-action spaces and sparse reward signals [1]. In these tasks, standard reinforcement learning tends to perform lot of useless exploration and easily fall into local optimal solutions. To eliminate this problem, previous works often use expert demonstrations to aid online learning, which adopt some successful trajectories to guide the exploration process [2. 3].
However, standard learning from demonstration algorithms often assume that the target leaning task exactly same with the task where demonstrations collected [4, 5, 6]. Under this assumption, experts need to collect the corresponding demonstration for each new task, which can be expensive and inefficient. In this paper, we consider new learning setting where expert data collected under single task, while the agent is required to solve different new tasks. For instance as shown in Figure 1, a robot arm aims solve peg-in-hole tasks. demonstration is collected on a certain type of hole while the target tasks have different hole shapes (changes in environmental dynamics) or position shifts (changes in reward function). This can be challenging as agents cannot directly imitate those demonstrations from mismatched tasks due to dynamics and reward function changes. However, compared to learning from scratch, those demonstrations should still be able to provide some useful information to help exploration.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Trajectories
Collect
Transfer
New tasks...
Demonstrations for task 0
Task 0
Task 0
Shape change
Position shift
Figure Illustration of our motivation. Demonstrations collected on a single original task are transferred to other similar but different tasks with either environmental dynamics changes (shape change) or reward function change (position shift), and aid the learning of these tasks.
To address the issue of learning with demonstrations from mismatched task, previous works in imitation learning consider agent dynamics mismatch and rely on state-only demonstrations [7. 8, 9]. However, this approach has an implicit assumption that the new task share the same reward function the original task [10]. Hester al. and Vecerik et al. [11, 3] receive sparse rewards in the environment and add demos into prioritized replay buffer. Sparse reward signal can be backward propagated during the Bellman update and thus guide the exploration. However, this propagation flow may be blocked due to the mismatch in new tasks. Another class of work [12, 13, 14] also considers that we have expert data multitasks and utilize meta-learning methods obtain diverse skills, and then transfer skills to new tasks. However, such strategy requires to collect huge expert dataset, which is expensive and inefficient. In our setting, we are only provided with a few demonstrations collected under single task.
In this paper, propose Conservative Reward Shaping from Demonstration (CRSID), which learns policies for new tasks accelerated by demonstrations collected in single mismatched task. We use reward shaping [15, 16] incorporate future information into single-step rewards while keeping the optimal policy unchanged. Moreover, we explicitly deal with out-of-distribution problem to encourage agent explore around demonstrations, Experimental results of robot manipulation tasks show that our approach outperforms baseline LID methods when learning in new tasks with mismatched demonstrations.
Our contributions can be summarized as follows:
We proposed reward shaping scheme for reinforcement learning with demonstration from mismatched task, which use estimated value function from expert demonstrations to re- shape sparse reward in tasks.
Built upon such scheme, propose the conservative reward shaping from demonstration (CRStD) algorithm to overcome the outto--distribution problem, we regress value function of OOD states to zero and use a larger discount factor in new tasks, which guides the agent to conservatively explore around expert data.
We conduct simulation and real world experiments of robot insertion tasks with mis- matched demonstrations, The results show that CRSID effectively guide the exploration process in new tasks and reach a higher sample efficiency and convergence performance.
2 Related Works
Learning from demonstration A prominent research subject is how to leverage expert data as- sist reinforcement learning. Imitation learning (II.) a broad family of such algorithms that enforce agents to directly imitate the expert. Behavior cloning (BC) is the simplest IL algorithm which

---

greedily imitates the step-wise action of the expert and can fall into the problem of distributional shift [17]. Inverse reinforcement learning [18, 4] and adversarial imitation learning [6] infer the ex- pert's reward function and learn the corresponding optimal policy jointly. The above II. algorithms assume environment rewards are not available, hence their performances are upper-bounded by that of experts [19]. Another line of work makes use of reward feedback from environment and lever- ages expert demonstration data to overcome the sparse reward issue or learn more natural behaviors. Vecerik et al. [3] add demonstration into a prioritized replay buffer. Rajeswaran et al. [20] add a behavioral cloning loss to the policy to speed up exploration and learn more natural and robust behaviors. Chen et al. [21] use generative models on single step transition to reshape reward of the original task. However, standard learning from demonstration algorithms always requires demon- strations to be collected under the same task and act nearly optimal under this task, which is not suitable for our setting.
Generalization of demonstrations There are few works relaxing the requirements for demonstra- tions to achieve generalization of demonstrations from different aspects. Some works assume that demonstrations are collected by sub-optimal policy under the same task [22, 23], early work [23] requires manually ranking of trajectories and later works [24, 25] move the needs for rankings by actively adding noise to demonstrations along with automatical ranking. Cao et al. [26, 27] assume that the demonstrations are a mixture of different experts and use a classifier to separate out the more feasible expert data for the new task. Other works [10, 28, 29] assume that the target task has different agent dynamics to the task where demonstrations are collected, so they only match the state sequence of demonstrations use inverse dynamic model to recover the action between two states in the new task. In our work, we further consider new tasks with the environment dynamics mismatch as well as reward function mismatch. Another branch of related works are meta imitation learning algorithms, which assume that we have expert data on multitasks and utilize meta-learning methods solve new tasks in zero-shot or few shots adaption [12, 13]. However, such strategy usually necessitates huge expert dataset which may be expensive and inefficient. Differently, we consider the problem where only small number of demonstrations collected in single task are provided, and the agent needs to use them to accelerate the learning of other similar but different tasks.
3 Problem Statement
In our problem setting, we have collected a few demonstrations under single task and want to utilize these data in reinforcement learning for other similar but different tasks. A task can be formalized as a standard Markov decision process MDP, which is modeled as M; = (S. A, P., R2, 1)). The task where demonstrations are collected denoted as Mo = (S, A, Po. Ro, 70), and the new tasks we target to solve are denoted as M; (S, A, P., RA 7) 2 1. S and A are the shared state space and action space for each task. P; : S x A S - [0. 1] are state transition probability functions of each task, R, 5 A x S > R reward functions for task M,, describing the natural reward signal in each task. Due to differences of environment and agent dynamics, P. and R, often varied between different tasks. % is discounted factor of M, which reflects how much we care about future, typically set to a constant slightly lower than 1. A policy n; : S > A defines probability distribution in action space. For a task M, and policy Ti state value function (s) = E.,=s,7, [En) Re(se, a, $141) estimates the discounted cumulative reward of the task under this policy n.. Vi (s) estimates the discounted cumulative rewards for state under the optimal policy π..
As many works [30, 31] point out, directly applying RL in sparse reward environment can be sample inefficient and fail to find good solution. In this work, we want to make use of the demon- strations D (To, .I....) collected in task Mo to facilitate reinforcement learning for the different but similar new tasks M,. Note each trajectory т. contains a sequence of state action transitions [So, ag $1, a1 ...3, a,] in task Mg.
Challenges There are two key issues when leveraging demonstrations from mismatched tasks. First, how to get effective guidance from these mismatched demonstrations? Although we should not purely imitate these demonstrations, we do need to obtain some useful guidance from them to ac- celeration exploration in new tasks with sparse rewards. Second, since our goal is to maximize the reward defined under the new task, guidance from mismatched demonstrations should not influence the optimality of the learned policy in new tasks.
3

---

4 Conservative Reward Shaping from Demonstrations
Provided with demonstrations in a particular task Mo A, Po: Ro, 70). we aim to help the re- inforcement learning process of different tasks M1, M2....K,, which may have different transition functions (s a) and reward functions [s a, s'). In this work, we use SAC [32] as our base reinforcement learning algorithm as it holds an excellent exploration mechanism which leads to higher sample efficiency than policy gradient algorithms [33, 34] and is shown to perform well on continuous action tasks [32]. Nevertheless, it is also possible to base our method on other RI al- gorithms including on-policy ones. To make use of expert demonstrations, DDPGfD [3] proposes a mechanism compatible with the off-policy method, which adds the demonstration data into the replay buffer with prioritized sampling. Under such framework, the sparse reward signal can prop- agate back along the expert trajectory to guide the agent. By combining SAC and DDPGID [3], we obtain the backbone of our method and labeled as SACID, which is also our best baseline method.
4.1 Reward Conflict under Mismatched Task Setting
Although LID methods such as SACID benefit from demonstrations in sparse reward reinforcement learning, they may not benefit from demonstrations when the target tasks are mismatched from that of the expert. When following the demonstrations, agent may consistently fail and can not get any sparse rewards signals. As failure time increases, agent may consider expert trajectories to have low value since few rewards are received. The agent will then avoid following the expert and the demonstrations cannot provide effective guidance, resulting in inefficient exploration in the whole free space.
Although totally following the demonstrations may not be able to receive any sparse reward in new tasks, it can still provide useful exploration directions since in our settings the new tasks are similar to the original one. We formally introduce method as conservative reward shaping from demon- stration (CRSID). Intuitively, CRSID assigns appropriate reward signals along the demonstrations to efficiently guide the agent towards the goal, and allows exploration around the goal to maintain optimally. Details are described in the following subsection.
4.2 Conservative Reward Shaping from Demonstrratoos((RRS)))
Reward Shaping with Value Function Reward shaping [15] provides an elegant way to modify re- ward function while keeping the optimal policy unchanged. Given original MDP M and an arbitrary potential funetion &b 5 R. we can reshape the reward function to be:
R' R'(s, a, s') = R(s, a, s') - xb(ss) D(s), ~ a()
(1)
Denote the new MDP as M' (S, A, P. R², 7) obtained by replacing reward function R in M to R'. Ng et al. [15] proved that the optimal policy Tis. on M' and the optimal policy म i on the original MDP Mo are the same: The- T- Furthermore, the optimal state-action function Qir a) and value function Vin (8) are shifted by (()):
(s, a) (s,(s, a) (s), (s) (s) (s))
In particular, Ng et al. [15] pointed out that when the potential function is chosen as the optimal value function of the original MDP (s)) V (s), then the new MDP M' becomes trivial to solve. What remained for the agent is to choose each time-step's 's action greedily, because the transformed
single-step reward already contains all the long-term information for decision making.
Conservative Value Function Estimation The reward shaping method provides a principled way to guide the agent with useful future information and keep the optimal policy unchanged. Ideally, an accurate 4 (s) VM, (s) will lead to simple and optimal policy in new MDP M', but a perfect (s) Vi, (s) is unavailable in advance. Practically, we estimate V. ~ Vir, (s) using demon- strations from task Mo by Monte-Carlo regression and treat V.D, (s) as a prior guess of VM. (s). We then shape the sparse reward in the new task M, to:
(, a, s') R4(s, a. s') + V. V (s)
(3)
However, demonstration trajectories only cover a small part of the state space. For out-of- distribution states, estimated 1/ may output random values and lead to random single-step re- ward after reward shaping, which may mislead the agent. We make two improvements over the above reward shaping method to encourage the agent to explore around the demonstrations conser-

---

Ma (s) of the out-of-din distirbrinttion states to 0, thus discouraging exploration far from demonstrations. The OOD states are sampled randomly from free space. (2) In- creasing the discount factor in new tasks. From equation 3, we can find that increasing 7 will give higher single-step reward for state with large Vo(s') the original task, thus encourages exploration around demonstrations. Our method can be summarized as follows: (D stands for demonstration buffer, S stands for free space, 7/4 70)
vatively: (1) Regress value function
Algorithm Conservative Value Function Estimation
Input: Demonstration transitions, demo discount factor Yo. new task discount factor Ya(7k
7oo). regression steps scale factor X.
Initialization: Initialize value function Vo(s)
Monte-Carlo policy evaluation on demonstrations, Calculate cumulative reward for states in de- mos using "oo V (s) Σ =77-700 (r=;;
for in regression steps n1, do
Sample minibatch B1 from demo buffer D with regression target V (s) - Σ "r. Sam-
ple minibatch B2 from whole free space with regression target O.
perform regression: 0 arg arg min Eer=B (Vo(s:) 20- rr)2 + AB (Vo(se) - -0)
8
end for
Shaping reward with yk: (s, a, 8') R,(s, a, 8') 7h Vo('') Vo(s).
Perform SACID update. (detalis can be found in appendix.)
Conservative Properties In the last paragraph, we introduced some conservative techniques and give some intuitively explanations why those improvements can encourage exploration around demonstrations under the proposed reward shaping framework. The following theorem can quantize the benefits of proposed methods.
Theorem For task My with transition To and new task My with transition Tk. define to- tal variation divergence Dry(s,a) a) Eg [To(s) s, (T(("(s,,)) a)l 8. If we have ô < (Th 'r0 " (V (s') max.' Vil (8), then following the expert policy in new task will result in immediate reward greater then 0:
(s, 2 (Tx T0 (eo (s') 70d max VD (8') >
(4)
Detailed proof can be found Appendix 7.5. The above theorem indicates that for similar but different tasks (8 smaller than the threshold), exploration along demonstrations will lead to positive immediate rewards which guide the learning process.
Conservative Reward Shaping from Demonstrations After reward shaping by demonstrations from mismatched task, we perform online learning based on SACID as described in Section 4. Pseu- docode can be found in supplementary materials. Although the estimated V (s) can be inaccurate, it still provides enough future information, thus facilitates exploration for the agent. Moreover, nice theoretical properties of reward shaping guarantees that we will not introduce bias to the learned policy in new tasks.
CRSfD(ours)
SACID
BC
SAC
GAIL
GAIFO
POID
SQIL
RS-GM
1.0
1.0
1.0
1.0
0.8
0.8
0.4
8.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2,0 2.5 3.0
Samples[1e5]
Sa Samples(ae))
Samplen(1a5)
Samplen(
(a) Hole "1"
(b) Hole "2"
(c) Hole "3"
(d) Hole "4"
Figure 2: Evaluation of algorithms on 4 new tasks with demonstrations from task "0". The solid line corresponds to the mean of success rate over 5 random seeds and the shaded region corresponds to the standard deviation. Y-axis reflects success rate range in [0, 1]. X-axis reflects interaction steps range [0, 3e5].
5

---

5 Experimental Results
We perform experimental evaluations of the proposed CRSfD method and try to answer the follow- ing two questions: Can CRSID help the exploration of similar sparse-rewarded tasks with demon- strations from a mismatched task? Will CRSID introduce bias to the learned policy in new tasks?
We choose the robot insertion tasks for our experiments, which has natural sparse reward signals: successfully inserting peg into hole get reward +1, otherwise O. We perform both simulation and real world experiments. The simulation environment is built under robosuite framework [35] pow- ered by Mujoco physies simulator [36]. We construct a series of similar tasks where the holes have different shapes and unknown position shifts, reflecting changes in dynamics and reward functions respectively, as shown in Figure 1. Then we verify the effectiveness of CRSfD under the following settings: (1) Transfer collected demonstrations to similar insertion tasks with environment dynamics mismatch. (2) Transfer collected demonstrations to similar tasks with both environment dynamics and reward function mismatch. Finally, we address the sim-to-real issue and deploy the learned pol- icy on a real robot arm to perform insertion tasks with various shapes of holes the real world. We use Franka Panda robot arm in both simulation and real world. The comparison baseline algorithms are chosen as follows:
Behavior Cloning [17]: Just ignore the task mismatch and directly perform behavior cloning of the demonstrations.
SAC [32]: A SOTA standard RI. method which does not use the demonstrations and di- rectly learn from scratch in the target tasks.
GAIL [6]: Use adversarial training to recover the policy that generates the demonstrations, which allieviates the distributional shift problem of behavior cloning.
GAIPO [8]: A variant of GAIL which trains a discriminator with state transitions (s, s') instead of (s, o.) in GAIL to alleviate dynamics mismatch.
POrD [37]: A variant of GAIL which combines the intrinsic reward from discriminator and extrinsic reward from the new task.
SQIL [38]: An effective off-policy imitation learning algorithm that adds demonstrations with reward +1 to the buffer and assign reward 0 to all agent experiences.
Incorporate effective demonstration replay mechanism from [3] with SAC as described in Scction 4, which is also the best baseline as well as the backbone of our method.
SACID [32, 3J:
RS-GM [21]: Reward Shaping using Generative Models, which is an extension on discrete reward shaping methods [39, 40J. After learning a discriminator Dz(s, a), they shape the reward into Rt' (s, a) R(s, a) + > (s', a') (s,
5.1 Simulation experiments
We set a nominal hole position as the original-poii of our Cartesian coordination. Observable states include robot proprioceptive information such as joint and end-effector position and velocity. Action space includes the 6d pose change of the robot end-effector in 10 Hz, followed with a Cartesian impedance PD controller rumning at high frequency. Only sparse reward +1 is provided when the peg is totally inserted inside the hole. Demonstrations are collected by a sub-optimal RL policy trained with SAC in task Mo under carefully designed dense reward, where the hole has shape "0". This process can be replaced by manual collection in the real world. Then demonstrations are tagged with the corresponding sparse reward. We collected 40 demonstration trails with 50 time steps each.
Setting 1: Tasks with environment dynamics mismatch. To reflect environment dynamics changes of the tasks, we create experiments domain on insertion tasks with holes of various shapes in the simulator, represented by different digit numbers, as shown in Figure 1. Different shapes of holes will encounter different contact mode thus lead to different environmental dynamics. We collect demonstrations from hole "0", and our method use them to help training similar new tasks with various hole shapes from digit to 4.
Analysis The comparison results of CRSID and baseline algorithms under the above setting are show in Figure 2. As we expected, the simplest BC algorithm simply imitates the expert action of

---

the original task and can only complete the insertion with small chance. The SAC algorithm does make use of the demonstration data and conducts lot of useless exploration, which leads to poor performance. GAIL algorithm and its variants GAIfO, POfD also fail for most of times as they try to purely imitate the demonstration collected the mismatched task. SQIL ignores the reward in the new task and only obtains limited success rate. SACfD) can not be effectively guided by demonstrations from the mismatched task under sparse reward. Our proposed CRSID provide guidance through reward shaping, and consistently achieves the best performance on all the four insertion tasks with different hole shapes.
Setting 2: Tasks with both dynamics and reward function mismatch Next, we consider more challenging scenarios where we aim transfer the demonstrations to new tasks with both environ- mental dynamics mismatch and reward function mismatch. We assume that the hole has unknown random shifts relative to the nominal position, thus the reward function changes. At the beginning of each episode, the hole is uniformly initialized in square area centered at the nominal position. This can be challenging because the robot 'blind' these unknown offsets and requires further search for the entrance of the hole. Practically, we collected demonstrations from task with hole "0" with fixed hole position, and transfer to new tasks with random hole shifts and different hole shapes.
Original task Environmental
New tasks
dynamicr changes
CRSID(Ours:
Shupe "2"
SACID
random shift
Shupe "("
Original task
4mm=4mm random shift
8mm 4mm random shift
Reward function changes
random shift
random shift
Figure 3: Evaluations of CRSID and the best baseline SACID. The solid line corresponds to the mean of success rate over 3 random seeds and the shaded region corresponds to the standard de- viation. X-coordinate reflects changes in reward functions and Y-coordinate reflects changes in environmental dynamics. Our algorithm outperforms bascline with increasing margins the task changes become larger.
Analysis 2: We compare algorithm with the best baseline algorithm SACID under varying de- grees of environmental dynamics and reward function changes, as shown in the Figure 3. Due to space limit, more comparison can be found in Figure ?? in appendix. The x-coordinate represents the increasing changes of the reward function, where the random range of the holes becomes larger The y-coordinate represents increasing environmen-
(from 4mm+ 6mm*6mm, tal dynamics change, from hole "0" in its original shape to hole "2" in a different shape. Straight- forwardly, coordinate origin can represent the original task where demonstrations are collected, and a 2d coordinates [r. y] represents new task with varying degree of mismatch.
From Figure 3, we can observe that when applying to the original task very similar task such as [4mm, shape 'O'], our method has a similar performance to the SACíD baseline. When the task changes become greater (e.g. [8mm, shape '0'], [4mm, shape '2'], [6mm, shape '2'], [8mm, shape "2')), SACID gradually lose the guidance from original demonstrations as task mismatched more significantly, while CRSID achieves significant performance gains with help of the conservative reward shaping using estimated value function.
Ablation study As mentioned in section 4.2, we make two improvements over the reward shap- ing method to encourage the agent to explore around the demonstrations conservatively. (1) Regress value function of OOD states zero. (2) Use larger discount factor in new tasks.
7

---

We ablate these 2 improvements and compare their performance. Ab- lations are tested under new task with hole shape "3". results for other shapes can be found in the supplementary materials. As shown in Figure 4, compared to original CRSfD algorithm, moving away either of these
1.0
Success Rate
0.6
2 techniques will lead to a performance drop, where the agent needs to take more effort in exploration.
0.4
CRSID
(1)
(1)(2)
0.0
5.2 Real World Experiments
0.0
0.5 1.0
1.5 2.0 2.5 3.0
Samples(1e5)
After completing the insertion tasks of various-shaped holes in the sim- ulator, we deploy the policy the real robotic arm. To overcome the sim-to-real problem, we use domain randomization in the simulation. The initial position of the robot arm end-effector and holes are random- ized in a 6cm 6em"6cm" "6cm"6mm 66m space and 4mmm2mm plane respectively, and the friction coefficient of the object is also randomized in [1. 2]. We use real Franka Panda robot arm and 3d print the holes corresponding to digit numbers "0-4". Holes are roughly sizes 4cm*4cm, with Imm clearance between the peg and the hole. We performed 25 insertion
Figure 4: Ablation stud- ies of the conservative- ness techniques. means regressing value function to zero for OOD states. (2) means setting
(1)
larger discount factors.
trials under each shape hole, and counted their success rates separately, as shown in the table The robot achieves high success rate in all tasks.
Hole in simulation
Hole
Success
During insertion
Shape
Rate
3D Print
Initial
Digit "0"
1.0
Digit "1"
1.0
Start
Finish
0
2
3
Digit "2"
0.92
Digit "3"
0.92
Digit "4" 0.96
Table 1:
Success
Figure 5: Real world robot insertion experiments.
for real world robot in- sertion tasks.
6 Conclusion
Summary. In this paper, we studied the problem of reinforcement learning with demonstrations from mismatched tasks under sparse rewards. Our key insight is that, although should purely imitate the mismatched demonstrations, we can still get useful guidance from the demonstrations collected in similar task. Concretely, proposed conservative reward shaping from demonstra- tions (CRSID) which uses reward shaping by estimated value function of mismatched expert to incorporate useful future information to augment the sparse reward. with conservativeness tech- niques to handle out-of-distribune issues. Simulation and real world robot insertion experiments show the effective of proposed method under tasks varied environmental dynamics and reward functions.
Limitations and Future works. Provided with demonstrations from mismatched task, our pro- posed method aids the online learning process for each new task separately. However, one may need learn policy to solve multiple
tasks the same time, and exploration these tasks may benefit each other. So future works include using demonstrations to accelerate the joint learning pro- cess of multiple tasks. Another limitation is that our method is only applicable to new tasks similar to original task. The effectiveness of CRSID gradually decays when the tasks differ too much from the original task so that the demonstrations do contain any useful information. also worth to mention that the whole algorithm pipeline should be able to be implemented directly on hardware, which is promising research direction.

---

Acknowledgments
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0150000);
References
[1] O. M. Andrychowicz, В. Baker, M. Chociej, Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[2] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6292-6299. IEFF, 2018.
[3] M. Vecerik, T. Hester, Scholz, F. Wang. O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv: T:X:x:/707... 100000077777777 2017.
[4] B. D. Ziebart, A. L.. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. In Aaai, volume 8, pages 1433-1438. Chicago, II., USA, 2008
[5] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Pro- ceedings of the twenty-first international conference Machine learning, page 1, 2004.
[6] Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[7] Y. Schroecker and C. L. Isbell. State aware imitation learning. Advances Neural Information Processing Systems, 30, 2017.
[8] Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.061 00000700618.. 2018.
[9] W. Sun, A. Vemula, B. Boots, and D. Bagnell. Provably efficient imitation learning from observation alone. In International conference machine learning, pages 6036-6045. 6036-6055.PMLR, 2019.
[10] T. Gangwani and J. Peng. State-only imitation with transition dynamics mismatch. arXiv preprint arXiv:2002.11877 11879, 2020.
[11] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[12] V. H. Pong, A. Nair, L. Smith, C, Huang, and Levine. Offline reif-reiinorremmett learning with online self-supervision. arXiv preprint arXhv:2107.03974, 2021
[13] Z. Zhao, Luo, Sushkov, R. Pevceviciute, Heess, J. Scholz, Schaal, and Levine. Offline meta-reiifforemment learning for industrial insertion. arXiv preprint arXiv:2110.04276. 2021.
[14] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100, PMLR, 2020.
[15] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278-287, 1999.
[16] C.A.. Cheng, A. Kolobov, and A. Swaminathan. Seuristic-guided reinforcement learning- Advances in Neural Information Processing Systems, 34, 2021.

---

[17] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[18] A. Y. Ng. S. J. Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2000.
[19] D. Rengarajan, G. Vaidya, A. Sarvesh, D. Kalathil, and S. Shakkottai. Reinforcement learning with sparse rewards using guidance from offline demonstration. arXiv preprint arXiv:2202.04628, 2022.
[20] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint
2017.
[21] Y. Wu, M. Mozifian, and Shkurti. Shaping rewards for reinforcement learning with imperfect demonstrations using generative models. In 2021 IEEE International Conference on Robotics IEEE, 2021.
and Automation (ICRA), pages
[22] Y. Gao, H. Xu, J. Lin, F. Yu, S. Levine, and T. Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:/802.088
2018.
[23] D. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International conference machine learning, pages 783-792. PMLR, 2019.
[24] L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self- supervised reward regression. arXiv preprint arXiv:2010.1172: 2020.
[25] D. S. Brown, W. Goo, and S. Niekum. Better-than-demoonstatoo imitation learning via automatically-ranked demonstrations. In Conference on robot learning, pages 330-359. PMLR, 2020.
[26] Z. Cao and D. Sadigh. Learning from imperfect demonstrations from agents with varying dynamics. IEEE Robotics and Automation Letters, 6(3):5231-522 5538,, 2021.
[27] Z. Cao, Y. Hao, M. Li, and D. Sadigh. Learning feasibility to imitate demonstrators with different dynamics. arXiv preprint arXiv:2110.151422 2021.
[28] I. Radosavovic, X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In 202/ JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7865-7871, IEEE, 2020.
[29] F. Liu, Z. Ling, T. Mu, and H. Su. State alignment- based imitation learning. arXiv preprint arXiv:1911. 1044, 2019.
[30] J. Oh, Y. Guo, S. Singh, and H. Lee. Self-imitation learning. In International Conference Machine Learning, pages 3878-3887. PMLR, 2018
[31] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning, pages 4344 4353. PMLR, 2018
[32] Haarnoja, Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[33] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[34] Schulman, F. Wolski, Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
10

---

[35] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin- Martín. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 2020.
[36] E. Todorov, T. Erez, and Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[37] B. Kang. Z. Jie, and Feng. Policy optimization with demonstrations. International con- ference machine learning, pages
PMLR, 2018.
[38] S. Reddy, A. D. Dragan, and S. Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv: 11108, 2019.
[39] T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Nowé. Reinforcement learning from demonstration through shaping. In Twenty-fourth international joint conference on artificial intelligence, 2015.
[40] A. Hussein, E. Elyan, M. M. Gaber, and C. Jayne. Deep reward shaping from demonstrations. In 2017 International Joint Conference on Neural Networks (IJCNN, pages 510-517. IEEE, 2017.


Reviews:
[{"review_id": "d7Ac3J9VxQS", "reviewer": "Reviewer_hEn1", "summary": "Learning from Demonstration (LfD) is a mechanism to help guide reinforcement learning (RL) algorithms when in the setting of learning with sparse rewards. This paper explores a problem with using LfD to guide RL when the demonstrators providing the LfD data are performing a different (\"mismatched\") task. The paper develops Conservative Reward Shaping from Demonstration (CRSfD), which estimates and provides a shaped reward to the RL agent to support intelligent exploration. Empirical results show that the approach outperforms baselines, and the paper provides a robot demonstration.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-The paper evaluates the proposed approach, CRSfD against numerous baselines, including both pure RL methods (SAC), pure LfD mechanisms (BC, GAIL, GAIfO), and hybrid appraoches (POfD, SQIL, and SACfD). \n-The evaluation shows consistently positive results for CRSfD (Figure 3) across a set of custom tasks. The evaluation shows CRSfD can outperform its most challenging competitor, SACfD, when multiple aspects of the task have changed.\n-An ablation study is provided to show the relative contributions of various aspects of the algorithmic framework for CRSfD.\n-A real robot demonstration is provided (Table 1).\n-Code is provided, along with a video.\n\nWeaknesses: \n-It would be helpful if the paper contained more details about the experimental domain (it wasn't until watching the video that it was clear what was going on) and why these tasks are the right tasks for benchmarking. \n-This paper evaluates two forms of task misspecification (i.e., the (1) target location and (2) geometry for an insertion task move). However, \n-The paper states that [22] \"enforce[s] the agents to imitate the demonstrations only initially and then gradually discard them as learning progresses.\" However, in reading [22], it appears the method is grounded in the Preference-based Reinforcement Learning (PbRL) literature and generally does not imitate the demonstrations and gradually discard them. Rather, it takes a fundamentally different approach in which a reward function is learned based upon ranking data from humans, and a policy is optimized against that reward function. Later work by Brown et al. removed the need for rankings (Brown et al., CoRL'20), and Chen et al. then improved upon this method (CoRL'21). I think it would be helpful to benchmark SACfD against these approaches for learning from suboptimal demonstration.\n\nBrown, D.S., Goo, W. and Niekum, S., 2020, May. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning (pp. 330-359). PMLR.\n\nChen, L., Paleja, R. and Gombolay, M., 2021, October. Learning from Suboptimal Demonstration via Self-Supervised Reward Regression. In Conference on Robot Learning (pp. 1262-1277). PMLR.\n\n-Further limitations could have been discussed other than multi-task learning. What are the limits of the approach? When would it fail? Would this work for pick-and-place or other robot tasks? How does the approach degrade with the quality of the demonstration?\n-The paper provides multiple figures (e.g., Figure 1 and Figure 2) that attempt to give intuition to the problem being studied. However, the paper spends very little time actually describing how the approach works (Lines 165-196). Pseudocode is relegated to the appendix. It would have been more helpful had the paper removed Figure 2 and spent more time describing the algorithm itself. The mismatched reward problem is relatively self-explanatory, and the \"reward signal backward propagation blockage\" doesn't seem to be a novel problem or even a problem that needs to be discussed at length. If a robot fails to accomplish a new task because it has learned a different one, it may not be able to receive any (sparse) rewards, so learning will be difficult. It isn't clear to this reviewer than anything beyond this point is being argued. Is the reviewer missing something? Also, is \"reward backward propagation\" the same issue than (though with different words -- perhaps a special case of) the canonical credit-assignment problem?\n-Prior work has also considered reward shaping that should be considered:\n\nGoyal, P., Niekum, S. and Mooney, R.J., 2020. Pixl2r: Guiding reinforcement learning using natural language by mapping pixels to rewards. arXiv preprint arXiv:2007.15543.\n\nQuestions:\n-What would happen if one collected demonstrations on '4' but trained on '0' (or any other digit)?\n-How could multi-task RL/IRL be helpful here? Consider:\n\nGleave, A. and Habryka, O., 2018. Multi-task maximum entropy inverse reinforcement learning. arXiv preprint arXiv:1805.08882.\n\nSingh, A., Jang, E., Irpan, A., Kappler, D., Dalal, M., Levinev, S., Khansari, M. and Finn, C., 2020, May. Scalable multi-task imitation learning with autonomous improvement. In 2020 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2167-2173). IEEE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper develops an interesting reward-shaping approach to enable an RL algorithm to benefit from demonstrations that were provided for a different task. The results are positive; code and a robot demo are provided. However, the paper leaves out some important baselines (e.g., Chen et al., CoRL'21), does not devote enough attention to describing the technical method, and the limits of the algorithm are not sufficiently discussed/explored.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please address the issues of related work, clarity in the presentation of the algorithm, and improved discussion/exploration of limitations.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "Lvy5bf7xGRb", "reviewer": "Reviewer_CiFM", "summary": "This paper proposes a learning from demonstration method to learn a certain task from demonstrations of a different task. Starting from a recent method that is able to learn from off-policy data using a replay buffer, they additionally include a shaped reward function to provide a learning signal even in failed trajectories. A key component of this shaped reward is explicitly setting actions that lead to out of distribution states to 0 reward. In their experiments, they show that their method achieves superior performance in two settings: (1) when environment dynamics differ (in their case changing the shape of a hole for an insertion task) and (2) when both environment dynamics and the reward differs (in their case randomly perturbing the location of the hole from the demonstrations). They also include a detailed ablation study.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. Rigorous experiments considering numerous baselines. The constructed baseline of extending SAC with the method of Vecerik et al. is a nice addition. Ablation studies illustrating the impact of model components is also good. Also show zero-shot sim-to-real transfer.\n2. Methods section is clear, and the proposed method is simple and principled. \n\nWeaknesses\n1. While the related work is thorough, I am missing the motivation here. Why not just use inverse RL? The authors state that inverse RL methods are not able to learn from environment rewards, but this is a bit of a misrepresentation, as most inverse RL methods are assuming that such a reward function is unavailable.\n2. Explanation of \"reward signal backpropagation blockage\" is confusing to me. Section could be reworked to better explain the phenomenon. The corresponding figure 2 is also confusing, not sure if it is adding to the understanding.\n3. No baselines considering inverse RL methods.\n4. Only one type of task (object insertion) is considered. What about multi-step tasks? Navigation? I feel that the method may be more limited when considering more significant shifts between demonstrated and target task as the reward shaping mechanism seems to rely on the original value function learning a complete enough representation to transfer to new tasks, which is a strong assumption. \n5. No mention of how many demonstrations are used. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The experiments are well constructed with nice results, and the method is well-principled and elegant. However, as mentioned before, I do not completely see the motivation behind this method, and one of the main issues addressed, the blocked reward signal backpropagation, is unclear. Also, evaluation on only a block-insertion task is limited. If the authors can provide a better motivation and clarify the blocked reward signal section, I would be willing to change my score.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Section 4.1 and Figure 2: Rewrite/update to better explain phenomenon of blocked reward signal backprop.\nIntro/Related Work: Discussion of inverse RL and it's limitations could be improved\nOccasional typos"}}, {"review_id": "JKjAIrDarv8q", "reviewer": "Reviewer_Mmxm", "summary": "The paper studies how demonstrations can be used to improve the performance of online reinforcement learning (RL). In particular, the authors study settings where the demonstrated task does not exactly match the tasks which robot will have to execute. This removes the requirement of obtaining expert demonstrations for every new task the robot encounters. The main contribution is a conservative reward shaping algorithm for reinforcement learning with demonstrations from mismatched tasks. The work is evaluated in simulation and real-world experiments, including comparisons against state-of-the-art Learning from Demonstration (LfD) methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is well written, clearly structured and addresses an important challenge in robot learning – using demonstrations from mismatched tasks in RL. The authors identify shortcomings of existing approaches, and propose a novel solution technique. The evaluation in simulation and real-world experiments deliver convincing results.\nOn the downside, the paper lacks detail in the theoretical part, and the evaluation is limited to only one experiment. Please see detailed comments below:\n\n•\tresolved -- The paper identifies a key issue for standard methods combining LfD with RL, called Sparse Reward Signal Backward Propagation Blockage. While the description and illustration are convincing on a high level, I would like to see more technical details on this.\n\n•\tresolved -- Related to above comment, the paper lacks a more rigorous description of how the use of reward shaping avoids the backward propagation blockage. Are there any theoretical guarantees on this?\n\n•\tresolved -- The limited theoretical discussion also raises questions about the overall contribution of the paper: is the use of reward shaping with demonstrations novel? I don’t think any related work for this was cited, yet it is not entirely clear to me. \n\n•\tUpdated -- The second main contribution is addressing the out-of-distribution problem. The approach for this is only described very briefly in lines 176-180, and leaves questions about the details: How does the regression work? How is the discount factor increased? \n\nUpdate: While the overall description has improved in the revised paper, the details about how the discount factor is updated are still unclear (including the extended algorithm in the appendix)\n\n•\tresolved -- What is the ‘Conservative’ part of the proposed technique? Is there a less or more conservative variant of the algorithm, and how does it compare? Since it appears in the name of the algorithm, it should be elaborated more.\n\n•\tUpdated -- The parameter \\lambda in equation 3 is not explained. \n\nUpdate: Similar to the discount factor, I am still missing details on how to choose this p.arameter\n\n•\tUpdated -- While the evaluation is overall providing detailed insights and delivers promising results, the major weakness is that it only considers slight variations of a peg-into-hole problem. It could be significantly strengthened by being extended to a completely different robot task.\n\nUpdate: the paper improved greatly by additional experiments with varying mismatch. Thus, the experiments are rich enough for a conference paper.\n\n•\tresolved -- An important question left unanswered is how mismatched the tasks can be. While it might be difficult to get specific theoretical results, empirical data would be interesting. I would expect that with increasing mismatch, the advantages of the proposed approach will start to disappear. For large mismatch, using information from the demonstrations might actually be detrimental to learning progress.\n\n\nMinor comments:\n•\tresolved -- Typo line 77 Standard should be lower cas: standard \n\n•\tresolved -- Line 156, shouldn’t the function map be  $\\Phi : S \\rightarrow \\mathbb{R}$ not … \\rightarrow R ?\n\nLimitation Section:\n•\tresolved -- The limitation section only discusses joint learning for multiple tasks. While this is an interesting future work direction, it is only partially a limitation of the presented work – multiple tasks is not part of the problem statement. Some discussion of how to find the discount factor, if there are problems where the proposed method is not beneficial at all, or the influence how mismatched demonstrations are would be very interesting\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper discusses an interesting and relevant problem and is well structured. However, the technical presentation lacks some more detailed description and discussion and thus is not entirely convincing. \n\nUpdate: The technical description has improved through the revisions, and the experiments were extended which helped to show the robustness of the proposed work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1) Resolved -- Add a theoretical discussion on how the proposed algorithm avoids Sparse Reward Signal Backward Propagation Blockage\n2) Resolved -- Elaborate the description of the regression and discount factor (lines 176-180), as well as the conservative character of the approach.\n3) Resolved -- Extend evaluation to at least one of the following: i) increasingly larger task mismatch, ii) entirely different problem (plots for ii could just be in the appendix)."}}, {"review_id": "4LncacMFsMG", "reviewer": "Reviewer_rKEH", "summary": "This paper proposes a method to leverage information from expert traces on an MDP M to help solve a sparse reward task on an MDP M', where M' differs from M only by its reward.  The method involves performing policy evaluation on expert traces on M to calculate a value function V_M, and then leveraging that value function as a potential function for reward shaping on M'.  Experiments are performed both in simulation and on a physical robot for variants of an insertion task, where the sparse-reward variants of the task are shown to be solved faster than with existing methods thanks to the use of information from an expert solving a different instance of a similar task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well written and easy to follow, and the ideas are easy to understand.  The experiments are convincing and clearly applicable to a robotics domain.  \n\nI think the paper could be improved with some more pertinent baselines, and a bit more detail on the actual algorithm.  I am also dissapointed that the robot experiments are just a sim2real transfer of the simulator-learnt policy, whereas the whole point of this type of algorithm is to be able to perform sufficiently efficient sparse reward learning such as to be able to train directly on hardware.  Another concern is that this approach requires a dense reward for the original expert demonstrations, which is something that is generally not available in a realistic scenario.\n\nThe paper is also missing some import related work:\nShaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models [https://arxiv.org/pdf/2011.01298.pdf]\nReinforcement Learning from Demonstration through Shaping [https://openreview.net/forum?id=S145CVGuWB]\nDeep Reward Shaping from Demonstrations [https://ieeexplore.ieee.org/abstract/document/7965896]\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the ideas in the paper, although well-executed, is not particularly novel.  Using a potential function from expert demonstrations to shape the reward for a given task is something that has been looked at on a couple of occasions.  The novelty of the paper is to look in particular at domain shift, and leverage the existence of a well-defined reward during the expert demonstration, ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": " I'm not convinced the 'sparse reward signal backward propagation blockage' is exactly what's going on, or if it is that would require further discussion.\nI'm not sure I understand the point of using pure imitation algorithms such as GAIL as baselines, as it seems pretty obvious they would not be able to solve a novel task.  Perhaps I missed something here in how the setup is done, but I'm not sure these provide much insight beyond BC, there is no expectation they would work in this setup anyways.\nIf the authors could convince me of the novelty of the contribution, and the types of scenarios where one would have access to a high-quality reward on the expert demonstrations, that would allow the calculation of V, as well as perhaps baselines that also use reward shaping coming from different potential functions (perhaps inspired from some of the existing literature) this could potentially make for a stronger paper."}}]

Author rebuttal:

Decision: Accept (poster)