Imitation Learning from Observation with Automatic Discount Scheduling
Authors: Yuyang Liu
Weijun Dong
Yingdong Hu
Chuan Wen
Zhao-Heng Yin
Chongjie Zhang
Yang Gao
Venue: ICLR 2024 poster (2024)
Keywords: imitation learning
inverse reinforcement learning
reinforcement learning
currilulum learning
optimal transport
Primary area: reinforcement learning

Abstract: Humans often acquire new skills through observation and imitation. For robotic agents, learning from the plethora of unlabeled video demonstration data available on the Internet necessitates imitating the expert without access to its action, presenting a challenge known as Imitation Learning from Observation (ILfO). A common approach to tackle ILfO problems is to convert them into inverse reinforcement learning problems, utilizing a proxy reward computed from the agent's and the expert's observations. Nonetheless, we identify that tasks characterized by a progress dependency property pose significant challenges for such approaches; in these tasks, the agent needs to initially learn the expert's preceding behaviors before mastering the subsequent ones. Our investigation reveals that the main cause is that the reward signals assigned to later steps hinder the learning of initial behaviors. To address this challenge, we present a novel ILfO framework that enables the agent to master earlier behaviors before advancing to later ones. We introduce an Automatic Discount Scheduling (ADS) mechanism that adaptively alters the discount factor in reinforcement learning during the training phase, prioritizing earlier rewards initially and gradually engaging later rewards only when the earlier behaviors have been mastered. Our experiments, conducted on nine Meta-World tasks, demonstrate that our method significantly outperforms state-of-the-art methods across all tasks, including those that are unsolvable by them. Our code is available at https://il-ads.github.io.

Published as a conference paper at ICLR 2024
IMITATION LEARNING FROM OBSERVATION WITH AUTOMATIC DISCOUNT SCHEDULING
Yuyang Liu 1,2,, Weijun Dong Yingdong Hu Hu?? Chuan Wen 12, Zhao-Heng Yin?, Chongjie Zhang', Yang Gao 1,26,f "Institute for Interdisciplinary Information Sciences, Tsinghua University Shhnggaii Qi Zhi Institute SUC Berkeley
*Washington University in St. Louis SShangaai Artificial Intelligence Laboratory [yyliu22, dwj huyd21, zhachengyinges berkeley. edu, [oongiie@wsst.... eduu gaoyangiiis www.iiiissmmmii....ii wiigguuu...iiniia.a.a. edu.
is ts tnighuu edu. cn
ABSTRACT
Humans often acquire new skills through observation and imitation. For robotic agents, learning from the plethora of unlabeled video demonstration data avail- able on Internet necessitates imitating the expert without access to its action, presenting challenge known as Imitation Learning from Observation (ILfO). A common approach to tackle ILfO problems is to convert them into inverse rein- forcement learning problems, utilizing a proxy reward computed from the agent' s and the expert's s observations. Nonetheless, we identify that tasks characterized by progress dependency property pose significant challenges for such approaches; in these tasks, the agent needs to initially learn the expert's preceding behaviors before mastering the subsequent ones. Our investigation reveals that the main cause is that the reward signals assigned to later steps hinder the learning of ini- tial behaviors. To address this challenge, we present novel ILfO framework that enables the agent to master earlier behaviors before advancing to later ones. We introduce an Automatic Discount Scheduling (ADS) mechanism that adaptively alters the discount factor in reinforcement learning during the training phase, pri- oritizing earlier rewards initially and gradually engaging later rewards only when the earlier behaviors have been mastered. Our experiments, conducted nine Meta- World tasks, demonstrate that our method significantly outperforms state- of-the-art art methods across all tasks, including those that unsolvable by them. Our code is available at wulhtps:/// :/il 11:1-aceh -ads. rads.giii. tttb......
INTRODUCTION
Observing and imitating others is an essential aspect of intelligence. As humans, we often learn by watching what other people do. Similarly, robotic agents can learn new skills by watching ex- perts and mimicking them through their observation- pairs, method often far more sample- efficient than relying solely on self-guided interactions with the environment. Beyond the conven- tional demonstrations, there is a vast repository of unlabeled video demonstration data available on the Internet, lacking explicit information on the actions associated with each state. To utilize these valuable resources, we direct our attention to specific problem known as Imitation Learning from Observation (ILfO; Torabi et al., 2019). this setting, agents solely have access to sequences of demonstration states without any knowledge of the actions executed by the demonstrator.
Canonical imitation algorithms, such as behavior cloning (Bain & Sammut 1995] Ross et al. 2011 Daftry et al. (017), can not be directly applied to ILfO, they rely access to the expert's 's actions for behavior recovery. To deal with I.fO problems, one prominent category of approaches involves getting proxy rewards based the distribution of the agent's 's and the expert's visited states Torabi et al.
2018b Yang et al. [019 Lee et al. 2021 Kidambi et al.) 2021 Jaegle et al., [021]] Liu et al.]
2022
These approaches first derive stepwise reward signals through techniques like occupancy
"Equal Contribution. Corresponding Author.

---

Published as a conference paper at ICLR 2024
Expert Demo
D bjecttve
Oijective
Learned Behavior
7
the ball successfully grasped
the ball is swept away
(a) Take the initial part the demo as objoctive
(b) Take the full demo as objective
ILfO methods on a task with progress dependency. For the task basketball, (a) taking only the initial part of the expert demonstration as the imitation objective, the agent efficiently acquires the grasping skill; (b) taking the entire expert demonstration as the imitation objective, the agent fails to grasp the ball and instead sweeps it away.
Figure An example of employing proxy-reward-baee
measure matching (Ho & Ermon, 2016) or trajectory matching (Haldar et al., 2023a and then employ reinforcement learning (RL) to optimize the expected cumulative reward. However, the performance of these methods remains unsatisfactory, particularly in challenging manipulation tasks with high-dimensional visual observations, where agents struggle to achieve task completion despite extensive interactions with the environment.
To understand why traditional wroxy-reward-basee methods fail, we conduct experiments on the task basketball from the Meta-World suite (Yu et al., 2020). As shown in Figure [ a robotic agent needs to grasp basketball and deposit it in the basket. We first experiment with a simplified setting, which only instructs the agent to learn the expert's ' early behaviors of reaching for and grasping the ball. The agent quickly acquires these skills (see Figure [ indicating that grasping the ball is not inherently difficult and can be learned efficiently. However, when tasked with learning the entire ex- pert demonstration, the same method fails acquire the initial grasping skill and instead moves the empty gripper directly to the basket (see Figure (b)). Comparing these two scenarios, we discover that rewarding later steps in trajectory negatively impacted the agent's ability learn the earlier behaviors, which resulted in difficulties in mastering subsequent actions and the overall task. This pattern not unique to the basketball task. We observe a similar phenomenon in many manipula- tion tasks. All these tasks share property: the agents must first acquire earlier behaviors before progressing to later ones. Our research shows that conventional ILfO approaches often struggle with tasks characterized by progress dependencies, primarily because agents fail to mimic the expert's early behaviors. Instead, agents resort to optimizing rewards in later stages by moving to states that appear similar demonstrated states. However, these states differ from the demonstrated ones because the agent has not yet completed the necessary preliminary steps. Therefore, these locally optimal but incorrect solutions can hinder the agent's exploration of earlier critical behaviors.
Based on our previous analysis, we introduce a novel ILfO framework to handle tasks with progress dependencies. We propose encouraging the agent master earlier parts of demonstrated behaviors before proceeding to subsequent ones. To achieve this, we restrict the impact of later rewards until the agent has mastered the previous behaviors. We implement this idea in simple yet effective way by incorporating dynamic scheduling mechanism for a fundamental term in RL- the discount factor y. During the initial training phase, we employ a relatively small 7, leading to value functions focusing on short-term future rewards. For the initial states, these short-sighted value functions will reduce the impact of misleading proxy rewards from the later episode stages, thus helping the imitation of early episode behaviors. As the agent advances in the task, the discount factor increases adaptively, allowing the agent to tackle later stages only after it has effectively learned the earlier behaviors. This mechanism, we call Automatic Discount Scheduling (ADS), is reminiscent of Curriculum Learning (CL) introduced by Bengio et al. (2009), which structures the learning process to increase the complexity of the training objective gradually. Experimental results demonstrate that ADS overcomes the challenges associated with traditional proxy-reward-baee based methods and surpasses the state-of-the-ae in complex Meta- World tasks.
2

---

Published as a conference paper at ICLR 2024
Our contributions are summarized follows:
We discover that conventional ILfO algorithms struggle on tasks with progress dependency.
We introduce novel II.fO framework featuring Automatic Discount Scheduling (ADS), enabling the agent to master earlier behaviors before advancing to later ones.
In all of the nine evaluated challenging Meta-Worke manipulation tasks, our innovative approach significantly outperforms prior state-of-the-ae ILfO methods.
BACKGROUND
In this section, we delve into the idea of imitation learning through proxy rewards, which widely used framework to tackle the ILfO problem. Furthermore, we introduce Optimal Transport (OT), which is a reward labeling technique employed by our method to compute the proxy rewards.
2.1 IMITATION THROUGH PROXY REWARDS
We consider agents acting within a finite-horizon Markov Decision Process (8, A, P, R, 7, Pinit T), where S is the state space, A is the action space, P is the transition function, is the reward function, 7 is the discount factor, Pinit is the initial state distribution, and 7' is the time horizon. In image-based tasks, a single frame may not fully describe the environment's underlying state. Following common practice (Mnih et al. 2013) Yarats et al. 2021). we use the stack of 3 consecutive RGB images (denoted by observation o₂) as the approximation of the current underlying state 8t-
We assume that a cost function over the observation space O x O + T is given. This cost function will be used reward inferring (Section 2.2 and progress recognizing (Section 4.2
In the context of ILfO, the environment does not provide reward function. Instead, our goal is to train an agent using set of N observation-only trajectories denoted as D = {-")== which demonstrated by an expert. Each trajectory 7 is composed of a sequence of observations 7" {of)
A prevalent approach to address the ILfO problem involves transforming it into Reinforcement Learning (RL) problem by defining proxy rewards based on the agent's trajectory 7 and the expert
demonstrations: (r. LT == fr (T, D=), where f. represents a criterion for reward assignment (Toabi et al. 2018b Yang et al., 2019; Lee et 2021 Jaegle al. [021]] Liu et al., [2022 Huang
ct al.
2023). Subsequently. RL is employed to maximize the expected discounted sum of rewards:
E
Σ
Tt
2.2 REWARD LABELING VIA OPTIMAL TRANSPORT
Optimal Transport (OT; Villani et al. 2009) is an approach for measuring the distance between prob- ability distributions. For simplicity, clarify its definition in the scope of ILfO. Given predefined cost function c(,,)) over the observation space, we define the Wasserstein distance between an agent trajectory (0r .0r) and an expert trajectory T {of,- .04 as:
W(T,T")
min
ΣΣ =1
clos, mf)
(2)
PE
subjected to
Σ
((,))
1 T
[m(i,)) 1 T
(3)
Each E T satisfying Equation 3 called a transport plan. When a transport plan achieves the minimization specified in Equation 2 is designated as the optimal transport plan pr,

---

Published as a conference paper at ICLR 2024
We can use OT to derive proxy reward function for the agent's trajectory т. Let 7" D be the expert trajectory with minimal Wasserstein distance to The rewards (re) are assigned by:
c(o. 0f)
Due to its practicality and efficacy, OT has become widely used approach for calculating rewards
Arrovssy et al. 2017: Papagiannis & Li 2022 Luo et al. 2023 Haldar et al.) 2023a
3 CHALLENGES IN ILFO ON TASKS WITH PROGRESS DEPENDENCY
In this section, we provide more discussion on the basketball task illustrated in Section I We elab- orate on why a proxy-rewarr meexy-reward-baee method (see Section 2.1 fails to solve this task, and conclude that this phenomenon reveals unique challenge in ILfO on tasks with progress dependency.
As shown in Figure 2 a). when learning the basketball task with proxy-reward ally learns a plausible policy that sweeps the ball out of the camera's s view and then moves the empty gripper to the basket. Though the agent has not successfully grasped the ball, this policy still maximizes the sum of proxy re- wards since it advance to states that resemble the ex- pert's demonstrations the gripper's moving path without the ball. While ob- taining this sweeping policy, the agent can also explore behaviors that successfully lift the ball, as shown in Fig- ure (b). However, despite picking the ball up, the agent usually fails to move the ball to the basket or quickly drops the ball in these trajectories. Compared to the sweeping policy, the trajectory in Figure (bb) receives a higher proxy reward in the initial steps, but much lower proxy reward in the later steps. These rewards cause an RL agent to estimate much lower value for lifting the ball the initial stage than for pushing away. Thus, when using usual RL algorithm, the agent will rarely explore picking the ball and get stuck in the suboptimal sweeping policy. In summary, the proxy rewards assigned to the later steps in a tra- jectory negatively impacted the agent's ability to learn the earlier behaviors in the basketball task, which is in line with the observation in Figure [b). Similar patterns can be observed in many tasks with progress dependency, which challenges the conventional ILfO approaches.
method, the agent usu-
(a)
subsequent actions by imitating
(b)
Figure 2: (a) The agent learns a subop- timal policy that sweeps the ball away. (b) The agent can also collect explo- rative trajectories that successfully pick the ball up for a certain height, but it still fails to acquire this skill.
Remark. This challenge is highly related to the nature of imitation through proxy rewards. In usual RL tasks with manually designed rewards, the progress dependency property will challenge the RL algorithm, since the manually designed rewards usually incorporate the characterization of this property. For example, in the basketball task, handcraft reward function only assigns positive rewards to the states where the ball is grasped. This assignment naturally eliminates the previously mentioned suboptimal solutions.
METHOD
In this paper, we aim to overcome the challenges confronted by traditional proxy-reward-basee ILfO
algorithms (see Section 2.1 when tackling tasks characterized by the progress dependency property,
as discussed in Section 3 In Section 4.1, we illustrate solution for this challenge and propose
a novel framework called ILFO with Automatic Discount Scheduling (ADS). Section 4.2 further elaborates on the design of several challenging components in this framework.
4.1 FRAMEWORK
Recall that in a task with progress dependency property, rewarding later steps in a trajectory can negatively impact the agent's ability to learn the earlier behaviors. We posit principle to avoid this
4

---

Published as a conference paper at ICLR 2024
Algorithm Imitation Learning from Observation with Automatic Discount Scheduling
Require: Expert Demonstrations D
1: Initialize RL agent π
2: Initialize progress recognizer with D
3: Initialize discount factor y t-
4: for episode = 1, 1, do
OT) 6: Compute proxy rewards {ri.... , + fo Dr)
5: Sample a trajectory
7:
Update RL agent with the rewarded trajectory fol (1,,01, ,o) Update progress recognizer & with
8:
9: Query & about the current progress & o (.Curreentrogreess)
10: Update discount factor + fn (k)
end for
problem: if the agent has not mastered the early part of the demonstrated sequences, should not incorporate imitating the later parts in its current learning objective. We highlight that setting a lower value for fundamental term in RL the discount factor ? - naturally serves as a soft instantiation of this principle. From an RL perspective, low discount factor prioritizes the rewards obtained in the initial stages of an episode. Specifically, while optimizing the cumulative discounted reward (Equation] 1. the reward received at step is weighted by - Therefore, utilizing low discount factor can encourage the agent to focus on optimizing the rewards obtained in early episode steps, which corresponds to imitating the early part of the demonstrations in the context of ILfO.
However, an inappropriately low discount factor can make the agent too shortsighted and perform unsatisfactory behaviors in the late episode steps. It is critical to increase the discount factor once the agent masters the early part of the demonstration, ensuring that the later segments of the demon- strations are also learned sufficiently. To achieve discount scheduling mechanism that is adaptive to distinct properties of various tasks, we propose a novel framework called ILfO with Automatic Discount Scheduling (ADS).
Training pipeline. In ADS, we deploy progress recognizer + to continuously monitor the agent's learning progress, and dynamically assign discount factor that positively correlates with the progress. The overall training pipeline is outlined in Algorithm [ At the start of the training process, the agent assigned a low discount factor of 7 = Yo. which facilitates the agent to mimic the expert's 's myopic behaviors. As the training advances, we periodically consult the progress rec- ognizer & to track the extent to which the agent has assimilated the expert's behaviors. The function Q CurrentProgy (eggessss))))) returns an integer & between 0 and T. indicating that the agent's current policy can follow the expert's behavior the first k steps. Once & is updated, the discount factor is updated according to Jr (k), where Jy is monotonically increasing function. Then, the agent will continue its trial-and-eme loop with regard to the new 7.
Designing suitable discount scheduling method, including progress recognizer & and mapping function fy, is the major challenge of instantiating our framework. We will further explore these components in Section
4.2 DISCOUNT SCHEDULING
Progress recognizer 4. The progress recognizer & receives the agent's collected trajectories (line 8 in Algorithm and need output the agent's learning progress (line in Algorithm []. To de- velop this progress recognizer, we initially introduce a measurement to evaluate the progress align- ment of one trajectory evaluate how close this pair of trajectories is to forming monotonic frame-by-frame alignment. To be specific, we consider the sequence p = {pI, ,Pn),, Pn), where p; = arg min, c(o,, ) is the index of the nearest neighbor of o, in r'. If T and r' are exactly the same, then p becomes a strictly increas- ing sequence. On the contrary, if 7 and 7' characterize totally different behaviors, becomes a dis- ordered sequence. Following this intuition, we propose to measure the progress alignment between and r' by the length of the longest increasing subsequence (LIS) in p. denoted by LIS(T, The longest increasing subsequence problem chooses a (not necessarily contiguous) subsequence of p, such that it is strictly increasing (w.rt the order in p) and achieves the longest length. For
{o1.... on) ) to another trajectory 7' {o), ... .... We intend to
5

---

Published as a conference paper at ICLR 2024
instance, p (1, 2,, 2,6,,,,,,,,7 then its longest increasing subsequences can be 21,2,4,5, or {1, 4, 73. The LIS measurement focuses on the consistency of these trajectories' macroscopic trends, which avoids overfitting the microscopic features in the observed frames.
Now, we utilize this measurement to design the progress recognizer 4. & keeps tracking the agent's s learning progress k. Each time & receives the agent's s recently collected trajectory т, it considers the first k + steps of the agent's 's and the demonstrated trajectories. the progress alignment between the agent's and some demonstrated trajectory is comparable the progress alignment between two demonstrated expert trajectories, then we posit that the agent's current policy can follow demonstra- tions in the first k steps. Specifically, we increase by one if the following inequality holds:
max LIS (TS r'ED"
min X T' and r' LIS(T T(R+1))
(5)
where the subscript 1: a hyperparameter that controls the strictness with which we monitor the agent's 's progress.
k + means extracting the first k 1 steps of the trajectory, and A [0,
Mapping function f.. Taking the progress indicator k as input, f- outputs for the agent. One straightforward idea of setting f, is to make the discount weight of every reward received after step & not larger than hyperparameter a (0, Recall that the RL objective (Eq. lb. the reward received at step i is weighted by g-11. Therefore, we propose (k) 1/2 In Section [5.]] we will show that simply setting a 0.2 makes our algorithm work well across variety tasks.
discount factor
EXPERIMENTS
We conduct a series of experiments to evaluate the performance of our approach. We show the performance against baseline methods in Section 5.2 validate the effectiveness of ADS in Section
5.3
and do meticulous ablation studies Section 5.4
5.1
EXPERIMENTAL SETUP
Tasks. We experiment with challenging tasks from the Meta-Word YYu al. 2020) suite. Ap- pendix B provides brief introduction to these tasks.
OT+ADS (Ours)
OT
GAITO
BCO
assembly
hammer
basketb all
100
100
100
75
75-
Success
50
50-
0.0
1.5
2.0
0.0
1.0
1.5
Env Frames
Frames (x10F)
(x105)
door-unlock
lever-pull
stick-push
100
100
100
75
75
75
Succers
Success
50
50
50
25
25
Eni Frames (x10")
Frames (x11f)
(x10*)
door-lock
pick-place-wail
door-open
100
100
75
75-
Success
Staccesss
50
50
50
25
25
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
Env Fromes bx109)
Enw Frames (x10f)
Env Fromes (x105)
Figure 3: Evaluation ILfO methods on 9 Meta-world tasks (2 million environment frames). Each curve reports the mean and standard deviation over random seeds.

---

Published as a conference paper at ICLR 2024
ILfO settings. All the agents not given any access to the environment's rewards cess/failure signals during training. Instead, the agent is equipped with 10 expert demonstration sequences, which solely comprise observational data. These demonstrations ploying hard-coded policies from Meta- Seta-s open-source codebase and consist of a series of RGB frames. To construct the cost function over this observation space (see Section 2.T). utilize cosine distance over the features extracted by a frozen ResNet-50 network (He et al. 2016) which pre-trained on the ImageNet (Deng al. 2009) dataset.
generated by em-
Baselines. We compare our approach against three representative ILfO methods, including two proxy-reward-biee methods OT and GAIFO, and one wwwerre-moeel-baeeeee method BCO. The de- tailed descriptions of these methods are deferred to Appendix [A22 To ensure a fair comparison, we equip all the moxy-reward-basee methods (OT, GAIfO and our approach) with the same underlying RL algorithm, DrQ-v2 Yarats et al. 2021 By default, we equip the baselines with y = 0.99.
5.2
MAIN RESULTS
Figure provides comprehensive comparison between approach and the baseline methods. To minimize uncertainty and obtain reliable findings, we report the mean and standard deviation over random seeds. In terms of final performance, superior performance in out of 9 tasks. Additionally, concerning sample efficiency, which refers the number of online interactions with the environment, our approach shows significant improve- ments across of the 9 tasks, with the exception being the door-lock task, where all methods exhibit low final success rates. Notably, our approach achieves more substantial performance gains on more challenging tasks, such as basketball and lever-pull, which exhibit pronounced progress dependency properties. Finally, is worth emphasizing that ADS serves as general solution for overcoming the challenges in tasks with progress dependency. Therefore, in addition to its integration with the OT method, ADS can readily adapted to improve the performance other ILfO methods. Additional results of applying ADS to the GAIfO method are available in Appendix C.I
measured by task success rate, our method yields
5.3
ADAPTIVE SCHEDULING
OTS (Ours)
OT (y= 0.99)
OT (y=0.
OT (y=0.93)
(y
assembly
hammer
basketball
100
00
80
Success 9
60
Success
60
60
20
20
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Enw Frames (x105)
Enw Frames (K11F)
Enw Frames (X11P)
Figure 4: Comparing OT+ADS against OT equipped with fixed discount factor (1 million environ- ment frames).
ADS (Ours)
OT (y 0.9 0.99 in 0.5M frames)
OT (y 0.9 0.99 in 1M frames)
assembly
hammer
basketball
100
100
80
80
80
Succoess %
60
Success
60
20
20
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Emw Frames (K109)
Enw Frames (x10F)
Env Frames (x10F))
Figure 5: Comparing OT+ADS against OT equipped with an exponential discount scheduling (1 million environment frames). The discount factor for the baselines exponentially increases from 0.9 to 0.99 within 0.5 or million environment frames.
This section delves into the design of discount scheduling. In Figure we compare the perfor- mance of ADS with constant discount factors (7 0.9, 0.93, 0.96, 0.99) and observe that ADS

---

Published as a conference paper at ICLR 2024
consistently outperforms all constant discount factors, underscoring the advantages of dynamic discount schedule. Moreover, Figure contrasts ADS with two manually crafted dynamic discount schedules. These schedules exponentally increase y from 0.9 to 0.99 within 0.5 or 1 million en- vironment frames. We observe that these schedules also generally fall short of ADS's exceptional performance due their inherent lack of adaptability.
assembly
hammer
basketball
0.975
100
0.375
100
0.975-
100
Discount
0.950
0.350
Success
Discount
0.950
Success
0.925
2.0
0.5
2.0
Frames (x10F)
Frames (x10F)
Frames (x100)
Figure 6: Visualization of the discount factor scheduled by our method during the training process.
In Figure 6 we demonstrate how ADS showcases adaptability tailored to the unique characteristics of different tasks. In the assembly and basketball tasks, observe that as the discount factor gradually increases, it reaches a plateau phase where it stabilizes at a constant value. This plateau signifies the task entering challenging phase demanding extensive exploration. Specifically, in the assembly task, the accurate attachment of the ring to the pillar poses a challenge, while in the basketball task, grasping and lifting the ball is the challenging step. Utilizing ADS ensures that the discount factor remains at an uppropriate level, facilitating sufficient exploration and optimization of these challenging stages. Consequently, the agent can effectively acquire the necessary skills during these phases and subsequently advance to learning later expert behaviors.
5.4
ABLATION STUDIES
1.0
- 0.9
0-88
À
assembly
hammer
basketball
1.00
100
100
Succoss 9%
60
Success
60
40
40
0
0.0
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
0.2
0.6
1.0
Emn Fiaaes (x109)
Enw Trames (x10F)
Env Frames (x(10))
Figure 7: Ablation study on hyperparameter in the progress recognizer 4 million environment frames). À is set to 0.9 defaultly in our method.
0.1
0.2
a- 0.3
a- 0.4
assembly
hammer
basketball
1.00
100
100
60
Success
Success
60
40
40
0.0
0.4
0.6
0.8
1.0
0.2
0.4
0.8
0.2
0.6
1.0
Env Frames (x10f)
Enw Frames (x10%)
Env Frames (X110%)
Figure 8: Ablation study on hyperparameter in the mapping function f- (1 million environment frames). a is set to 0.2 defaultly in our method.
In our ADS method introduced in Section 4.2 we involve two hyperparameters: for the progress recognizer t and a for the mapping function f-.. We perform ablations on À and a in Figure 7 and Figure { respectively, with results averaged over random seeds. Figure 7 demonstrates that ADS exhibits robustness regarding the value of X; it achieves satisfactory performance with values of 0.8. 0.9, and 1.0. Figure [ iillsstrttes the impact of alpha: smaller alpha values are more beneficial for early critical tasks, e.g., basketball, while larger alpha values significantly enhance success rates for later challenging tasks, such as assembly. In our main results presented Section 5.2,, we employ 0.9 and 0.2.
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORK
Imitation learning from observation. ILfO Torabi et al. 2019) asks an agent to learn from observation-only demonstrations. Without expert actions, ILfO presents more challenges compared to standard imitation learning (Kidambi al., 202T). One prevalent line of ILfO algorithms infer proxy rewards from the agent experiences and the expert demonstrations, and deploy reinforcement
learning to optimize the cumulative rewards. The proxy rewards can be derived by matching agent's
and expert's 's state/trajectory distributions (Torabi et al. [0018b Yang et al., 2019 Jaegle al. [021
Huang et al., 2023 Al-Hafez et al. 2023) or estimating goal proximity (Lee et al. 2021: Bruce
et al.] 2022). Among these series of literature,
based algorithms (Dadashi et al. [2020] Haldar et al.) 2023a). They derive proxy rewards by calculating the
work is most related
Wasserstein distance between the agent's 's and expert's 's trajectories. Our method is built upon ILfO
through proxy rewards, and we choose OT as our basic block due to its promising performance in complex domains.
An alternate strand of IIfO literature leverages model-based methods. Some approaches train an inverse dynamics model by the agent's collected data, and use this model infer the expert's missing action information (Nair et al., 2017, Torabi et al.) 2018a Sa Pathak et al.] 2018 Radosavovic et al,)
2021]. Recent work also integrates the inverse dynamics model with
based algorithms (Liu et al., [022) Ramos et al. (023). Taking a different approach, [Edwards et al. (2019) learns a
forward dynamics model on a latent action space. Our automatic discount scheduling framework orthogonal to these model-based algorithms. It is also possible to leverage model-based components
in our framework to further enhance the performance. We leave this study as future work.
Curriculum learning in RL. Curriculum Learning (CL) (Bengio et al. 2009) is a training strategy where the learning process is structured to gradually increase the complexity of the training data or tasks, demonstrating its efficacy across wide range of deep learning applications (Wang et al. 2018
Soviany et al. 2020 Wang et al. 2019 Gu et al. 2022). In RL, existing literature deploys curricul lum learning by sorting the collected experiences in replay buffer (Schaul et al.. [015] Ren al. 2018
or training the agent in easier tasks and transferring it to more complex scenarios (Florensa et al,
2017 Silva & Costa 2018 Dennis al. 2020 Zhang al. 20200] Dai et al., 2021 Forestier al.,
2022
Our method requires the agent to first focus on imitating the expert's early behavior, and progress later segments after mastering those behaviors. This idea can be treated as an implicit organization of curriculum learning, which is different from formations in previous work.
Discount factor in RL. Existing literature extensively studies the role of the discount factor in RL. It is justified that a lower discount factor can: (1) tighten the approximation error bounds when re- wards are sparse (Petrik & Scherrer 2008, Chen al.) 2018); (2) reduce overfitting (Jiang et al.] 2015;; (3) serve as regularizer and improve performance when data is limited or data distribution is highly uniform (Amit et al. 2020); (4) be used to learn a series of value functions (Xu et al. 2018 Romoff et al., 2019); (5) achieve pessimism in offline RL (Hu al., 2022). We propose to use a lower discount in a setting different from these works. This idea is motivated by a unique property of imitation through proxy rewards (see Section B). which does not exist in common RL tasks with manually designed rewards. For discount scheduling, |François-Lavet et al. (2013) sug- gests progressively increasing the discount factor with a handcrafted scheduling during training. In contrast, we propose an automatic discount scheduling mechanism through monitoring the agent's learning progress, facilitating adaptability distinct properties of various tasks.
CONCLUSION
In this paper. we introduce a conceptually simple ILfO framework that is especially effective for tasks characterized by progress dependencies. Our approach necessitates the agent to initially learn the expert's 's preceding behaviors before advancing to master subsequent ones. We operationalize this principle by integrating a novel Automatic Discount Scheduling (ADS) mechanism. Through ex- tensive evaluations across 9 Meta- tasks, we observe remarkable performance improvements when employing our framework. We hope the promising results presented in this paper will inspire research community to focus more developing more general ILfO algorithm. Such algo- rithm could leverage a wealth of valuable learning resources on the web, including videos of humans performing various tasks.

---

Published as a conference paper at ICLR 2024
REPRODUCIBLE STATEMENT
With the code released online and the hyperparameter settings in Appendix A. the experiment results are highly reproducible. We also utilize sufficient random seeds in Section ensure repro- ducibility.
ACKNOWLEDGEMENT
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA01500000, This work is also supported by the National Key R&D Program of China (2022ZD0161700).
REFERENCES
Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. arXiv preprint arXiv:2303.000 00039, 2023.
Ron Amit, Ron Meir, and Kamil Ciosek. Discount factor as regularizer in reinforcement learning. In International conference on machine learning, pp. 269-278. PMLR, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. pp- 41 41-48, 2009,
Jake Bruce, Ankit Anand, Bogdan Mazoure, and Rob Fergus. Learning about progress from experts. In The Eleventh International Conference on Learning Representations, 2022.
Yi-Chun Chen, Mykel Kochenderfer, and Matthijs TJ Spaan. Improving offline value-function approximations for pomdps by reducing discount factors. In 2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pp. 3531-35366 IEEE, 2018.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imita- tion learning. arXiv preprint arXiv:2006.0460 2020.
Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monoc- ular reactive mav control. In 2016 International Symposium on Experimental Robotics, pp. 3-11. Springer, 2017.
Siyu Dai, Andreas Hofmann, and Brian Williams. Automatic curricula via expert demonstrations. arXiv preprint arXiv:2106.091::: 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255 leve, 2009.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049 10499-1333,, 2020.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International conference machine learning, pp. 1755-1763. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2024
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495. PMLR, 2017.
Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre- Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. The Journal of Machine Learning Research, 23(1):6818 6858, 2022.
Vincent François-Lavet. Raphael Fonteneau, and Damien Ernst. How to discount deep reinforce- ment learning: Towards new dynamic strategies. arXiv preprint arXiv:1512.0200 220.0201122200 2015.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural 2020.
information processing systems, 33:21271-2111
Yaowen Gu, Si Zheng, Zidu Xu, Qijin Yin, Liang Li, and Jiao Li. An efficient curriculum learning- based strategy for molecular graph learning. Briefings in bioinformutics, 23()):bbac099, 2022.
Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and match: Supercharg- ing imitation with regularized optimal transport. In Conference on Robot Learning, pp. 32 43.43. PMLR, 2023a.
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach robot fish: Versatile imitation from one minute of demonstrations. arXiv preprint vrXiv:2303.01499 2023b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjic Zhang. On the role of discount factor in offline reinforcement learning. In International Conference on Machine Learning, pp. 9072- 9098. PMLR, 2022.
Jialei Huang, Zhao-Heng Yin, Yingdong Hu, and Yang Gao. Policy contrastive imitation learning. In International Conference on Machine Learning, pp. 14007-14022. PMLR, 2023.
Andrew Jaegle, Yury Sulsky, Arun Ahuja, Jake Bruce, Rob Fergus, and Greg Wayne. Imitation by predicting observations. In International Conference Machine Learning, pp. 4665-4676. PMLR, 2021.
Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 1181-1189, 2015.
Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation learning from observation alone. Advances in Neural Information Processing Systems, 34:28598-288 28611, 2021.
Youngwoon Lee, Andrew Szot, Shao-Hua Sun, and Joseph Lim. Generalizable imitation learn- ing from observation via inferring goal proximity. Advances neural information processing systems, 14:16118-16138 2021.
Minghuan Liu, Zhengbang Zhu, Yuzheng Zhuang, Weinan Zhang, Jianye Hao, Yong Yu, and Jun Wang. Plan your target and learn your skills: Transferable state-only imitation learning via de- coupled policy optimization. arXiv preprint arXiv:2203.02211 2022.
Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, and Mare Peter Deisenroth. Optimal transport for offline imitation learning. arXiv preprint ar3.v:2303. 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 132.5660, 2013.
11

---

Published as a conference paper at ICLR 2024
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE international conference robotics and automation (ICRA), pp. 2153-2153. IEEE, 2017.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A univer- sal visual representation for robot manipulation. arXiv preprint 1rXXi:2203./2600 2022.
Georgios Papagiannis and Yunpeng Li. Imitation learning with sinkhorn distances. In Joint Eu- ropean Conference on Machine Learning and Knowledge Discovery in Databases, pp. 116-131. Springer, 2022.
Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp- 2050-2053, 2018
Marek Petrik and Bruno Scherrer. Biasing approximate dynamic programming with a lower discount factor. Advances in neural information processing systems, 21, 2008
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. In IROS, 2021.
João A Cândido Ramos, Lionel Blondé, Naoya Takeishi, and Alexandros Kalousis. Sample-efficient on-policy imitation learning from observations. arXiv preprint arXiv:2306.099% 2023.
Zhipeng Ren, Daoyi Dong, Huaxiong Li, and Chunlin Chen. Self-paced prioritized curriculum learning with coverage penalty in deep reinforcement learning. IEEE transactions on neural networks and learning systems, (9()) 2018.
Joshua Romoff, Peter Henderson, Ahmed Touati, Emma Brunskill, Joelle Pineau, and Yann Ollivier. Separating value functions across time-scales. In International Conference on Machine Learning, pp. 5468-5477. PMLR, 2019.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arXiv:1511.05955 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, PMLR, 2015.
Felipe Leno Da Silva and Anna Helena Reali Costa. Object-oriented curriculum generation for re- inforcement learning. In Proceedings of the 17th international conference autonomous agents and multiagent systems, pp. 1026-1034, 2018
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matri- ces. Pacific Journal of Mathematics, 21(2):343 1967.
Petru Soviany, Claudiu Ardei, Radu Tudor Ionescu, and Marius Leordeanu. Image difficulty riculum for generative adversarial networks (cugan). In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3472-3472, 2020. 2020.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. arXiv preprint arXiv: arXiv:1805.011 1805.01954 2018a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: a 0007006588, 2018h.
12

---

Published as a conference paper at ICLR 2024
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. arXiv preprint arXiv:1905.13500 2019.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Jiasi Wang, Xinggang Wang, and Wenyu Liu. Wed semi-supervised faster r-cnn with cur- riculum learning. In 2018 24th international conference on pattern recognition (ICPR), pp. 2416- 2421. IEEE, 2018.
Wei Wang, Isaac Caswell, and Ciprian Chelba. Dynamically composing domain-data selection with clean-data selection by" co-curricular learning" for neural machine translation. arXiv preprint arXiv:/906.01 /30, 2019.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Ad- vances in neural information processing systems, 31, 2018.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamies disagreement. Ad- vances in neural information processing systems, 32, 2019.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous trol: Improved duga-nee reinforcement learning. arXiv preprins arXiv:2107.09645, 2021.
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. Advances Neural Information Processing Systems, 33:7648-7659, 2020.
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
A.1 HYPERPARAMETERS
We equip all the proxy-reward-basee methods (OT, GAIfO and approach) with the same derlying RL algorithm, DrQ-v2 (arrats et al. 2021). The hyperparameters are listed in Table [
Table Hyperparameters.
Config
Value
Replay buffer capacity
150000
n-step returns
3
Mini-batch size
512
Discount y (for baselines)
0.99
Optimizer
Adam
Learning rate
10-4
Critic Q-function soft-update rate т
0.005
Hidden dimension
1024
Exploration noise
(0, 0.4)
Policy noise
clip(N 0.1), 003,
Delayed policy update
X (for progress recognizer ))
0.9
a (for mapping function f-)
0.2
A.2 BASELINES
In this section, we briefly introduce the baselines used in our experiments:
) OT is a imitation learning method that computes proxy rewards by calculating the optimal trans-
Optimal Transport (OT, Papagiannis & Li (2022)):
port (Sinkhorn & Knopp 1967 Cuturi 2013) between the agent's and expert's is visited
trajectories.
General Adversarial Imitation Learning from Observation (GAIfO, Torabi al. (2018b)): GAIfO is an ILfO algorithm based on generative adversarial imitation learn- ing (GAIL; Ho & Ermon 20166). It trains a discriminator (ss, s') to distinguish the agent's and the expert's transition pairs. In our implementation, we replace the input of the dis- criminator with the observation o,, as it has already stacked consecutive RGB images. is also worth noting that vanilla GAIfO is built upon TRPO (Schulman et al., 2015). and we replace the underlying RL algorithm with DrQ-v2 comparison.
Yarats et al. 2021) to ensure fair
Behavior Cloning from Observation (BCO, Torabi et al. 2018a): In BCO, the agent infers the expert's missing actions with an inverse model, which trained on the state- action pairs that the agent has gathered. This approach turns the ILfO problem into traditional problem, allowing for Behavioral Cloning (BC). We use BCO(a) introduced in the paper our baseline, where both the imitation policy and the inverse model are periodically updated.
B EXPERIEMENT TASKS
In this paper, we experiment with 10 tasks from the Meta-World suite (Yu al. 2020
1. Assembly: to pick nut and place it onto a peg.
2. Hammer: pick up a hammer and use it to hammer a screw on the wall.
3. Baskethall: to pick up baskethall and dump it into a basket.
14

---

Published as a conference paper at ICLR 2024
4. Door unlock: unlock the door by rotating the lock counter-clockwis.
5. Lever pull: pull lever up 90 degrees.
6. Stick push: pick up a stick and push kettle with the stick.
Door lock: to lock the door by rotating the lock clockwise.
8. Pick place wall: to pick up a puck, bypass a wall, and place the puck.
9. Door open: to open a door with a handle.
10. Button press: to press a button.
Assemblly
Hammer
Basketball
unlock
Stick push
Door lock
Pick place wall
Door open
Button press
Figure 9: Meta- Woold tasks used in our paper.
C ADDITIONAL EXPERIMENT RESULTS
C.I APPLY ADS TO ANOTHER PROXY-REWARD REWRRR-BERDDDD METHOD
OT+ADS (Ours)
GAITO + ADS (Ours)
OT
GAIIO
BCO
assembly
hammer
basketbal
100
100
100
75
75
Success
Successs
50
50
0.0
2.0
0.0
Env Frames
Frames (x10F)
(x105)
door-unlock
lever-pull
stick-push
100
100
100
75
75
75
Succers
Success
50
50
25
25
(x10")
(x11f)
(x10")
door-lock
pick-place-Wail
door-open
100
100
75-
Success
Staccesss
50
50
25
25
25
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
Env Frames Ix109)
Enw Frames (x10f)
Env Frames (x105)
Figure 10: Additional results on Meta-World tasks (2 million environment frames). Each curve reports the mean and standard deviation over random seeds.
15

---

Published as a conference paper at ICLR 2024
Additional results of GAIIO+ADS is shown in Figure 10 We observe that ADS significantly hances the performance of GAIfO over five of the nine tasks.
C.2 COMPARISON AGAINST ALGORITHM WITH GOAL-BASED BASED REWARDS
In this section, compare algorithm against RL with goal- basl- rewards. In this algorithm, the stepwise reward is defined as r = -c -o ). As shown in Figure II this algorithm performs poorly in 6 Meta- World tasks. Compared other proxy-reward heay--rwwrdddassse methods, it does not utilize the information given by the first frames of the demonstrations.
(Ours)
goal-based
assembly
hammer
basketball
100
200
100
75
75
75
Success
50
0.8
Eni Frames (x.109)
Frames (x11F)
Frames (x10F))
door-unlock
door-open
button-press
75
Success
Success
50
50
-0
25
25
0.0
1.0
0.8
Enw
Frames (x10f)
(x110)
Figure 11: Comparison Against RL with Goal-Based Rewards.
C.3 COMPARISON AGAINST ALGORITHM WITH LIS REWARDS
In this section, we experiment with an algorithm that uses the LIS measurement (see Section 4.2 to derive the proxy rewards. Recall that given the agent's trajectory T = {o,.... On and the expert's 's trajectory 7' {o',.... ), this measurement compute the longest increasing subsequence of P = (PI.... .p..ppppp J. where p; arg min, c(o;,0'; of)). We can assign rewards accordingly: we set r; to 1 if occurs in the longest increasing subsequence and 0 otherwise. Therefore, optimizing the cumulative rewards is equivalent to maximizing LIS(T,T').
OT+ADS (Ours)
assembly
hammer
basketball
100
200
100
75
75
75
Success
Success
50
0
Eni Framas (x1.09)
Frames (x11F)
(x10F)
door-unlock
door-open
button-press
75
Success %
75
Success
50
50
500
25
25
0.0
1.0
0.8
Enw Frames
(x10t)
(x10*)
Figure 12: Comparison Against RL with LIS Rewards.
16

---

Published as a conference paper at ICLR 2024
As shown in Figure 12 the performance of LIS is poor compared to other proxy- reward--bwade methods. The reason for this underperformance may be due to the fact that LIS ignores the fine- grained differences between the frames observed by the agent and the expert. For example, even if the agent's trajectory has already formed perfect frame-by- (i.e., for all the agent's frame is nearest to the ith frame in the demonstration), the learned policy may be still suboptimal, but LIS cannot provide more detailed feedback to further optimize the policy in this case. Hence, LIS is inappropriate to directly serve as the reward signal for the agent's policy learning.
alignment with the demonstration
C.4 COMPARISON AGAINST CONTINUOUS CURRICULA ON TRUNCATION LENGTH
In this section, we test the method of putting a continuous scheduler on truncating the expert trajecto- ries instead of on the discount factor. The time horizon given to the agent also changes accordingly. We evaluate with two schedulers:
1. A manually designed scheduler that linearly increases the truncation length in IM environ- ment frames.
2. An adaptive scheduler that sets the truncation length to the output of the progress recog- nizer.
The results shown in Figure 13 indicate that both schedulers demonstrate improvements over pure OT, which again validates the core argument presented in our paper. However, they still fall short when compared to OT+ADS, and we provide an analysis of the reason why this happens as follows.
In the truncating method, when the scheduler assigns an underestimated truncation length, the agent can not receive feedback for imitating later behaviors, and the policy learning will temporarily get stuck. To achieve satisfactory performance, it is necessary to have a high-quality scheduler that will seldom underestimate the truncation length, which requires significant efforts hyperparameter tuning. On the other hand, ADS is a softer instantiation of our high-level idea, as the later rewards not entirely excluded. Ablation studies in Section [5.] also support that ADS is not sensitive to the hyperparameters of the scheduler. Therefore, we prefer discount factor scheduling to truncation length scheduling.
OT + ADS (Ours)
OT continuous truncation curricula (linear in 1M frames) OT Co+tionnuuuus truncation curricula (adaptive)
OT
assembly
hammer
basketball
100
100
80
80
80
60
Success
Success
60
40
40
20
20-
20-
0.5
0.0
1.0
Env Frames (x109)
Enw Frames (x11f)
Env Frames (x10f)
Figure 13: Comparison against continuous curricula on truncation length.
C.5 COMPARISON AGAINST TIERED CURRICULA ON TRUNCATION LENGTH
In this section, we the method of setting tiered curricula to truncate the demonstrations. The time horizon given to the agent also changes accordingly. We set four levels of the curricula: truncate expert trajectories to 25%, 50%, 75%, and 100% of the full length, respectively. We experimented with two ways of scheduling the curricula:
1. A manually designed scheduler that updates the curriculum every 250k frames.
2. An adaptive scheduler that updates the curriculum once the progress recognizer estimates learning progress larger than the current curriculum.
The results are shown in Figure [4] Both implementations demonstrate improvements over pure OT, validating our paper's core argument. However, they still fall short when compared to OT+ADS.
17

---

Published as a conference paper at ICLR 2024
Please refer Section C.4 for further discussion between discount factor scheduling and truncation length scheduling.
ADS (Ours)
OT tired truncation curricula (+25% every 250k frames)
QT
OT tired truncation curricula (adaptive)
assembly
hammer
basketbal
100
100
100
60
Success
60
Success
60
40
40
20
0.0
0.3
1.0
13
2.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
Env Frames (x10")
Enw Frames (x10*)
Env Frames (x10")
Figure 14: Comparison against tiered curricula truncation length.
C.6 COMPUTE COSTS WITH R3M REPRESENTATIONS
In this section, we conduct experiments in which we utilize a different visual encoder for cost com- putation. Specifically, we employ ResNet-50 pretrained by R3M (Nair et al. 2022) to compute the costs. The results of the experiment are illustrated in Figure 15 They indicate that, while the performance worse than the original setting with an ImageNet pretrained ResNet-50 network, the positive effects of ADS still hold.
OT+ (ResNet)
OT (ResNet)
ATD (R3M)
OT (R3M)
assembly
hammer
basketball
100
160
100
Succers %
60
Success
60
40
40
40
20
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
Enw Frames (K109)
Enw Frames (x11F)
Env Frames (X105)
Figure 15: Results with visual encoder pretrained by another method (R3M).
C.7 COMPUTE COSTS WITH THE VISION ENCODER FINE-TUNED BY BYOL
In this section, we conduct experiments in which we utilize an in-domain fine-tuned visual encoder for cost computation. Compared with the original settings the paper, we further fine-tune the ImageNet pretrained ResNet-50 network on the expert demonstrations through BYOL (Grill et al.) 2020). The results are shown Figure 16 indicating that fine-tuning the visual encoder to the demonstration can further improve the performance, while OT+ADS still significantly outperforms pure OT.
ATDS
OT
OT+ADS BYOL finetuning)
(+BYOL finetuning)
assembly
hammer
basketball
100
100
80
Success 95
6.0
50
Succosss
60
40
20
20
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1.5
2.0
Fnv Frames (X10
Frw Frames (x100)
Env Frames (x100)
Figure 16: Results with vision encoder fine-tuned on the demonstrations through BYOL.
18


Reviews:
[{"review_id": "8xG2A6n2HN", "reviewer": "Reviewer_NwHk", "summary": "This paper proposes ADS, an imitation-learning-from-observation method that equips proxy-based reward with automatic discount scheduling. The core idea is to put a scheduler on the discount factor of the environment as the policy progresses to follow the expert demonstrations. Experiments show that the proposed method beat the selected baselines.\n\n### Post-rebuttal\nThanks for the response to my questions and for running the additional experiments. The additional experiments addressed my concerns regarding the truncation baseline, as well as comparison with other curriculum learning approach. However, I would like to maintain my score for the following reason. While I agree with the authors that it is harder to find a discount scheduler in generic RL settings compared to ILfO, RL practitioners have been trying gradually increasing the discount factor as a way to stabilize policy learning, for example, by simply looking at how episode returns are converging. Since the authors decided to formalize such a method, which I believe is good for the community -- it only makes sense to include a formal/theoretical analysis, as discount factor is such a fundamental component of any RL setting. Without it, the paper seems incomplete, in my personal opinion.", "questions": "My main question:\n\n- Why schedule the discount factor instead of expert demonstration (truncate) length\n- Implications on the RL setting when changing the discount factor\n\nAlso, I am curious to know the exact formulations of the cost functions used in the OT methods in the paper.\n\nPlease see above in the weaknesses section for details.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The presented idea is simple and well motivated.\n+ Strong empirical performance compared to selected baselines.", "weaknesses": "- While the presented idea is simple and interesting, it demands further analysis:\n  - If the goal is to first learn to follow earlier parts of trajectories first, and then move forward once policy learns, why not simply put a scheduler on truncating the expert trajectories, instead of on the discount factor? Changing the discount factor seems unnatural, especially considering that it is used together with an off-policy RL algorithm. As soon as one changes the discount factor, the target Q value for all data stored in the replay buffer changes even if one does not update the target Q network.\n- The main comparison in figure 3 does not seem fair: the baselines should be other curriculum learning approaches instead of vanilla proxy-reward approaches.\n- Scheduling the discount factor is not unique to ILfO but is generic to all RL problems. Can the authors provide more analysis on its implications in the generic RL setting? For example, how should we expect the convergence properties to change when we perform a discount factor scheduling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PTL9c9sjJQ", "reviewer": "Reviewer_K5dw", "summary": "This paper works towards the common difficulty of learning earlier behaviours in ILfO imitation learning tasks, which is due to the property of progress dependencies of ILfO. To encourage the agent to master earlier parts of demonstration before proceeding to subsequent ones, the authors propose a mechanism called Automatic Discount Scheduling (ADS). Experiments prove the idea works and brings great gain compared with SOTA approaches.", "questions": "1. How's the progress dependeny issue related to RL difficulty like instability or catastrophic forgetting?\n2. If a model-based planning is employed, will it also alleviate ILfO's problem? How does it compare with the ADS as proposed?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. As demonstrated by the paper, the problem of progress dependencies is a critical obstacle for effective ILfO learning. Several persuasive examples provided by paper illustrates this point. The proposed solution seizes a key part of the cause of this issue and posit a well-designed learning technique - ADS to avoid it. The demonstration is quite clear and algorithm design is intuitive and reasonable.\n2. Experiments are comprehensive with sufficient performance gain. Ablation study is abundant. Details are provided for possible reproduction of the results.", "weaknesses": "1. I'm quite curious about the motivation of this paper: it is clear by reading the introduction part to know that proxy reward based ILfO is susceptible to such progress dependency issue. However, the problem seems to be similar to a common issue for reinforcement learning which is called the catastrophic forgetting problem. Also classic methods like Q-learning already involves a replay buffer to avoid the possibility of being stuck by a local optimality, or the so-called instability problem of RL training. It would be more convincing to discuss the relationship between these issues and the one solved by this work.\n2. If a model-based planning is employed, will it also alleviate ILfO's problem? How does it compare with the ADS as proposed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V4VUIVUoem", "reviewer": "Reviewer_pLG5", "summary": "The paper delves into the challenge of Imitation Learning from Observations (ILfO) for robotic agents, where they must learn from unlabeled video demonstrations without knowing the expert's actions. While many convert ILfO problems into Inverse Reinforcement Learning (RL) issues using proxy rewards, the paper identifies a limitation: tasks with a \"progress dependency\" property. In such tasks, agents must first grasp the expert's earlier behaviors before mastering subsequent ones. The study finds that reward signals for later steps impede learning initial behaviors. To overcome this, the authors introduce a new ILfO framework with an Automatic Discount Scheduling (ADS) mechanism. This mechanism adaptively adjusts the RL discount factor during training, emphasizing early rewards and gradually incorporating later rewards once initial behaviors are learned. Tests on nine Meta-World tasks show this method surpasses existing techniques, even solving tasks previously deemed unsolvable.", "questions": "Although the experiments show significant improvement over the selected models, I'm interested in the following comparisons.\n1. A straightforward strategy to address the challenge of imitating sequences would be to divide the sequence into temporal slices and then imitate each slice in order. The absence of this seemingly obvious method in the comparative analysis is a missed opportunity. Including this approach in the experiments would provide a more comprehensive evaluation of the proposed ADS mechanism, especially when benchmarked against such a basic strategy.\n2. How does the model compare with RL learning with a goal-based reward?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The research stands out in its originality by identifying a previously unaddressed challenge in conventional ILfO algorithms, specifically their limitations in handling tasks with progress dependency. Moreover, the introduction of the Automatic Discount Scheduling (ADS) mechanism within the ILfO framework is a novel contribution, showcasing a creative combination of existing ideas to address a new problem.\n\nThe quality of the research is evident in its thorough approach to problem-solving. The authors not only diagnose the issue with current ILfO algorithms but also provide a robust solution in the form of the ADS mechanism. Their method's ability to outperform state-of-the-art ILfO methods in all nine Meta-World tasks further attests to its quality.\n\nThe paper clearly articulates the challenges faced by conventional ILfO algorithms, the intricacies of tasks characterized by progress dependency, and the proposed solution. The introduction of the ADS mechanism and its role in prioritizing earlier behaviors for agents is presented with lucidity.\n\nThe significance of the paper is twofold. First, it sheds light on a critical limitation in existing ILfO algorithms, broadening the understanding of the domain. Second, by introducing a solution that not only addresses this limitation but also excels in tasks previously deemed unsolvable, the research holds substantial importance for the advancement of robotic imitation learning.", "weaknesses": "At its heart, the paper's key proposition seems intuitive. Given that the objective is to imitate a sequence of actions, it's somewhat expected that there should be a dependency between actions. The current approach might be seen as a direct response to an oversight in the original problem formulation. Exploring more sophisticated reward designs or distance measurements could potentially offer a more nuanced solution to the challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "12GU1ZpfLn", "reviewer": "Reviewer_nyXG", "summary": "This work introduces \"Automatic Discount Scheduling\", a mechanism to alter the discount factor during RL training. It is argued that it is helpful to alter the discount factor in order to incentivize an agent to learn early behaviors before later ones, in cases where the reward signal is a proxy reward computed based on the agent's observations in comparison to expert demonstrations. An example of such a proxy reward is optimal transport (OT) between the expert's and agent's observations. Positive results are shown on 9 Meta-World environments (e.g., comparing OT to OT + Automatic Discount Scheduling).", "questions": "Please see Weaknesses section above. I have also included some minor additional questions here:\n\n- Have the authors experimented with using a simple curriculum learning approach? For example, e.g. Maximize OT with the first 25% of the demonstration, then the first 50%, then the first 75%, then 100% of the demonstration. How well would this perform compared to the proposed approach?\n- Have the authors experimented with other cost functions in the OT formulation (e.g. a different visual encoder), and do the positive effects of ADS still hold?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper presents a heuristic method for discount scheduling that helps overcome issues when doing imitation learning through proxy rewards on tasks that have \"progress dependencies\". While simple, the method seems to be novel and effective.\n- Results are presented on 9 tasks of various complexity in the Meta-World benchmark.\n- Ablations show comparisons of ADS to fixed discount factors and exponential discount scheduling, motivating the desire for adaptive scheduling.\n- The paper is well-written and presented clearly.", "weaknesses": "- I think the paper could state more precisely what the problem being addressed is. I appreciate the motivating example of the Basketball task in Meta-World, wherein the agent learns to grasp the ball successfully but then sweeps it away before moving towards the hoop. Does the problem lie in (1) using any \"traditional proxy-reward-based method,\" (2) using optimal transport specifically as the reward function, (3) using optimal transport with a visual encoder that does not capture task details well, and/or (4) using optimal transport over partial observations (where the partial observations are not sufficient to deduce task progress)? My feeling is that (3) is the main reason for the described behavior in the Basketball task, but the paper seems to imply that the problem is with (1), i.e., proxy reward methods in general. I think some additional clarification on this point would be valuable.\n- Related to to the above point, is the motivating example mitigated if one uses a visual encoder that is more specific to the task instead of a frozen pre-trained ResNet -- so that the similarity function induced by the visual encoder better captures task progress? It appears that (part of) the underlying problem is that there is high visual similarity in the end frames (e.g. the visual embedding is focusing on the robot in the frame and not the basketball). Would fine-tuning your visual encoder to the demonstration (as in [1]) help address this problem?\n- What is the motivation for using longest increasing subsequence a heuristic for progress alignment? As mentioned in the paper, this seems to correspond to \"macroscopic progress.\" What are the advantages to LIS over OT for the progress recognizer; and if LIS is good at measuring task progress, can we just use it for the reward function instead of OT?\n\n[1] Haldar, Siddhant, et al. \"Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations.\"  2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)