SPLID: Self-Imitation Policy Learning through Iterative Distillation
Authors: Zhihan Liu
Hao Sun
Bolei Zhou
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-goal RL
RL with sparse reward.

Abstract: Goal-Conditioned continuous control tasks remain challenging due to the sparse reward signals. To address this issue, many relabelling methods like Hindsight Experience Replay have been developed and bring significant improvement. Though relabelling methods provide an alternative to an expert demonstration, the majority of the relabelled data are not optimal. If we can improve the quality of the relabelled data, the sample efficiency, as well as the agent performance, should be improved. To this end, we propose a novel meta-algorithm Self-Imitation Policy Learning through Iterative Distillation (SPLID)  which relies on the concept of $\delta$-distilled policy to iteratively level up the quality of the target data and agent mimics from the relabeled target data. Under certain assumptions, we show that SPLID has good theoretical properties of performance improvement and local convergence guarantee. Specifically, in the deterministic environment, we develop a practical implementation of SPLID, which imposes $\delta$-distilled policy by discriminating First Hit Time (FHT). Experiments show that SPLID outperforms previous Goal-Conditioned RL methods with a substantial margin.

show that Under review as a conference paper at ICLR 2022
SPLID: SELF- POLICY LEARNING THROUGH ITERATIVE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Goal-Conditioned continuous control tasks remain challenging due the sparse reward signals. To address this issue, many relabelling methods like Hindsight Ex- perience Replay have been developed and bring significant improvement. Though relabelling methods provide an alternative to an expert demonstration, the major- ity of the relabelled data are not optimal. If we can improve the quality of the relabelled data, the sample efficiency, as well the agent performance, should improved. To this end, we propose novel meta-algorithm Self-Imitation Pol- icy Learning through Iterative Distillation (SPLID) which relies on the concept 8-distilled policy to iteratively level up the quality of the target data and agent mimics from the relaheled target data. Under certain assumptions, SPLID has good theoretical properties of performance improvement and local convergence guarantee. Specifically, in the deterministic environment, we de- velop a practical implementation of SPLID, which imposes Ó-distilled policy by discriminating First Hit Time (FHT). Experiments show that SPLID outperforms previous Conl-Conditi RL methods with a substantial margin.
INTRODUCTION
Coal-Conditioned Sparse Reward (GCSR) tasks one of the most challenging tasks in reinforce- ment learning. In these tasks, the goal is combined with the current state as the input of a policy, and only when the agent successfully achieves the desired goal will the agent be assigned with a non- trivial reward, making the task extremely challenging for mandom-xxplooation relloratioo-based algorithms. In many cases, the GCSR task is closely related to the Multi-Goal task, where the goal not fixed and can be set anywhere in the state space. In order to solve a GCSR task, an agent has to learn generalizable solution that can be applied grasping is such GCSR task: the target object can be placed at any position on the table, henceforth, the robot needs to control its arm to get closer to the position of the object and then grasps it. The objective of learning such a policy is to find a feasible path from the current state to the goal (Tamar (ct al. 2016). The work of Plappert et al. (2018) provides a series of these Condiiiond tasks in robotics control.
variety of similar tasks. For example, robotic object
In previous works, reward shaping Ng et al. 1999). hierarchical reinforcement learning (Dietterich) 2000 Barto & Mahadevan 2003). curriculum learning (Bengio al.] (209)). and learning from demonstrations (Schaal] 1997] Atteson & Schaal 1997 Argall et al. 2009; Hester et al. 2018 Nair
al. 2018) is proposed tackle the challenges of learning through sparse rewards. All of those approaches heavily rely on prior knowledge or expert demonstrations for a given task. The work of Hindsight Experience Replay (HER) (Kaelbling) 1993 Andrychowicz et al. 2017) proposed to relabel failed trajectories and assigns hindsight credits as complementary to the primal sparse rewards, to address the difficulty in exploration through the way of Temporal Difference (TD) learn- ing. Differently, the recent work of GCSL (Ghosh et al., 2019b) proposes to solve the GCSR tasks, extending HER through iterative supervised learning, i.c., there are no value networks that are opti- mized based on TD learning. However, in both HER and GCSL, there is no guarantee the quality of the relabeled data, nor there a guarantee on the policy improvement based on those relabeled data. On the other hand, the work of PCHID (Sun et al. (219)) proposes intuitive approach for GCSR tasks in a supervised learning manner, but the sample efficiency is limited by the explicit curriculum setting.

---

Under review as a conference paper at ICLR 2022
In this work, we aim at improving sample efficiency in GCSR tasks, by improving the quality of relabeled data. Notice that GCSL performs behavior cloning on all the successful experience af- ter relabeled without considering improving the quality of the relabeled data, where quality can be measured by the timesteps taken to achieve the goal or other means. Since the quality of the expert demonstration that behavior cloning relies on determines the performance and the sample efficiency of the training, it is natural to ask if we can also improve the quality in the target of the behavior cloning. To this end, we propose meta-algorithm Evolutionary Policy Distillation (SPLID), mo- tivated by the theoretical insight, and instantiate it with a practical algorithm in the GCSR tasks to further improve learning efficiency.
We summarize our contributions as follows:
To evaluate the quality of the policy in the Multi-Goal RL, we introduce the concept of d-distilled policy. Motivated by leveling up the quality of the target policy of the behavior cloning, we propose novel meta-algorithm called SPLID, which iteratively distills the target policy through pursuing d-distilled policy and conducting self-imitation learning.
From the theoretical aspect, we prove that under certain assumptions, SPLID has a mono- tonic performance improvement guarantee and local convergence guarantee.
We specialize such meta- meta-algorithm on a practical implementation in the GCSR tasks with the deterministic environment, where SPLID searches 8-distilled policy by distinguishing First Hit Time (FHT) and enhances exploration by parameter noise.
We demonstrate the proposed practical implementations on the goal-oriented tasks bench- mark and show method SPLID) can work in isolation to solve GCSR tasks with im- proved sample efficiency than previous methods involving HER, PCHID, GCSL, and ES.
2 PRELIMINARIES
Multi-Goal Markov Decision Process. We first consider a Markov Decision Process (MDP) de- noted by tuple M containing As, T, p(so), 7,),, where S is a finite discrete state space, A denotes the finite discrete action space, r(s, g): : S x G -> the reward function. In this paper we focus on a special case where the reward function is binary, i.e., tasks with sparse reward. Without loss of generalization, we also assume that reward bounded in [0, 1]. 7 E (0, 1) is the discount factor. T S x A A(S) is the transition kenrel and T(s++1 St, az) represents the probability of the state transition from 8t to St-1 given the action ae. Here we denote by (XX) distribution set defined on any measurable set X. p(so) is the start state distribution. G is the finite goal space and we assume the goal obeys the fixed distribution (-) E Ag) at the beginning of each episode. Policy of the agent takes the form Tia 8,g) : 8 g ASS) and induces the state distribution p'r()))
P'a ( g) = IP(*+ = g. We denote (= 9 I Ei=r T'p'(( (.) as the discounted average state distribution induced by π. For convenience, we assume that the environment transits to the null state with recieving reward each time step once the agent reaches the goal. For the introduction of nerual network, we parameterize the policy by Ta, where parameter lies in certain region @ E e. We denote that He (ma : E } the parameterized policy class and assume that optimal policy falls into it, i.e. E TTe-
Universal Value Function Approximator The Universal Value Function Approximator (UVFA) Schaul et al. (2015) extends the state space of Deep Q-Networks [DQN)[Mnil) et al. (2015) to include goal state as part of the input, which is useful in the setting where there are multiple goals to achieve. Moreover, (Schaul et al. (2015) shows that in the UVTA setting, the learned policy can generalize to previously unseen state- goal pairs. Specifically, let S( = Sxg denote the extended state space of M where G is a finite goal space. Normally, representation mapping m(-) : + G is assumed to be known in such multi-goal RL frameworks (Plappert et al. 2018))
Hindsight Experience Replay, Learning with sparse rewards in RL problems remains challenging due to the difficulty to reach the reward through random explorations. Hindsight Experience Replay (HER) proposes to relabel the failed rollouts as successful ones (Andrychowicz et al.) 2017) as a method to deal with the goal-oriented sparse reward problem. The agent in HER receives a reward when reaching either the original goal or the relabeled goal in each episode by storing both original transition pairs ((1t, g), at, (8e+1.9),r) r) and relabeled transitions ((st, g'), at, (8++1,9 g'), r') in the
2

---

Under review as a conference paper at ICLR 2022
replay buffer, where g' is certain hindsight goal that will be visited in the following steps, i.e., there exist & such that g' = m(((+*), and r' is the hindsight reward, which will be when the hindsight goal is achieved.
3 SELF-IMITATION POLICY LEARNING THROUGH ITERATIVE DISTILLATION
3.1 THEORETICAL FRAMEWORK
First, we consider multi-goal MDP with infinite horizon, where the discounted goal-specified return V(T,g) takes the form
V(T,g) g) = Σ (
2==
and P'z is the visitation measure induced by и. Then the target of the agent is to maximize the discounted return J(T), (x) = Egring (TT, g). This problem can be extremely hard when the goal specified reward r(::g) is sparse binary. To address this problem with sample efficiency, we are motivated by behavior cloning that mimics a policy from an oracle in a supervised learning manner, e.g. human experts. To implement such supervised learning method, one issue faced is the access to the oracle data, which is not available in our problem setting, and the quality of the oracle data also determines the performance of the mimicked policy. Although HER (Rauber iet al. 2017) proposes to relabel failure trajectory to successful one with a different goal and GCSL (Ghosh et al. 2019b) applies HER in the behavior cloning method, achieve sample efficiency. Because it considers all the policy reaching the goal as the target in the supervised learning and the quality of the target policy mimics may be burden in improving the sample efficiency. To this end, besides the combination of HER and behavior cloming as GCSL do, we desire to improve the target policy dynamically, that is assured by an oracle to be qualified to lead the present policy. Such a process corresponds to the policy distillation, where it is oriented with an evolutionary target policy selected by an oracle. We will introduce how we implement such oracle-guided policy distillation by the paradigm of behavior cloning to boost sample efficiency from the theoretical perspective. Similar to previous work (Ghosh et al. 2019b Xu et al. 2019), we only study the discrete case from the perspective of theory. To utilize the relabeling strategy (Rauber eet al. 2017) to address GCSR, we first analyze the formal definition of relabeling strategy.
inadequate for GCSL to
Definition 1 (Relabeling Strategy). We define the relabeling strategy of policy with initial goal g in the t-timestep as (9' 9 : 9 x 9 x N x IIg +> Ag)).
By definition I relabeling strategy is a time varying distribution, reflecting the relabeling preference w.r.t. the time step in one trajectory. We remark that in HER, such relabelling distribution matches the average state visitation distribution induced by the model policy n' (- g) x Ei= Px(: g)), and in GCSL it matches the last step visitation distribution (finite horizon T) i.e... g) p g),
for all g E G and æ E TT. The next step is to formally evaluate the quality of the target policy that acts the target of behavior cloning. We introduce the definition of d-distilled policy as follows.
Definition D--istiilled Policy). We define the 5-distilled policy स E Il of a policy π E Il. if the following condition satisfied:
G(+ll))
Σ (m (po()) (V(#, V(T (V(*.))-V(mmg))) 2 6, N'EG
where 8 is a nonnegative constant and (= (m)) g¹) is defined as E(oES:m(e= P (m((s) g²).
Intuitively, if स is the 8-distilled policy of æ, it means that स performs nearly strictly better than v for the goals that reached. Here the initial goal distribution (g) acts as the weight in the weighted sum, reflecting the prior favor of such performance advantage. We remark that d-distilled policy does not directly give the relationship of the return J and (TT) because of the covirate shift in the goal distribution. Nevertheless, such & d-distilled policy is easier search than the policy with higher return, since we only need to focus on the goals reached by the 5-distilled policy itself rather than the entire goal space g.
Before we move on, we first need to avoid the meaningless case when keeps zero, which is achieved by the following assumption.
3

---

Under review as a conference paper at ICLR 2022
Assumption (Avoid Keeping Zero ). We assume that supp(pg()) O supp( (m (on=-()) 11) t
for all goal g and policy He, where is the empty set.
Assumption 1 excludes the environments where the goal-oriented regions under pg are blocked away from the agents. This assumption can be easily satisfied by a wide variety of tasks in RL and multi-goal RL, e.g. goal-reaching tasks.
Based on the assumption and previous intuitive insights, we present meta-algorithm in Algorithm to solve GCSR tasks iteratively, which named Imiiationnion Policy Learning through Iterative Distillation (SPLID) and involves three main stages for each episode.
Stage 1: Exploration To search for such 8-distilled policy, we need produce new behavior policy ((k) from old model policy Tg(k-1) with the enhancement of exploration. To ensure robustness, we control such exploration span by constrainting the total variation between z and T as
sup gEG,aES
DTv (. 8,9)) Se
(1)
where e is an absolute constant. Stage 2: Distill the Target We owe oracle to judge whether this new behavior policy =(k) is our desired 5-distilled policy. If it is, then we mix (() with old target policy x x(--1) with a certain proportion, otherwise we skip this stage without changing target policy, ie., = =<(+-1) The mixture process satisfies that
Pmc = (1 - oxk Px)) + (lpP
where og E (0, 1) is the mixture proportion at the k-tth episode. Such a mixture process is also used for ensure the smoothness of the update.
Stage 3: Behavior Cloning To update parameter ((²), we conduct behavior cloning towards the o (k) after relabeling, where the relabeled strategy is If we define the surrogate function (To [| as
Fo = E Ey)pp())
Σ Ba -p (19)) (o x=
(19) B Drv (F() 1,,9 ( &,gg)),
for all policy प, E Ie, and goal g G. where (rv(plla) == E.
q(>)1/2 the total
variation. Although in behavior cloning, KI. divergence is usually adopted as the distance of polices, minimizing KL divergence also equivalent to minimize total variation by Pinker's Inequality. Then the behavior cloning process corresponds to the following optimization w.r.t. 0.
k) argmin (T (A))
(4)
dce
Algorithm 1 lmiftttmtttion Policy Learning through Iterative Distillatio((PLLID): Meta Algorithm
1: Initialize parameter ((() and target policy ạ (0) with uniform distribution.
2: for episode A 1..... K do
3: Enhance exploration from Tock 1) generate new behavior policy (8) with constraint
Equation]] [
4: Sample trajectories from the enviromment following (x)) 9,)))
/Exploration
5: if ((k) is d-distilled policy (Definition 2 of Ta(k-1) then
6:
Mix ((k) and (k-1) to obtain (k) via Equation
I/Distill the target
7:
else
8:
Keep target policy unchanged (R) = (k-1)
9:
end if
10:
Relabel the target policy (k) and conduct behavior cloning to update g(k) via Equation 4
11:
//Behavior Cloning
12:
end for
4

---

Under review as a conference paper at ICLR 2022
3.2 THEORETICAL ANALYSIS
As SPLID is aimed at boosting the sample efficiency, we analyze the theoretical performance of SPLID in this section. To begin with, assert the following assumption restrict the relabeling strategy vyk which ensures that relabeling will contribute to the policy learning.
Assumption (Proper Relabeling). It holds that
inf Byrrp g) C+ gEmpp(pe())
for all policy x Ile E IN and an absolute constant Cx only relevant with n, where supp(-) the support set.
Assumption is easy to be satisfied when the relabeling strategy is selected as random sampling with a certain probability, which has been applied in practice. Another usual method is to the visitation measure as the relabeling strategy, for example, HER applies the uniform future state g). When π near optimal, the relabeling
relabeling strategy, i.e.." (- 9) x 2'=,7'p'' strategy v of HER also satisfies this assumption.
Based the assumption, the following theorem shows the local convergence of Algorithm
Theorem (Local Convergence). For multi toali--gaalaa MDP with infinite horizon, under Assumption holds for Algorithm Ithat.
[((*)) [(ToC))) S (1- (1-7)
[J(m*) (xr(1)))
for all policies x² Th and lo E [K], where AE is defined
A =
Tao0) =
(TgN)) || (8)),
where Craass and are specified in Assumption and Equation respectively.
Proof of Theorem See detailed proof Appendix B.1
Theorem D reveals that Algorithm | can converge to the local optimality if the locally optimal pol- icy 't has the same discounted return with m(a) and the surrogate function is minimized to zero Furthermore, we establish the following theorem to show the guarantee of monotonicity on the dis- counted return.
Theorem 2 (Policy Improvement Guarantee). For multi-goal MDP with infinite horizon, under Assumption ? 2 holds for Algorithm 7 that,
2
J [Tons) - (TT()-))) 2 8
7)2 (a))(a))A----1 + AnJ,
(5)
where Ag is defined in Theorem
7 is specified in Equation 7 and ag is defined in Equation ?
Proof of Theorem See detailed proof in Appendix B.I
We motivate the proof by the following decomposition.
J(Taos)) [[Toc 1)) = J[Tgus) - [- (x) (+)) - (1)) + J(*(K))
-[(Toc 1)).
(6)
(i)
(ii)
Term in Equationnccoree corresponds the performance gap from the behavior cloning and is direct conclusion of Theorem Term (ii) in Equation 6 matches how (*() is mixed by (k--)) and (kk)
although this term requires more steps to control. Term (iii) in Equation reflects how 8- distilled policy # (R) contributes to the performance improvement.
From Theorem it suggests that once the surrogate function can be minimized near to zero and the exploration rate Is controlled, then the positive 8 on the RHS of Equation 5 will dominate, ensuring the monotonicity of the return.
5

---

Under review as a conference paper at ICLR 2022
Generate #(
Rollout using (R)
Selection with (K) Relabel
Behavior Cloning
(Supervised Learning)
(()
Mg + Z,
20,
(2. (...m) 11)-1......
at argningg
Gr -
11.
Relabeled Buffer
Figure 1: Protocol of Self-Imitation Policy Learning through Iterative Distillation (SPL.I)) in the practical implementation (Algorithm 2
3.3 PRACTICAL IMPLEMENTATION
In this section, desire to propose practical implementation of the meta-algorithm SPLID (Algo- rithm[]] Although Algorithm| I has the theoretical performance improvement and local convergence guarantee, the selection of the d-distilled policy (Line Algorithm and how to generate new behavior policy (Line 5S of Algorithm D require detailed implementations. To this end, we con- sider instantiating Algorithm iin the case where the transition kernel of the environment T and all the policies in Il are deterministic and we show that we are able to select such 8-distilled policy in a practical method by the following proposition, which characterizes the relationship between 5-distilled policy and First Hit Time (FHT).
Proposition 1. Suppose the transition kernel of the environment T and all the policy in Ill are deterministic and Assumption Shonais holds. Then t is the ö-distilled policy of if achieves goals with strictly less FHT than n, where FHT is defined as FHT(T,9) == min(t E :N: (((e)) = 8t ~ P'a)-
Proof of Proposition
See proof in Appendix B33
Based on Proposition we instantiate Algorithm I to concrete algorithm in Algorithm 2 (sce full algorithm in Appendix A), which also achieves empirical success in the continuous environments with mild stochasticity. The whole pipeline of Algorithm is illustrated in Figure For simplicity, we also name Algorithm 2 SPLID and we introduce three main stages of for each episode as follows.
Stage 1: Exploration with a New Policy SPLID utilizes the idea of parameter noise (Plappert et al. 2017) to enhance exploration to generate new behavior policy. Specifically, SPLID maintains an deterministic policy Thk-, at the (A - 1) 1) episode, parameterized as policy network. In the k-th episode new stochastic behavior policy # is generated by
# 1)(h-11 Z,Z (0.
According to Equation the behavior policy (*) comes from adding a Gaussian exploration noise upon the old model policy 1) as in previous deterministic policy learning literature (Lillicrap jet al. 2015 Fujimoto et al. 2018). Concretely, during training, (k) first interacts with the environ- ment and collects a batch of transition samples, permitting us to generate batch of state-goal sate-goall-action pairs, regardless of their optimality. These catte-ooll-cction pairs contain a set of transition tuples (s, .').), a). where g' denotes the hindsight goal. i.c,,th starting point, finally achieved the goal, and the corresponding actions are included each of these transition tuples. From an perspeppesee these state-action pairs be regarded ones generated from series of unknown deterministic individual
policies instead of known stochastic behavior policy (A) and each pair provides solution for the state-goal pair (s, g') task. Such a method enables us to obtain more alternatives for the later selection process. which increases the sample efficiency. Owing to the Equation 7 the constraint in Equation| 1 between Tack 1) a nd (() is naturally satisfied with a high probability since e in Equation] I is upper bounded with the polynomial of a with a high probability.
Stage 2: Select and Relabel As the instantiation of Line 5 in Algorithm I into Algorithm 2 by Proposition [ SELECT function chooses 8-distilled policy by comparing the FHT. To this end, empirically apply SELECT function distinguish FHT and store the fragments of the policy

---

Under review as a conference paper at ICLR 2022
with strictly less FHT in buffer B {(s: a.)..=........ The SELECT function can be implemented by resetting the environment to given previous state, which always tractable in simulation (Nair al., 2018), Specifically, SELECT function takes in episode generated by ((k), Suppose the episode (st, at, St+11 ac-1; .... S(+h) is of length h. the SELECT function resets environment to the starting state of this episode 8n and runs Tg(k-1) for up to k steps, trying to reach the final achieved state St+k- i.e.,, at every step, action of - (s. m($+++)) is performed. If To is NOT able to reach $1+K within le steps, the corresponding transition tuple (St, m($t+)), ac) will be collected in the buffer B and will update 0 from these tuples later. As horizons in the practical tasks are usually finite, we truncate the horizon by T. While in practice, we apply a notion of learning horizon H < T, that is, during selection process
also truncate FHT with H H.....FFTT = min H FHT). The introduction of H is a necessary trade-off. which will be explained from the perspectives both the theory and ablation study Section
Stage 3: Behavior Cloning Then, procedure as in Algorithm| |
apply behavior cloning update wy following the same To be specific, we use supervised learning to minimize the Mcan Squared Error between the action stored in the buffer and the action Ty predicted follows
N
= arg min TTll N
Σ (To(si
ol) - -ai²,
(8)
where N is the mini-batch size (5i, 9/ a;) is sampled from the buffer B and (s(s.)) {a E A Ta(a s, s.g) = 1}. From this point of view, the SPLID method also composed of evolution strategy and policy distillation, where a stochastic target policy ((k) acts as the perturbation on the action space and produces diverse strategies (a population), and choose those well well-performed strategies to distill their knowledge into ny (a selection).
Moreover, although plenty of policy gradient methods leverage action space noise in exploration Ei= lierap et al. ( et al. (2017); Fujimoto et al. (2018). there is clear gap between those methods: The most important difference between
perturbations and the evolutionary insight action space noise in SPLID and previous policy gradient methods is that SPLID regards those stochastic variants as from series of unknown oracle deterministic policies, then function as filter the well-performed unknown oracle policies and finally distill those transitions the model policy. On the other hand, previous policy gradient methods do contain such a se- lection step, but instead, use all those variants to estimate the value function and improve the policy through policy gradient accordingly.
selection
4 EXPERIMENTS
4.1 RESULT ON THE GOAL CONDITIONED TASKS BENCHMARKS
FecchSlide-v1
FetchWkrskndPaooe.
Figure 2: Three goal-oriented tasks involved in this paper.
We demonstrate the proposed method on the Coal-Conditioned task, Fetch benchmarks (Plappert) ctt al. 2018). Specifically, we evaluate our method on the FetchPush, FetchSlide, and FetchPickAnd- Place (see Figure for illustration and Plappert et al. (2018) 8) for detailed description). These three en- vironments are nearly deterministic, satisfying the condition Proposition B.3 They are also reset- table so that can apply SELECT function in Algorithm 2 We compare our proposed method with the HER (Andrychowicz et al.) 2017: Plappert et al. 2018) released in OpenAl Baselines (Dhariwal al. 2017). We also include PCHII (Sun al. 2019), GCSL (Ghosh et al. 2019b), and Evolution-
7

---

Under review as a conference paper at ICLR 2022
SPUD
HER
POHDD
E5
SFLID an SELECT
Push
Side
1.0
0.5
04
Succomr Rate
Susswer Rakk
12
0.1
0.2
Figure 3:
Test performance (Success Rate) of SPLID (our proposed method), GCSL, PCHID, HER, ES, and SPLID without SELECT function. Results are averaged under 5 random seeds.
ary Strategy (ES) (Salimans et al. 2017) as the baseline. To show the effect of SELECT function, include the ablation study of SPLID without SELECT function.
Figune the comparison of different approaches, where we adopt the success rate as the metric of performance. For each environment, we conduct experiments with different random seeds and plot the averaged learning curve. SPLID shows superior learning efficiency and can learn to solve the task in fewer episodes in all the three environments, compared with HER, GCSL, PCHID, and ES. Ablation study on the SELECT function of SPLID reveals the indispensable role of SELECT function in distilling the goal and reaching high sample efficiency and corresponds to the theoretical analysis in Section 3.2 In Appendix D we also plot the performance of the ablation study on SELECT function metric of the average return, which demonstrates how SELECT function plays role in searching for 5-distilled policy and improves the quality of the target of behavior cloning.
We also observe that GCSL fails in all these three environments, even performing worse than SPLID without SELECT function. Because of the difficulty of these three benchmarks, only relabeling to the last step like GCSL is inadequate for effective learning. On the contrary, SPLID relabels with all future states with selective relabeling strategy and leverages the qualify of the target policy, making able to cope with the complicated tasks. We conduct more experiments where GCSL has non-trivial performance and SPLID still outperforms GCSL these environments. The details and results in Appendix
Slde
Prckedfaac Horico
12531-4
redass rodras
redrove
vattan
8.6
0.2
60
66
Opordz
Ephede
Figure 4: Ablation study on learning horizon H.
0.50
6.6
0.30
Suggus Rube
0.25
0.2
0.0
0.0
2003
4000
6830
0900
1000
2000
4000
6300
0306
10099
4000
10490
Epoces
Epbedd
Eabad:
Figure 5: Ablation study on exploration factor σ.
Exploration Factor The exploration factor a controls the randomness of behavior policy (K) and therefore determines the behavior of generated samples. While larger helps the agents benefit
8

---

Under review as a conference paper at ICLR 2022
exploration by generating samples with large variance, smaller helps to generate biased sample with little variance. Here we need to select a proper a to balance the variance and bias and help achieve higher performance guarantee in Theorem where is determined by exploration factor o. Figure 5 shows our ablation study on the selection of different exploration factors. The results are generated with 5 different random seeds. We find in all environments, the exploration factor provides sufficient exploration and relatively high learning efficiency.
Learning Horizon In our proposed method, the parameter of learning horizon H < T determines the maximal length of sample trajectories the policy can learn from. Intuitively, smaller H decreases the learning efficiency as the policy is limited by its small horizon, making it hard to plan for the tasks that need more steps to solve. Strictly, when H is smaller, then the d in Theorem ii higher and the stricter selection makes the buffer contain less data. On the other hand, larger H will provide a better concept of the local well as global geometry of the state space, and thus the agent may learn to solve more challenging tasks. Also, large H suggests lower but probably higher dg in Theorem 2, meaning have larger buffer but with perhaps lower quality as the target of behavior cloning. Another issue of using larger H is the introduction of more interactions with the environment to gain enough buffer data with the need of more computation time. Moreover, as the tasks normally do not need lots of steps to finish, when the learning horizon is getting too large, more noisy actions will be collected and be considered as better solutions and hence impede the learning performance. Figure Seuuee]shoos shows our ablation studies on the selection of learning horizon H. The results are generated with 5 different random seeds. We find that H 8 provides satisfactory results in all of the three environments.
worth noting that PCHID can be seen as a special case of SPLID we gradually increase the learning horizon in SPLID from 1 to H. From this perspective, shown in Figure where the beginning of learning a smaller Horizon leads to poor performance, the performance of PCHID is upper bounded by SPLID.
RELATED WORK
Supervised and Self- Approaches in RL. Recently, several works put forward to pervised learning to improve the stability and efficiency of RL. (Zhang et al. (2019) propose to utilize supervised learning to tackle the overly large gradients problem in policy gradient methods. In der to improve sample efficiency, the work chose to first design a target distribution proposal and then used supervised learning minimize the distance between present policy and the target policy distribution. The Upside-Down RL proposed by (Schmidhuber (2019) used supervised learning to mapping states and rewards into action distributions, and therefore acted normal policy in RL. Their experiments show that the proposed UDRI. method outperforms several baseline methods vastava et al. (2019). In the work of [Sun et al. (2019), a curriculum learning scheme is utilized to learn policies recursively. The self-imitation idea relevant to SPLID is also discussed in the con- current work of Ghosh et al. (2019a), but SPLID further uses the SELECT function to improve the quality of collected data for self-imitation learning. Ghosh et al. (2019b) propose a new algorithm GCSL applying behavior cloning methods in multi-goal tasks with relabelling methods and SPLID is aimed at improving the sample efficiency of their algorithm through leveling up the quality of the target policy to imitate on. Due to the page limit, we postpone the remaining discussions on related works in Appendix C
CONCLUSION
This work proposes novel meta-algorithm Self-Imitation Policy Learning through Iterative Distil- lation (SPLID), which relies on the concept of 5-distilled policy to iteratively level up the quality of the target and relabelled data that the agent mimies. From the perspective of theory, we study the local convergence and monotonic performance improvement guarantee, under certain assumptions. We also implement a practical version of SPLID and test it on the Multi-Goal Benchmarks. Experi- ments show that SPLID outperforms other goal goal-oriented RL methods with a substantial margin, which also corresponds to the theoretical analysis. Since alga--tgo SPLID shares good theo- retical properties, it is worthwhile in the future work to study how to instantiate it in other practical manners.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
For the theoretical work in this paper, all the assumptions have been clarified immediately after the occurrence and all the detailed proofs of theorems can be found in the Appendix. For the empirical work, we submit zip of Jupyter Notebooks as supplementary material, where we record all the codes used for the experiments. All these efforts guarantee the reproducibility of our work.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, et al. Hindsight experience replay. NeurIPS, pp- 5048-5058. 2017.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. RAS, 2009.
Himani Arora, Rajath Kumar, Jason Krone, and Chong Li. Multi-task learning for continuous con- trol. arXiv:1802.0103: 2018.
Christopher Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML,, volume 97, pp. 12-20. Citeseer, 1997.
Andrew G Barto and Sridhar Mahadevan, Recent advances in hierarchical reinforcement learning- DEDS, 2003.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pp. 41-48, 2009.
Víctor Campos, Xavier Giro-i Nieto, and Jordi Torres. Importance weighted evolution strategies. arXiv:181.006624, 2018.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, et Improving exploration in evolution strategies for deep reinforcement learning via population of novelty- agents. In NeurIPS, 2018.
Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero, et al. Distilling policy distillation. arXiv:1902.021888 2019.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, et al. Openai baselines. httpp::::/ htttps://githbb com/ htmm/pppna//baseiiees 2017.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. JAIR, 13:227-303, 2000.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv: 180.0097 2018.
Dibya Ghosh, Abhishek Gupta, Justin Fu, et al. Learning to reach goals without reinforcement learning. vrXiiv:/92..066 2019a.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv: 1912.09912000088: 2019b.
Todd Hester, Matej Vecerik, Olivier Pietquin, et al. Deep q-learning from demonstrations. In AAAI, 2018.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1099. Citeseer, 1993.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, et al. Continuous control with deep rein- forcement learning. 109.0915,, 2015.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep rein- forcement learning. Nature, 518(7540):529, 2015.
10

---

Under review as a conference paper at ICLR 2022
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, al. Overcoming exploration in reinforcement learning with demonstrations. ICRA, 2018.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application reward shaping. ICML, 1999.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, al. Parameter space noise for exploration. arXiv: 0990, 2017.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, et al. Multi-goal reinforcement learning: Chal- lenging robotics environments and request for research. arXiv:1802.09464 2018.
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Juergen Schmidhuber. Hindsight policy gradients. 1.Xiv:7111.00006 2017.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, et al. arXiv:1511.06295 2015.
Policy distillation.
Tim Salimans, Jonathan Ho, Xi Chen, et al. Evolution strategies as scalable alternative to rein- forcement learning. 2017.
Stefan Schaal. Learning from demonstration. In NeurlPS, pp. 1040-1046, 1997.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In ICML, pp. 1312-1320, 2015.
Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards- just map them to actions. (eXiv:9112.02873 2019.
Simon Schmitt, Jonathan J Hudson, Augustin Zidek, et al. Kickstarting deep reinforcement learning.
arXiv://803003835, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06344 2017.
Rupesh Kumar Srivastava, Pranav Shyam, Filipe Mutz, al. Training agents using upside-down reinforcement learning. 1rXiv:19/2.02877) 2019.
Hao Sun, Zhizhong Li, Xiaotong Liu, Dahua Lin, and Bolei Zhou. Policy continuation with hind- sight inverse dynamics. In NeurIPS, 2019.
Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. In NeurlPS, pp. 2154-2162, 2016.
Yee Teh, Victor Bapst, Wojeiech M Czarnecki, al. Distral: Robust multitask reinforcement learn- ing. In NeurlPS, 2017.
Tian Xu, Ziniu Li, and Yang Yu. On value discrepancy of imitation learning. arXiv preprint arXiv:1911.07027, 2019.
Chuheng Zhang, Yuanqi Li, and Jian Li. Policy search by target distribution learning for continuous control. arXiv: 1041, 2019.

---

Under review as a conference paper at ICLR 2022
A PESUDOCODE: PRACTICAL IMPLEMENTATION OF SPLID
Algorithm 2 SPLID: Practical Implementation
1: Require
2: 1.A reward function r(s, g) 1 if ms)) else 0.
3: 2. Learning Horizon H.
4: Initialize replay buffer B = { and ((0)
5: for episode k ......_ K do
6: Generate new policy ((R) Tg(h-1 1) Z, Z (0,
//Exploration
7: Generate initial state and goal by the environment
8: for forl=1, 1,.,
9:
Select an action by the behavior policy (5()
Execute the action a, and get the next state $1+1
11: end for
for 1..... do
13:
for h = 1,.., H do
14:
Calculate additional goal according to s1+- by g' = m(((44) if SELECT(3t, g') True then
15:
16:
Store (st. g. az) in end if end for end for
I/Distill the target
17:
18:
19:
20: Sample a minibatch from buffer B
Do supervised learning update 0 via Equation
//Behavior Cloning
22:
end for
B DETAILED PROOF
B.1 PROOF OF THEOREM
Proof. To prove Theorem T first introduce lemma bridge the gap between the behavior cloning and performance difference.
Lemma 1. For a multi-god MDP with infinite horizon, it holds that
2|]r(:.9]l E. Bs-ne()1g) (ry (T s. ()))[ (-10) s, s,g)),,
V(n, g) (TT'' .9)1 (1 7)-
where for every goal E G and policies TT" E ll. Here Drv(pllg) is the total variation between two probability measures p and g, and p=(- g) is the discounted average state visitation measure induced by policy n, and || ol is the co-norm.
Proof. It is direct corollary of one-goal MDP case in Theorem 5.1 of et al. (2019).
To the reading, define Tate in the proof. By the definition of surrogate function in Equation it holds that
(To E. (-
E
{ (,gg))
E B-)
|s (s,9')
1=
C2 C2Eyppe
E E ( ( 8,9))
1=
2
C+ () Dry ( g. 8,9)).
12

---

Under review as a conference paper at ICLR 2022
Applying Lemma]
1
it yields that
2 (1 7)2
Fy To [T) 2 gx (V V ())l)
Cs Bor (V (TO. g) (a gg)l
Cx ((To) [(T]..
where the first inequality follows from the fact that (r(sg))] I for all (s,g) E and the last inequality relies on the fact that absolute function - is convex. Then we prove Theorem | by taking additional triangle inequality.
B.2 PROOF OF THEOREM 2
Proof. Recall the motivated decomposition in Equation 6 as follows,
J(TT(O)) J[ ((TTCC-))) = (TT()) J (TT + [ (x) (( + ( (TTe (9)
(i)
Applying Theorem | with n- = (*),, then we immediately obtain the desired result of term (i) in
Equation Then suffices to bound the next two terms.
Term (ii) in Equation 9
By the mixture rule in Equation we have that
(1) (1
((1 (T) (z)(((()) (1
(1 (J)( [(
(10)
= (1 - ank) ((T - [TToc -1) + (T) J
(iib)
First we deal with term (iia) in Equation 10 and we observe that term (iia) has the similar forms with term (i) in Equation but differs with index k. Hence we bound term (iia) by applying Theorem suggests that
2 (iia) > -C (1- 7)2
Fy (T (1
As for the term (iib) in Equation 10 recall that the (K)) is generated by Tgce-1) with constraint in Equation] Again we applying Lemma]
I we derive that
(iib) 2 J (TT((=)) (A)))
= (yrro()) () (V ( (( .g))
2 E ( ()) (V
(V( ((+(4)
2
sup Dry (Tgu-1; () s,g)
(1 7)2
s.g
2e
(1 7)2
where the last inequality holds due to Equation follows.
Hence, we lower bound term (ii) in Equation 10
(ii) > 2(1 ax)
(C F (T (1-1)) + +
(1 - 7)22
13

---

Under review as a conference paper at ICLR 2022
Term (iii) in Equation To ease the reading, denote that == =(4) and I Tg((-11) in this
proof. By the idea of importance sampling and Definition we have that
(xx ((
E y--pat) (V( (
Pg (g)
By'-po() ()))0))
(V 9))
pg 0' Pr (m (g) g¹)
pg(g)
By' (m ( po
(TT,g) (
Σ pe(g) Ry (mb) (((,g)) - V(T.9))
G[ N 8.
To sum up. plugging term (i), term (ii), and term (iii) into Equation 9 we finally obtain that
[TTOO) - (TT((-))) 2 8
Cz90 (TOC)) (
(1 7)2
2(1 - (x))
Fn (1 + e),
(1 7)2
which concludes the proof of Theorem 2
B.3 PROOF OF PROPOSITION
Proof. Under the conditions of Proposition 1 it holds that
20
V(T,g)
FHT(T.)) 1(10))/111111 (1-7)
(=.T((
and 7 E (0, 1), then if FHT(A,g) > FHT g). it holds that V V(T, g). If achieve goals with strictly less FHT than we have that for all E G, (m- (-)o)) (V((a.)) ((T, 2 do O. By Assumption Proposition holds for certain absolute constant 8.
C MORE RELATED WORKS
Evolution Strategies and Parameter Noise. The Evolution Strategy (ES) was proposed by Salii mans et al. (0117) an alternative to standard RL approaches, where the prevailing temporal differ- ence based value function updates or policy gradient methods are replaced as perturbations on the parameter space resemble the evolution. Later on, Campos al. (2018) improved the efficiency of ES by means of importance sampling. Besides, the method was also extended to be combined with Novelty-Seekinn jet al. (2017) proposed to use Parameter Noise as an alternative to the action space noise injection for better exploration. They show such perturbation on the parameter space can be not only used for ES methods, but also collected to improve the sample efficiency by combining it with traditional RL methods. While previous ES algorithms apply perturbations on the parameter noise and keep the best- best-performed variates, our approach implicitly execute the policy evolution by distilling better behaviors, therefore our approach can be regarded as an evolutionary method based on action space perturbation.
to further improve the performance (Conti al. 2018). Thereafter, Plappent
Learning with Experts and Policy Distillation. Policy Distillation (PD) was proposed extract the policy of a trained RL. agent with smaller network to improve the efficiency as well as the final performance or combine several task-specific agents together (Rusu et al. 2015). Latter extensions proposed to improve the learning efficiency (Schmitt et al.) 2018), enhance multi-task learning (Teh) et al. 2017 Arora et al. 2018). All those methods start from a trained expert agent or human expert experience that can solve specific task (Czarnecki et al. 2019). As a comparison, proposed method focus on extracting knowledge from stochastic behaviors, which is capable to act feasible policy itself with regard to the primal task. In PD, deterministic teacher policy is
14

---

Under review as a conference paper at ICLR 2022
provided and can be queried generate enough data teach the student policy. However, in SPLID all the teacher policies are unknown as they are stochastic variants of the student policy. We regard they are from unkonwn deterministic oracle policies and distill those
those stochastic variants policies through the single corresponding trajectory. In SPLID, the teacher model is unknown and can not be queried as in PD.
D DISCUSSIONS ABOUT Í-DISTILLED POLICY AND SELECT FUNCTION
SPLD
SPLID wi SELECT
FeichPush
FelthhSidd
25
15.0
Amergga Resiin
20
12.5
Areriga Retum
12.5
10.0
15
7.6
2,5
0.0
spaode
spicods
Figure 6: Average Return of SPLID and SPLID w/o SELECT function on Fetch environments. Results are averaged under 5 seeds.
In this section, we discuss the role of d-distilled policy the practical implementation of SPLID. Theorem _ reveals the theoretical guarantees of the performance of SPLID, which relies the d- distilled policy. Only a proper and absolute å can SPLID enjoy performance improvement guarantee by Theorem
- In the practical implementation of SPLID, because of Proposition B3 we pursue such 8-distilled policy by comparing First Hit Time (FHT). With the help of SELECT function, we can distinguish whether [(k) has shorter FHT than T&(k-1) and hence implement Line of meta- algorithm SPLID (Algorithm D in a practical manner. To show how the 8-distilled policy selected by SELECT function contributes to our policy learning in practice, we conduct three experiments on Fetch environments and apply average return to show the performance of SPLID and SPLID w/o SELECT function. Here return is calculated by summing over all the rewards agent receives during one episode and we take 50 episodes to average the results. Once the agent reaches in the desired goal in fewer time steps, it will receive higher return. (At each step, the agent will receive reward once it reaches the goal and will receive reward otherwise.) According to figure 6 SPLID still outperforms SPLID w/o SELECT with substantial margin, revealing the function of 8-distilled policy in improving the qualify (FHT) of trained policy.
E MORE DISCUSSIONS ABOUT SPLID AND GCSL
As discussed in Section 4. GCSL has trivial performance out of serval reasons. To validate insight, we conduct GCSL and our proposed method on three new environments: Point2D Four Rooms, Sawyer Door, and Reacher. First introduce these three environments, which are illus- trated in figure)) 7
Point2D-FourRoom- is taken from the open-source environment malti world and it requires the blue point to reach the green circle. The state space [-5, 5.5] x [-5, 5] has two dimensions representing Cartesian coordinates of the blue point, and the action space [-1, 1] [-1,1] also has two dimensions meaning the horizontal and vertical displacement. There are four rooms separated by gray walls. The goal space is the same as state space, which means m(s) = s. The bule point and the green circle are randomly initialized in the state space. The al- lowable error c of reaching goal is the radius of the target circle and is set as 1. The reward function defined as:
Foint2D-FourRomd
r (exy.gxx) ( = (I1eXY 1x
The maximum frames of episode is 50.
15

---

Under review as a conference paper at ICLR 2022
SawyerDoor-v0 The SawyerDoor-vdl environment is revised from multi-world. The Sawyer robot required to open the door to a desired angle. The state space (4-dimensional) consists of the Cartesian coordinates of the Sawyer end-effector and the door's angle. The action space is 3 dimensional and controls the position the end-effector. The desired goals are uniform from 0 to 0.83 radians. And the reward function is defined
r (SXY: :9xY) = 1 Sungge -
Gangle S c)
where the allowable enror is set as 0.06. The maximum frames of an episode is 50.
The Reacher environment is revised from OpenAl Gym (Dhariwal et al. 2017). States are I --ddimensoonal, which indicates the angles, the positions, and the velocity of the joints. Actions are 2 -dimensional and control the movement two joints. The goals are 2 2-dimension representing the expected X position. And the state-to- to-goal mapping is m(s) s[-3 -1], where the last three dimensions are the XYZ position of the end-effect. The reward function is defined as:
Reacher-v2
(IIxYY gxr 5
(sxY,gxy)
where the allowable error is set as 0.05. The maximum frames of an episode is 50.
On these three environments, derive the results in figure and figure Although GCSL has non-trivial performance, it is clear that our proposed method SPLID outperforms GCSL in terms of both success rate and average return.
conduct our proposed method and GCSL respectively and
Will
Point
Figure 7: Three new environments: Point2D Rooms, Sawyer Door, and Reacher.
GCSL
PaintDFFuuRacm
SwwyerDoor
Average Peauum
Avorage Rotum
30
20
23
15
10
0
30
103
150
200
250
200
400
600
500
1000
200
420
500
800
1000
@pisode
op 1o.dod
spisode
Figure 8: Average Return of SPLID and GCSL three environments. Results averaged under 5 seeds.
5PI
GCSL
Point2FFouRoom
SawyerDoor
0.9
1.0
0.8
Sucowee Bale
0.7
0.6
Suctwes
0.6
0.6
0.4
0.4
200
200
400
1000
apisodu
ui uplacde
Figure 9: Success Rate of SPLID and GCSL on three new environments, Results averaged under 5 seeds.
16


Reviews:
[{"review_id": "NNQc4ufLysB", "reviewer": "Reviewer_Uax1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is suffering from a large number of grammatical as well as spelling mistakes that make it very difficult to follow. This goes beyond inelegant use of language and includes puzzling mistakes in technical terms such as “discounted factor” or “total variance”. Based on this alone, the paper is not fit for publication in its current state.\n\nOverall, it is unclear what insight is to be gained from this paper. The proposed concrete implementation is similar to an evolution strategy minimizing the hitting time and as such is broadly sensible; however, it is unclear in which way the proposed method has an advantage over existing approaches such as HER. I find the experimental evaluation to be unconvincing: first, HER can be significantly improved by using a more capable reinforcement learning algorithm and the sample-efficiency gains shown in the plots may be due to this alone. Similarly, it is unclear whether the method is still improving or converging on FetchSlide; final performance on this task should be significantly higher. \n", "summary_of_the_paper": "The authors propose an approach for goal-conditioned reinforcement learning via self-imitation. The paper first outlines a meta-algorithm which defines conditions under which an instantiation of the algorithm constitutes a valid policy improvement operator. Following this, the authors propose a concrete instantiation of the algorithm which perturbs the policies parameters, selects policies which decrease the hitting time on randomly chosen goals and then distills the selected behavior into a new policy \n", "summary_of_the_review": "The paper is suffering from a large number of grammatical as well as spelling mistakes that make it very difficult to follow. Based on this, the paper is not fit for publication in its current state.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HZIxjEM-ZN0", "reviewer": "Reviewer_xVN8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- I appreciate the theoretical analysis of the method, as well as the step-by-step walk-through of the algorithm in section 3.1.\n\nWeaknesses:\n\n- Despite similarity with GCSL, GCSL performs far worse than SPLID or even SPLID without SELECT, which as far as I'm concerned is pretty much the same algorithm as GCSL with different exploration (GCSL mentions they initialize their replay buffer with random actions and they play around with different exploration strategies in section A.7, so maybe SPLID w/o SELECT could be considered the same as GCSL). Is the main difference between these methods exploration?\n- The paper makes a bold claim that \"resetting the environment to a given previous state ... is always tractable in simulation\". This seems like a major assumption and is more than the baseline methods assume. Is there a way to implement SPLID without the ability to reset the environment to arbitrary states, perhaps using value functions?\n- GCSL only optimizes for reaching the goal state by its final step, which is a slightly different problem setting than this paper. The theoretical improvement that SPLID offers is that, through the SELECT function, it will be able to learn to achieve goals in a shorter number of time-steps. However, the experiments only measure success rate, which does not take into account how optimal these polices are in terms of time taken to reach the goal.\n- The paper claims that SPLID does parameter space exploration, but they simply add noise to the actions of the previous policy, which is not parameter space exploration.\n- There is no discussion on the assumptions (deterministic, resettable environment) that SPLID makes in relation to the baselines.\n\nMy main questions are as follows:\n\n- Why does GCSL perform so poorly?\n- Can you implement SPLID without the assumptions of the environment?\n- Does SPLID w/ select learn policies that reach the goal in fewer time-steps than SPLID w/o select?\n\nSuggestions (not taken into account for the score):\n\nThere are several grammar mistakes as well as other mistakes which caused some confusion (for example, in theorem 2, readers are pointed to the wrong spot to find the definition of $\\Delta_k$). ", "summary_of_the_paper": "This paper expands on prior works in self-imitative goal-conditioned RL by proposing to selectively filter the target policy which is used during the behavioral cloning phase. By ensuring that the target policy reaches goals more optimally than the behavior policy, the paper shows that under certain conditions, performance improvement can be guaranteed. The algorithm first collects trajectories under some exploration policy, then for fragments of these trajectories, the algorithm evaluates the current policy (starting in the first state in the fragment) to see if the fragment represents an improvement on the agent's ability to reach the state. If so, it is added to the replay buffer for behavioral cloning. In experiments, the algorithm (SPLID) performs well in deterministic, goal-conditioned environments.", "summary_of_the_review": "While this paper seems to get strong performance in the selected experiments, I find the experiments to be a bit suspicious and under-explored and I am unconvinced that the performance gained with this method are worth the strong assumptions (env is resettable to a selected state). I cannot recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GaQs1sfuTka", "reviewer": "Reviewer_JnUJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Exposition is clear and the problem is interesting. Authors clearly describe a number of specific concepts necessary to understand exactly what their contribution is, especially as compared to the state of the art.\n- Authors give convincing theoretical backing to their approach, as well as conditions under which their key results hold.\n- Authors clearly demonstrate an improvement over the state of the art in 3 experiment domains, and compare against several other baselines. Authors also give ablation analysis.\n\nWeaknesses:\n- My only major concern is that the state of the art (GCSL) (Ghosh et al. 2019) somehow performs essentially the worst of any baseline in all their experimental domains. The authors state that this is due to the difficulty of the domains -- perhaps then, the authors should include some experimental domains where GCSL performs at least better than trivially poor, so that it does not give the feeling that these set of experimental domains were somehow biased toward the proposed method. Showing the relative performance of the authors' method vs. SOTA in a domain where SOTA can effectively learn would also help more clearly describe exactly where the authors' contribution is having an impact as it relates to the features of different environments or training parameters.\n\n\n---------------------\nUpdate: I have read the author response. They have adequately addressed my primary concern, so I maintain my score. I recommend that the authors move some of the experiments from appendix E to the main text, since it is more convincing that domains were balanced for the competitor as well as the proposed method.", "summary_of_the_paper": "The authors consider the Goal-Conditioned continuous control RL task, which is difficult due to reward sparsity. The authors propose a method for training deep RL agents in a more sample efficient manner through self imitation on policy traces that have been \"distilled\", i.e., randomly perturbed and argmax'd according to some metric like \"First Hit Time\", then using hindsight experience replay to relabel traces (handling reward sparsity). This improves of over the previous state of the art which did not use the policy distillation step, simply relabeling with the most recent policy, prone to error propagation. The authors demonstrate that their policy distillation step improves the state of the art on 3 RL baseline settings, and also give theoretical backing to their approach and conditions for convergence. \n", "summary_of_the_review": "I am rating accept as the writing is clear, motivations and contributions are clear, theoretical results are convincing, and experimental results are mostly good -- the only issue is that the state of the art somehow performs very poorly in the given domains. I hope the authors can either explain this in greater detail or include a domain in the final version where the state of the art is at least able to learn something.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "42XWE2sBm2", "reviewer": "Reviewer_FXhj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The method is simple yet effective.\n2. Theoretical analysis of local convergence and policy improvement is provided.\n3. The paper is well written.\n\nWeaknesses:\n1. The SELECT requires resetting the environment to a given previous state, which can only be achieved in simulator but not real-world.\n2. The overall learning process is similar to Go-Explore. But Go-Explore is not compared.\n3. SPLID w/o SELECT does not perform well, which suggests that the method highly relies on the ability to reset the simulator.\n4. It is unclear to me why it is called \"meta-algorithm\".\n5. The experiments are weak. The role of $\\delta$-distilled policy is unclear. Perhaps some case studies on the identified $\\delta$-distilled policy could facilitate the understanding of the algorithm.\n6. It is unclear how the algorithm improves the quality of \"the timesteps taken to achieve the goal\".\n", "summary_of_the_paper": "This paper presents a self-imitation learning method for goal-conditioned continuous control tasks. The key idea is to identify a $\\delta$-distilled policy that performs better than the current policy. Then a mixture policy is generated if a $\\delta$-distilled policy is identified. Finally, the parameters are updated with behavior cloning to the relabeled data. Experiments on three goal-oriented tasks demonstrate the efficiency of the proposed method.", "summary_of_the_review": "The method is simple yet effective. However, it heavily relies on the ability of reseting the environment to a previous state, which is unfair to the baselines. The experiments also need improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel meta-algorithm, called Self-Imitation Policy Learning through Iterative Distillation (SPLID) , which relies on the concept of -distilled policy to iteratively level up the quality of the target data and agent mimics from the relabeled target data.
Several aspects of the paper can be improved. The reviewers are concerned in particular about the experimental section which might not exhaust the core set of tasks, where the method should be compared with baselines. Furthermore the presentation can be significantly improved (lots of grammatical errors). Another major point is the novelty of the presented algorithm.

In the rebuttal the authors tried to address some of the remarks, in particular by adding additional experiments to the empirical section of the paper. Those experiments still do not convince some of the reviewers. Furthermore, one of the biggest concerns is still a limited novelty of the approach. The presentation of the paper still needs to be substantially improved. Thus the paper still requires nontrivial work.
Decision: Reject