Skill-Based Reinforcement Learning with Intrinsic Reward Matching
Authors: Ademi Adeniji
Amber Xie
Pieter Abbeel
Venue: Submitted to ICLR 2023 (2023)
Keywords: Unsupervised Reinforcement Learning
Reinforcement Learning
Deep Learning

Abstract: While unsupervised skill discovery has shown promise in autonomously acquiring behavioral primitives, there is still a large methodological disconnect between task-agnostic skill pretraining and downstream, task-aware finetuning. We present Intrinsic Reward Matching (IRM), which unifies these two phases of learning via the $\textit{skill discriminator}$, a pretraining model component often discarded during finetuning. Conventional approaches finetune pretrained agents directly at the policy level, often relying on expensive environment rollouts to empirically determine the optimal skill. However, often the most concise yet complete description of a task is the reward function itself, and skill learning methods learn an $\textit{intrinsic}$ reward function via the discriminator that corresponds to the skill policy. We propose to leverage the skill discriminator to $\textit{match}$ the intrinsic and downstream task rewards and determine the optimal skill for an unseen task without environment samples, consequently finetuning with greater sample-efficiency. Furthermore, we generalize IRM to sequence skills and solve more complex, long-horizon tasks. We demonstrate that IRM is competitive with previous skill selection methods on the Unsupervised Reinforcement Learning Benchmark and enables us to utilize pretrained skills far more effectively on challenging tabletop manipulation tasks.

Under review as a conference paper at ICLR 2023
SKILL-BASED REINFORCEMENT LEARNING WITH INTRINSIC REWARD MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
While unsupervised skill discovery has shown promise in autonomously acquir- ing behavioral primitives, there is still a large methodological disconnect between task-agnostic skill pretraining and downstream, task-aware finetuning. We present Intrinsic Reward Matching (IRM), which unifies these two phases of learning via the skill discriminator, pretraining model component often discarded dur- ing finetuning. Conventional approaches finetune pretrained agents directly at the policy level, often relying expensive environment rollouts to empirically deter- mine the optimal skill. However, often the most concise yet complete description of a task is the reward function itself, and skill learning methods learn intrinsic reward function via the discriminator that corresponds to the skill policy. We pro- pose to leverage the skill discriminator match the intrinsic and downstream task rewards and determine the optimal skill for an unseen task without environment samples, consequently finetuning with greater sample-efficiency, Furthermore, we generalize IRM to sequence skills and solve more complex, long-horizon tasks. We demonstrate that IRM enables us to utilize pretrained skills far more effec- tively than previous skill selection methods the Unsupervised Reinforcement Learning Benchmark and on challenging tabletop manipulation tasks.
INTRODUCTION
Generalist agents must possess the ability to execute a diverse set of behaviors and flexibly adapt them to complete novel tasks. Although deep reinforcement learning has proven to be a potent tool for solving complex control and reasoning tasks such as in-hand manipulation (OpenAl et al., 2019) and the game of Go (Silver et al., 2016), specialist deep RL. agents learn each new task from scratch, possibly collecting new data and learning to a new objective with no prior knowledge. This presents massive roadblock in the way of integration of RL in many real-time applications such as robotic control where collecting data and resetting robot experiments is prohibitively costly (Kalashnikov et al., 2018).
Recent progress in scaling multitask reinforcement learning (Reed et al., 2022; Kalashnikov et al., 2021) has revealed the potential of multitask agents encode vast skill repertoires, rivaling the performance of specialist agents and even generalizing to out-of-distribution tasks. Moreover, skill- based unsupervised RL (Laskin et al., 2022; Liu & Abbeel, 2021; Sharma et al., 2020) shows promise of acquiring similarly useful behaviors but without the expensive per-task supervision re- quired for conventional multitask RL. Recent skill-based RL results suggest that unsupervised RL can distill diverse behaviors into distinguishable skill policies; however, such approaches lack a prin- cipled framework for connecting unsupervised pretraining and downstream finetuning. The current state-of-the-art leverages inefficient skill search methods at the policy level such as performing sampling-bart optimization or sweeping a coarse discretization of the skill space (Laskin et al., 2021). However, such methods still exhibit key limitations, namely they (1) rely on expensive envi- ronment trials to evaluate which skill is optimal and (2) are likely to select suboptimal behaviors as the continuous skill space grows due to the curse of dimensionality.
In this work, we present Intrinsic Reward Matching (IRM), a scalable algorithmic methodol- ogy for unifying unsupervised skill pretraining and downstream task finetuning by leveraging the learned intrinsic reward function parameterized by the skill discriminator. Centrally, we introduce novel approach to leveraging the intrinsic reward model multitask reward function that, via

---

Under review as a conference paper at ICLR 2023
Task-Agnostic Reinforcement Learning Pretraining
Task-Aware Reinforcement Learning Finetuning
Intrinsic Reward Matching
Optimization Step
a
Z
EPIC Loss
Agent Ty(2")
Pp
R
R
t
J T1(z)
Figure I: Intrinsic Reward Matching (IRM) Framework. IRM takes place in three stages: (1) Task- agnostic RL pretraining learns skill primitives conjunction with a skill discriminator. (2) With no environment interaction, IRM minimizes the EPIC Loss between the intrinsic reward parameterized by the discriminator and the extrinsic reward with respect the skill vector 2. (3) The skill policy conditioned on the optimal 2 finetunes to task reward to solve the downstream task.
interaction-free task inference, enables us to select the most optimal pretrained policy for the ex- trinsic task reward. During pretraining, unsupervised skill discovery methods learn discriminator- parameterized, family of reward functions that correspond to family of policies, or skills, through shared latent code. Instead discarding the discriminator during finetuning as is done in prior work, observe that the discriminator is an effective task specifier for its corresponding policy that can be matched with the extrinsic reward, allowing us to perform skill selection while bypassing brute force environment trials. Our approach views the extrinsic reward distribution with measurable proximity to a pretrained multitask reward distribution and formulates an optimization with respect to skills over reward distance metric called EPIC (Gleave al., 2020).
The key contributions of this paper are summarized as follows: (1) We describe a unifying discriminator reward matching framework and introduce a practical algorithm for selecting skills without relying on environment samples (Section 3). (2) We demonstrate that our method competitive with previous finetuning approaches on the Unsupervised Reinforcement Learning Benchmark (URLB). suite of 12 continuous control tasks (Section 4.1). (3) We evaluate our ap- proach on more challenging tabletop manipulation environments which underscore the limitations of previous approaches and show that our method finetunes more efficiently (Section 4.2). (4) We gen- eralize our method sequence pretrained skills and solve long-horizon manipulation tasks (Section 4.3) as well as ablate key algorithmic components. (5) We provide analysis and visualizations that yield insight into how skills are selected and further justify the generality of our method (Section 5).
Contributions
2 BACKGROUND
2.1 UNSUPERVISED SKILL PRETRAINING
The skill learning literature has long sought to design agents that autonomously acquire structured behaviors new environments (Thrun Schwartz, 1994; Sutton al., 1999; Pickett & Barto, 2002). Recent work in competence-basee unsupervised RL proposes generic objectives encouraging the discovery of skills representing diverse and useful behaviors (Eysenbach et al., 2019; Sharma et al., 2020; Laskin et al., 2022). A skill is defined conditional policy π (a) (s, z). In order to learn such a policy, this class skill pretraining algorithms maximizes the mutual information between sampled skills and their resulting trajectories (Gregor et al., 2016a; Eysenbach et al., 2018; Sharma al., 2019):
latent code vector 2 E Z that indexes the
T(r; 2) = H(=) H(2x)) = H(T) - H(T[2)
Since the mutual information (ss; 2) is intractable to calculate practice, competence based meth- ods instead maximize variational lower bound proposed in (Barber & Agakov, 2003) which is
2

---

Under review as a conference paper at ICLR 2023
parameterized by learned neural network function qo(T, called skill discriminator. This dis- criminator, along with other terms independent of 2, parameterizes an intrinsic reward that the skill policy T(-|s, 2) maximizes during pretraining. Given an unseen task specification, the agent needs to infer which skill will finetune to solve the task with the fewest samples. For more detailed ex- planations of the mutual information decompositions of various skill discovery algorithms, refer to Appendix A.2.
Pretrained Multitask Reward Functions: We observe that the intrinsic reward function learned during skill pretraining can be viewed as a multitask reward function, where the continuous skill code 2 determines the task. In other words, we have some function:
R int (T, == VLB(T,22
(2)
where VLB < 2() is the variational lower bound proposed in (Barber & Agakov, 2003) is
trajectory representation such as (s, s')). Since skill discovery algorithms aim to maximize [(T, z), we can view its parameterized lower bound VLB as multitask reward function: scoring transitions based on their alignment with a skill code (Laskin et al., 2022).
2.2 SQUIVALENT-POLICE INVARIANT COMPARISON
We can formalize general notion of reward function similarity by equivalent-polie invariant com- parison (EPIC) as established in (Gleave et al., 2020). EPIC defines distance metric between two reward functions such that similar reward functions induce similar optimal policies. We consider the case of action-independent reward:
DEPIC RB) = RE De DD...SSS
Sc), C(RR) sp Sc, Scc
where Dp V (1-0Xx)) is the Pearson distance between two random variables X and Y. sp, s'p are samples from the Pearson distribution Dp, and Sc, Se are batches sampled from the Canonical distribution Dc. We compute the Pearson distance over Pearson samples sp, s'p, with additional canonicalization with batches Se, S² to ensure invariance over constant shifts and scaling. The canonicalized reward function is defined as:
(4) where R: SSS R is reward function. The expectation is taken over the Canonical distribution De: for simplicity, we sample these batches Sc. S6 De ahead of time. The canonicalization ensures invariance to reward shaping such that rewards that have different shaping but induce similar optimal policies are close in distance. In practice, the final term can be omitted the Pearson correlation is invariant to constant shifts and scaling.
(p. Sc, Sc) = R(sp, sp)) + E/s - - -R(Sc. Sc)l
3
INTRINSIC REWARD MATCHING
3.1 TASK INFERENCE VIA INTRINSIC REWARD MATCHING
A multitask reward function that supervise the learning of diverse behaviors is useful in its own right. However, in the case skill-based RL, we have additionally learned a corresponding Talss,)) 2). Therefore, for any "task" that can be specified by our intrinsic reward function, already have an optimal policy, so long as we condition on the corresponding skill. If we have learned sufficiently diverse library of skills, we might expect that some of skills share behavioral similarity to the optimal policy for the downstream task. It thus also holds that the corresponding intrinsic reward for that skill is semantically similar task specification to the downstream task.
Given this interpretation of intrinsic reward, we posit that the task of identifying which our pretrained skills to apply a downstream task can be reframed as inferring which task in our multitask reward function is most similar to the downstream task. Moreover, we should hope to find the skill code that produces the reward function most semantically aligned with the downstream task reward.
With this formalism, we can formulate the task inference problem as performing the following optimization:
z = arg min DEPIC (Rint (T, 2), Re (T))
(5)
2
3

---

Under review as a conference paper at ICLR 2023
in order to find most aligned with the task reward. Moreover, Equation 5 performs minimization
of a novel loss we name the EPIC loss with respect to the skill parameter 2. By EPIC's s equivalence know that if the EPIC loss is small for some 2", and T s,2) x²) is near optimal for (T," then π <aas,22 =*) approaches the optimal policy for the task as specified by Re
class invariance,
Notably, require access the task reward function Rext To compute the EPIC loss. Leveraging a known task reward function is a divergence from conventional skill selection methods.
Computing Rint during reward matching During pretraining, for some methods such (Laskin et al., 2022; Sharma et al., 2020), we require negative samples in order to compute the variational objective in Equation 2 and avoid a degenerate optimization where all embedded trajectories have high similarity with all skills. However, during selection when skills are fixed, the negative sampling component amounts to a reward offset which does not impact the task semantics. Furthermore, since we may not in general have access to large amount of negative samples a given downstream task, we choose to simplify the objective to the following:
Rint (r, 2) == VLB(T, = (T(
(6)
where qs is the skill discriminator. This parameterization of the intrinsic reward preserves the align- ment semantics of VLB without the normalization by negative samples. For more details regarding the discriminator parameterization of the intrinsic reward for (Laskin et al., 2022; Sharma et al., 2020) refer to Appendix A.3 and Appendix A.4.
Algorithm 1: Intrinsic Reward Matching (IRM)
Require: Downstream task T, Ds, Ps
Require: Pretrained policy TH (a(a,,,2) 2), intrinsic reward (ine(8, 8', 2), and extrinsic reward (E 5') for T.
Require: Optimization Nop 5000 steps and finetune NET 100K steps.
/* Skill Selection of z via EPIC Loss
for Nor steps do
Sample batch of Pearson samples Sp, Spp ~ Dr, Dp. Sample Canonical samples So, Do, Do.
for s;, & in S'p p do
Calculate EPIC Loss as DEric (n (St, 5/ 2), Fezt (S,, s))) (P(CC,
=
s Sc, S) CD. [R (s,, sy So, Sc)) as in Equation 3
end for
Take optimization step on batch with respect to (gradient descent, CEM step, etc.) in Equation 5.
end for
Evaluate zero-shot performance and finetune RL agent for NFT steps with x on downstream
task T
3.2 EPIC SAMPLE- HAMPLE-BASED APPROXIMATION
We make a number of sample-based based approximations of various unknown quantities in order to con- cretize the continuous optimization Equation 5 as tractable loss minimization problem.
Canonical State Distribution Approximation: In order to canonicalize our reward functions, we estimate the expectation over the state and next state distributions with sample-based average over 1024 samples. These distributions can be entirely arbitrary, though using heavily of-oofitriibution samples with respect pretraining can weaken the accuracy of the approximation. We choose to instantiate uniform distribution bounded by known workspace constraints for both of these distributions.
Sampling Distribution for Pearson Correlation: We find that generating samples uniformly roughly within the environment workspace bounds, just as with the reward canonicalization, often leads to strong approximations. Furthermore, both sample generation and relatively inexpensive function evaluation are independent the online-finetume finetunifinuning phase, we can perform the full skill op- timization celf-contained preprocess to downstream policy adaptation without any environment samples. Rough knowledge of workspace bounds represents some amount of prior environment knowledge. We leave general options such as sampling from learned generative model over
4

---

Under review as a conference paper at ICLR 2023
trajectories encountered during pretraining or sampling from saved pretraining data to future work. We ablate various sampling distribution choices in Table and present the full algorithm in detail in Algorithm I.
3.3 GENERALIZATION TO SKILL SEQUENCING
Many realistic downstream tasks derive additional complexity from temporally extended planning horizons. In contrast to hierarchical reinforcement learning (HRL) approaches, which aim to stitch together pretrained skills at the policy level with higher-level manager policy, can extend the task matching framework of IRM to efficiently solve the problem of skill sequencing, entirely do- ing away with the manager policy. Consider the long-horizon setting where we have sequence of reward functions to optimize over some task horizon H. Central to the finetuning problem is de- termining over what time intervals should potentially different pretrained skills be selected. In this work we predetermine a fixed skill horizon [H/N] where N is the number of rewards. This skill horizon could in principle be specified as a parameter and learned from the task reward signal.
Next, in order to perform skill selection over each time interval, we perform the IRM algorithm in parallel for each reward. We note the key assumption that IRM requires access to the reward functions for each of the subtasks. For example, for sequential goal reaching task, we divide the episode into N segments for each of the N goals and corresponding goal-reaching rewards. We then perform the IRM skill selection algorithm for each reward to select the optimal skill over each interval. After selecting the skills, we freeze our selections and finetune the skill policies jointly.
4 EXPERIMENTS
In this section we aim to experimentally evaluate whether IRM improves the adaptation sample- efficiency of skill finetuning downstream reinforcement learning task as compared to baselines. For pretraining skills, we experiment with both the CIC (Laskin et al., 2022) and DADS (Sharma al., 2020) algorithms. We consider IRM Random a version of IRM that randomly samples skills and picks the one with the lowest EPIC loss, IRM CEM which selects elites as those skills with the lowest EPIC loss, and IRM Gradient Descent which minimizes the EPIC loss using the Adam optimizer and uses backpropagation through the discriminator to regress the optimal skill.
Environments We evaluate IRM on URLB (Laskin et al., 2021), which consists of twelve downstream tasks in three challenging contin- uous control domains in the DMControl suite: Walker, Quadruped, and Jaco. We also design reaching and a tabletop pushing environment in the OpenAl Gym Fetch environment (Brock- man al., 2016) with further details in Ap- pendix A.5.
Fetch Environment
Baselines We benchmark many conventional finetuning approaches after a single skill pre- training phase of Contrastive Intrinsic Control (CIC) (Laskin et al., 2022). The Grid Search (GS) baseline coarsely sweeps each of 10 skills evenly from the all 0's skill vector to the all I's skill vector and finetunes the skill which achieves the best evaluation reward over episode. Env Rollou! randomly samples 10 skills to evaluate with a rollout and Env Rollout CEM uses the episode reward the metric by which to select clites. Random Skill selects skill at ran- dom. Relabel relabels saved skill rollouts obtained during pretraining with the task reward function, and selects the skill that achieved the highest reward. All baselines use the TD3 (Fujimoto et al., 2018) RL algorithm.
In our Fetch Push environment, we dis- cover skills that move the block in different di- rections. Downstream tasks may involve simple goals more distant goals that require compo- sition of multiple skills across an extended time horizon and around obstacles.
Figure 2:
Evaluation We follow identical evaluation to the 2M pre-training setup in URLB. First, pretrain each RL agent with the intrinsic rewards for 2M steps. Then, we finetume each agent to the downstream task with extrinsic rewards for 100k steps. Since our primary contribution involves
5

---

Under review as a conference paper at ICLR 2023
Task
IRM CEM
IRM GD
IRM
Rand
Eav
Roll.
Env
CEM
GS
Relabel
Rand
Top Leit
0,000
00000 1:00
0.000 1.000
0.186 1on
0.770
1.81 mm
0.000
0.000 10000
Jaco Top Right
0,0860 0.00
0.640 0:024
0.120 2000
7.34
34
9.82
53
16.1
0.000
800
3,50
2.
Jaco Bot. Left
0,0520
0.000
0.000
1.00
0.175
0.408 1.22
0.102
1.00
Jaco Bol. Right
2,48 4:22
0.000 a 0.00
0,360
0:331
0.086
0.00
9.07 3.
0.191
0:00
0.000
aco
0,00100 000000
Walker Stand
19.9 9:3
9.75 = 1.4
12.5 =33
18.9 3.
22,4 4.4
13.9 44
3.00
2.158
20.8 7.6
Walker Walk
5.86 1:034
7.48 0:555
15.5 5.54
14.9 2:33
13.3
9.40 1:38
4.99 1:13
15.6
4.9
Walker Run
6.82 1.0
7.17 1.23
1.007
7.92
5.87
6.56
2.67 1:025
8.81
Walker Flip
20.6 1:13
14.8 #:
17.3 2:3
23.8
17.3 1M
21.8 0.00
3.29 1:000
14.4 11
Quadr. Stand
51.5 + 12
40,3 =11
40.2 1:
40.6 997
47.5 5.
57.4 11
56.1 II
44.6 #:13
Quadr. Run
23.9 1:55
24.4 4.4
20.2 6.65
20.6 44
24.2 4:11
17.3 5:55
24.9
21.7 6:62
Quadr. Jump
38.1 #.85
41.1 994
35,9 =1
30,5 10
36,8 63
29.7 =98
35.5 9:00
33.3 #.95
Quadr. Walk
17.5 6:62
11.5 4M
17.1 6:60
19.3 27
25.5
9.21 8:30
31.3
16.4 58
Fetch Keach
95.9 + 1.1
87.5
0.20
92.5 1111
85.0 63
87.8 19
97.3 9.00
43,9 =:000
16.7 = 19
Fetch Push
80.2 25
73.1
0.45
77.6
74.3
602
75.4
72.1
23.6 0.02
51.5 125
Table 1: IRM with various optimization methods compared to environment rollout-based skill se- lection, reward relabelling of pretraining data, and random skill selection. IRM based methods rival or exceed skill selection baselines that are reliant on expensive environment trials.
skill selection, we especially focus on zero-shot episode rewards: rewards achieved by selected skill policy but without any RL updates on the task reward. We report results averaged over 5 seeds with standard error bars.
4.1 UNSUPERVISED REINFORCEMENT LEARNING BENCHMARK
In Table 1, we display the zero-shot performance of IRM-based methods compared to interaction- based methods over all 12 URLB tasks. On most of the Walker and Quadruped tasks IRM is either comparable to or outperforms the interaction baselines. Reward relabelling fails to consistently se- lect optimal skills across the benchmark, likely because its options are limited to the finite set of skills sampled during pretraining. IRM by contrast leverages continuous optimization in the skill space to find the best skill for the task. An important insight that IRM uses the environment in- teractions to immediately begin finetuning the selected skill policy instead of spending significant amounts of samples on skill selection. This allows IRM- based methods to obtain greater sample- efficiency than rollout-based methods, even when both initial skill selections obtain similar perfor- mance as demonstrated in Figure 7. Unsurprisingly, methods like IRM GD and IRM CEM tend to perform better than IRM Random which does not have the luxury of iterative refinement EPIC loss manifold as shown in Figure 5. We find that neither our method nor the baselines are well-suited for skill selection on the Jaco tasks. This is likely because these tasks are very sparsely rewarded, making it unlikely that many samples, either randomly generated as in IRM or rolled out, will consistently result in high rewards. We provide analysis demonstrating the relationship between task reward sparsity and the smoothness of the EPIC loss manifold in Appendix A.12.3.
smooth
4.2 TABLETOP MANIPUTATIOON
Reach Target We evaluate IRM on the Reach Target task, where the Fetch robot is rewarded for reaching target position. IRM outperforms closely matches environment-nollou methods while requiring no environment samples to perform skill selection. As shown in Table 1, the random skill policy performs particularly poorly and with very high variance relative to the IRM and environment- rollout based methods. Moreover, appropriate skill selection is required for strong zero-shot perfor- mance as certain skills obtain much higher rewards than others. Figure 3 shows the finetuning performance of the methods on the downstream task reward. IRM-based methods are more sam- ple efficient in reaching the optimal performance than environment-rollon based methods due to improved skill selection.
Push Block to Goal Next, we evaluate IRM on a more complex manipulation task involving pushing block to a goal position. We report the zero-shot IRM skill selection performance in Table and finetuning performance in Figure 3. This more complex task similarly benefits from bootstrapping the appropriate pretrained skill policy evidenced by the performance gap of the selection based methods over random skill selection. We remark that even for more complex manipulation tasks, IRM is robust in consistently guiding optimal skill selection without requiring any interaction with the environment. Although Env Rollout CEM is one of the stronger basclines in terms of shoo reward, it exceeds the computational budget of 100k interactions entirely on

---

Under review as a conference paper at ICLR 2023
Finetuning Performance Fetch
Fetch Reach
Fetch
IRM Random
Average Return
UM0 CEN
Descent
Env Itolout
Average
Rollout CEM
Random Skill
Environment Steps
Env ronment Steps
Fetch Reach Sequential
Fetch Barrier Sequential
Average Return
18.1 Randem tee
Dradientt Descent Her
Average
Environment Steps
Environment Steps
Figure 3: The performance gap between the IRM skill selection methods and random skill selee- tion evidences the sample efficiency gains be had from bootstrapping pretrained policy with task-level semantics similar to the task reward. IRM-based methods select optimal skills with no environment interaction and consequently finetune efficiently. Top: Fetch Reach and Block Push tasks. Bottom: Long-horizon Fetch Reach and Block Push with obstacles tasks.
skill selection. For illustrative purposes, we show the plot starting at 50k.
Task
IRM Rand Seq
IRM CEM Seq
IRM GD Seq
Env Seq
HRL
Fetch Reach Seq
88.1 I. 1.5
89.5 1 0.34
86.7 1. 0.64
80.7
28.4 31.0
Fetch Push Seq
84.9 0.12
84.9
1. 0.12
81.4 1 1.9
83.7 0.30
78.9 3.1
Table 2: Zero-shot rewards on long-horizon manipulation tasks
4.3 EXTENSIONS AND ABLATIONS
Long-Horize Manipulation Building on the results Section 4.2, we demonstrate that IRM fully generalizes to solving long-horizon tasks in the setting of tabletop manipulation. During the unsupervised pretraining phase, skill discovery methods can acquire useful skills such as directional block pushing pushing the block to certain spatial locations. We show that IRM can intelligently select a sequence of such skills to finetune via reward matching, avoiding learning a hierarchical manager policy that finetunes the policy level.
For the Fetch Reach environment, we consider an extended horizon where the agent tasked with reaching sequence of goals in particular order. For the Fetch Push task, we consider the en- vironment depicted in Figure 2, where the agent must navigate around a barrier introduced during the finetuning phase in order to reach the goal. We compare IRM methods out baseline (Env Seq) and hierarchical RL baseline (HRL). The 'IRM Seq' methods select skills based on each defined sub-task's 's reward function according the IRM optimization scheme. 'Env Seq' chooses the best combination of skills based on extrinsic reward from rollouts. "HRI'' is ini- tialized with random skills and simultaneously optimizes a manager policy over skills and the skill policies themselves. In both settings and across optimization methods, IRM outperforms the envi- ronment rollout and HRL method in identifying (Table 2) and finetuning skills (Figure 4.3 Row 2). Implementation details are provided in Appendix A.10
environment roll-
Matching Metric Ablations We validate the importance of employing the EPIC pseudometric for formulating the matching loss by ablating its contribution against more naive selections in Table 4. LI and L.2 losses are common metrics in supervised regression problems but are poor choices for comparing task similarity with rewards. Moreover, rewards have arbitrary differences in scaling

---

Under review as a conference paper at ICLR 2023
EPIC Loss and Extrinsic Reward are Negatively Correlated
Epic
Epic Losses
Epic
Losses
Losses
Figure 4: (a) Scatter plot of extrinsic reward vs. EPIC loss. (b) Trajectories with low and high EPIC losses for planar goal-reaching. (c) Trajectories for sequential goal-reaching. (d) Trajectories for Fetch Reach.
and shaping that LI and L2 are not invariant to. To strengthen these comparisons, we include learned reward scaling parameter for LI and L2 and similarly observe that EPIC is superior matching metric.
Reward Matching
IRM CEM
Skill Discovery Algorithm Ablations IRM fully general to any mutual information maximization based, RI pretraining algorithm shown in Table 3. We validate on the Fetch Reach task that IRM CEM and IRM Rand convincingly outperform all episode rollout baselines in zero-shot episode reward.
IRM L1
21.0 : 0.75
8.71 + 0.70
1.2
7.87 + 0.86
L1 Learn Scale
5.51 + 1.9
L2 + Learn Scale
3.95 = 2.2
Table Reward matching metric ablation
5 ANALYSIS
EPIC Loss Visualizations
Does optimizing the EPIC loss lead to ef- fective skill selection? Figure 4, we ver-
Epic Losses for Top Left Goal
Epic Losses for Tap Right Goal
ify that EPIC loss is strongly negatively cor-
0.75
0.75
related with extrinsic reward on a Planar Goal Reaching task detailed in Appendix A.8. Thus, optimizing for low EPIC loss an ef- fective substitute for optimizing the environ- ment reward, and crucially, it forgoes collect- ing expensive environment samples.
0.70
0.70
0.65
Epic Bottom
Epic Right Gool
0.80
How can we understand skills through EPIC losses? In Figure 5, plot EPIC losses between intrinsic rewards and goal- reaching rewards across the 2D continuous skill space. Not only is the loss landscape smooth, which motivates optimization meth- ods like gradient descent, but there is also a banded partitioning of the manifold. Further- more, the latent skill space is well-structured as different darker-colored partitions of the skill space correspond the group of skills with low EPIC loss from each task reward. EPIC losses concisely represent desirability of skills with respect
0.75
0.70
0.60
0.65
Figure 5: We examine EPIC losses between extrin- sic rewards and intrinsic rewards conditioned on the skill vector. We sweep across the 2D skill vector for pretrained planar agent.
Intr. Rew. DADS
IRM CEM
IRM GD 69.9 2.22
IRM Rand
Eny Roll
Env CEM
GS
Rand
83.4 + 2.19
77.2 + 3.83
74.6 + 5.15
70.3 + 5.38
68.9 281
28.3 19.5
Table 3: Zero-shot rewards for DADS skill discovery algorithm.
8

---

Under review as a conference paper at ICLR 2023
to downstream reward function, skills that achieve low EPIC loss for the Top Left goal will achieve high EPIC losses for the opposite reward, Bottom Right goal.
We include scatter plot and trajectory visualizations in Figure 4. As Figure 4 suggests, skills with the lowest EPIC loss receive high extrinsic reward, reaching the goal with high spatial precision. Skills with the highest losses produce the opposite behavior: moving in the direct opposite direc- tion of the goal. In the sequential case, low-EPIC loss skills attempt to reach the 1st goal then the 2nd goal, while high-EPIC loss skills perform the behavior in the inverse order. The intrinsic re- ward module provides a much deeper insight into the semantics of skills than the extrinsic rewards obtained by skill policy rollouts.
RELATED WORK
Several works including (Sharma et al., 2020; Eysenbach et al., 2019; Achiam et al., 2018; Gregor al., 2016b; Baumli et al., 2020; Florensa et al., 2017; Laskin et al., 2022) employ mutual infor- mation maximization for skill pretraining. While (Laskin et al., 2022) leverages coarse grid search select skills for downstream RL, methods such as (Sharma al., 2020) instead plan through learned skill dynamics model at finetuning time. Our approach is similar in that it leverages pre- training model components other than the policy to guide skill selection. However, rather than generating reward maximizing plan through possibly complex, learned environment dynamics, we instead look to match policy to the task reward directly through a pretrained discriminator.
In the context of sequential finetuning, (Baumli et al., 2020; Eysenbach et al., 2019) employ hi- erarchical RL to chain pretrained skills with manager policy requiring additional environment interactions. Works on such HRL methods include (Nachum et al., 2018; Frans et al., 2017; Vezh- nevets et al., 2017; Springenberg et al., 2018) and more classically (Sutton et al., 1999; Stolle & Precup, 2002). By contrast, we demonstrate that the intrinsic reward matching framework can be extended to choose skill sequences without reliance on environment samples. The successor fea- tures line of work also adopts a unified view of skill-based RL. Such work relies the assumption that arbitrary rewards can be parameterized linearly in some learned features and some task vector in (Liu & Abbeel, 2021; Barreto et al., 2016). Our approach relaxes this assumption the fully general setting by instead searching for a pretrained task with minimal proximity to an arbitrarily parameterized task reward.
7 DISCUSSION
We present Intrinsic Reward Matching (IRM), a framework for algorithmically unifying information maximization unsupervised reinforcement learning with downstream task adaptation. We instanti- ate a practical algorithm for implementing this framework and demonstrate that IRM outperforms current methods from past works in leveraging the discriminator for downstream task inference and consequently performing skill selection without environment interactions in the short horizon setting, We also show that IRM can be readily extended to the general skill sequencing setting to solve more realistic long-horizon tasks as an alternative to hierarchical methods. Central to our contribution is novel loss function, the EPIC loss, which serves as both a skill selection utility as well as new way to interpret the task- semantics of pretrained skills.
continuous control benchmark and tabletop manipulation tasks. IRM diverges
We acknowledge a number of limitations of our approach. IRM relies on samples of the state, roughly within workspace boundaries as well as access to an external reward function, ideally well- shaped, which trades off with IRM's reduced reliance on environment interactions. In order to obtain realistic image samples to compute the EPIC loss, an agent could learn an expressive gener- ative model such as VAE over the image states obtained during pretraining and sample from the model to generate diverse and realistic sampled states. For learning unknown state- bate- rewards, the agent could additionally learn image-reward model by regressing the rewards encountered during exploration (Hafner et al., 2019). This further relaxes some of the assumptions made in this contribution and represents exciting direction for future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. CoRR, abs/1807.10299) 2018. URL
abs//b////// 100007.10299000 10299.
David Barber and Felix V. Agakov. The im algorithm: A variational approach to information maximization. In NIPS, pp. 201-208, 2003. URL http:://aapess..npppooco///aaaaa
André Barreto, Rémi Munos, Tom Schaul, and David Silver. Successor features for transfer in rein- forcement learning. CoRR, abs/1606.05312. 2016. URL httptttt///axx..oorrg 6006. 05312.
Kate Baumli, David Warde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrin- sic control. CoRR, abs/2012.07827, 2020. http:: /ttp://///a.ww..///aaaa/aa
http/aaxvvv..ro/e hwtp:///gg/bb//////bo// 078.207822 07827.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function, 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference Learning Representa- tions, 2019.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. CoRR, abs/1704.03012, 2017. URL http.ttttt///aai.oorrr//////// 1744 03012.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. CoRR, abs/1710.09767. 2017. URL http:://taaatt wwtg//aa////aaoeee/e 09767.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. CoRR. abs/1802.094777, 2018. URL http::/arxii,..r org/ httpgg//bbb//////// 09477.
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. CoRR, abs/2006.133900 2020. URL http:://wwwwwww.aa///////ocoaa// 13900.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control, 2016a.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. CoRR, Abs/1611.07500 2016b. URL http: /ttp://aaav..//ee/eee/eeee..ae.a www.vorg//aaee
.75507.
Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to con- trol: Learning behaviors by latent imagination. CoRR, abs/1912.01603, 2019. URL http:
//arxiv. www.vvi.rrgaeoo /191 00............. 01603.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scal- able deep reinforcement learning for vision-based robotic manipulation. CoRR, abs/1806.10293, 2018. URL http: /aaaaaivvvvoro/. org/ /abs///////// /180//80.00 1..06.02222 10293.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski. Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic re- inforcement learning at scale, 2021. URL https: /ttp:://xxxxvvvovo 2204. 08212.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee. Albert Zhan, Kevin Lu, Catherine Cang, Ler- rel Pinto, and Pieter Abbeel. URLB: unsupervised reinforcement learning benchmark. CoRR, abs/21 15.111 2021. URL http:///aavvv..o////////////
15191.
10

---

Under review as a conference paper at ICLR 2023
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery, 2022. URL https: arxiv. org/abs. abs//202/ 00161.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features, 2021.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical rein- forcement learning. CoRR, abs/ (bs//800500826, 2018. URL htt.t://aaxiii..tr org/ abs /1005. 08296.
OpenAl, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with robot hand. CoRR, abs/1910.07 2019. URL http:://e /ttp:///axvv...//// arxiv. orgo abs 0tt//1910.011111
Mare Pickett and Andrew Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. 06 2002.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Had- sell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent, 2022. URL https::/arxiis org/ als 220.006155.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint 1907000977 2019.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
URL https: http//ppnwweeeeeeeeeee/ee net forum? id=HJg1ZR4KvH.
David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner. llya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484 489, 01 2016. doi: 10.1038/aatree16661
Jost Tobias Springenberg, Karol Hausman, Martin Riedmiller, Nicolas Heess, and Ziyu Wang- Learning embedding space for transferable robot skills. 2018
Martin Stolle and Doina Precup. Learning options reinforcement learning. In Sven Koenig and Robert C. Holte (eds.), Abstraction, Reformulation, and Approximation, pp. 212-223, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artificial Intelligence, 1. URL https: /ettss://Ae //tp:://wwr //www. ISSN 0004-3702. doi: http:///ss.aaooeo...../a//////// sciencedi rect.......... wttpccccccccccec.cc
11((1)1111-211 1999.
Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. In G. Tesauro, D. Touretzky, and T. Leen (eds.), Advances in Neural Information Processing Systems, volume 7.
MIT Press, 1994. URL https: httttp:://proceeddnng. neurips. wwtprriss.cccpppppe cc/paaprr1994/Eil 7ce3284b7- ar www.eeeeeeeeeeeeeeeeeeeoeeeeeeee
Paps----pe.... .dff.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. CoRR, abs/1703.01161, 2017. URL http: //axxiiv wttpvi..rrga/////veeeee/s 703. 01161.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.I BACKGROUND AND NOTATION
The goal of reinforcement learning is to maximize cumulative reward in uncertain environment it interacts with. The problem can be modelled Process (MDP) defined by r,, Y), where S is the set of states, A is the set of actions, P is the transition probability distribution, is the reward function and 7 is the discount factor.
Markoy Decision Process:
Markov Decision
Unsupervised Skill Discovery: In competence- based unsupervised RL the aim is to learn skills that generate diverse and useful behaviors (Eysenbach al., 2019). The broad aim that are skill-conditioned and generalizable. Formally, we also learn skills E Z and take actions according to T (-|ss, 2). As an illustrative example, applying this formalism to the Mujoco Walker domain, we might hope to find a lkill-conditionns policy and skills Awalk, 2run such that (-|ss, zwalk) makes the agent walk, while π rt|ss, Zrun) makes it run. Further, if we allow for continuous skills, we can also imagine being able to use the policy to "jog" at different speeds by interpolation the zwalk and Zrun skills. That is, taking "jog = a Zwalk + (1 a) Zrum should, intuitively, yield a policy T (-18; ) that makes the agent jog at speed dictated by the parameter a.
learn policies
Finetuning Pretrained Skills: With a skill-conditioned policy л <(1ss,)). an agent needs to infer which skill to index for downstream task (e.g. identifying if it needs to use Swalk or žrun) dur- ing finetuning. This is a relatively under-explored area, with the most universal approach being a coarse, discretized grid search. Least squares regression has also been investigated in the context of successor features (Liu & Abbecl, 2021).
A.2 COMPETENCE-BASEE BASED SKILL DISCOVERY
Competence saaset skill discovery algorithms aim to maximize the mutual information between trajectories and skills:
I(r;2) H((z) H( = H(T) - H(T[2)
Since the mutual information [(s; intractable to calculate in practice, competence- based meth- ods maximize a variational lower bound. Many mutual information maximization algorithms,, such as Variational Intrinsic Control (Gregor et al., 201 6a) and Diversity is All You Need (Eysenbach al., 2018), use the estimate T(T; H(x)) H(=lT). Other competence-based methods, such as Dynamics-Awar Unsupervised Discovery of Skills (Sharma et al., 2019), Active Pretraining with Successor Features (Liu & Abbeel, 2021), and Contrastive Intrinsic Control (CIC) (Laskin et al., 2022), maximize a lower bound for ?((T) - H((T2z).
While the decompositions of the mutual information objective equivalent, algorithms make dif- ferent design choices regarding how to approximate entropy, represent trajectories, and embed skills. These choices affect the distillation of skills: for instance, without explicit maximization of H(() in the decomposition of mutual information, behavioral diversity may not be guaranteed when the state space is much larger than the skill space (Laskin al., 2022).
A.3 CIC
Contrastive Intrinsic Control (CIC) (Laskin et al., 2022) a state of the art algorithm for competence- skill discovery. CIC maximizes a lower bound for I(r;2) H(T) H(rlz) through a particle estimator for H(T) and a contrastive loss from Contrastive Predictive Coding (CPC) (van den Oord et al., 2019) for H()]z). The lower bound for [(T; 2) is:
T(T;2) 2)) = Hparticte (T:) + E qo( 2:) log Σ exp((o(T,,
(8)
where Hpaniete (7) x Ei= log ||, - 11ll. he is the k-Nearest Neighbors embedding. N₂ is the number of k-NNS used to approximate entropy, and N 1 is the number of negative samples.
12

---

Under review as a conference paper at ICLR 2023
A.4 DADS
We additionally use Dynamics-Aware Unsupervised Discovery of Skills(DADS) (Sharma et al., 2020) for skill discovery, it is of the few skill discovery algorithms successfully scale up to continuous skills. DADS maximizes a lower bound for T(;; = H(T)) H(T[z) through learning skill-conditioned transition distributions. The lower bound for [[: 2) is:
[(T; 2 (TADD(T:2) (T == =log g0(5'|ss 2)
+ log L E (0 s,
(9)
For our experiments, we reimplement the on-policy DADS algorithm in PyTorch. We follow the default hyperparameters and train for 20 million environment steps, per (Sharma et al., 2020).
A.5 ENVIRONMENT DETAILS
The URLB domains are Walker, Quadruped, and Jaco. Walker requires a bipedal agent to perform variety of navigation based tasks on a 2D-plane while preserving its balance. Quadruped, a more challenging domain due to nigher-dimensional state-action space, requires quadrupedal agent to perform navigation tasks in a 3D environment. Jaco robot arm is 6-DOF manipulator with three-finger gripper which contains variety of directional reaching tasks
For URLB (Laskin et al., 2021) environments, we follow default environment settings. Like many skill-discovery methods (Sharma et al., 2020) (Eysenbach et al., 2019), we restrict the discriminator input. For Quadruped, we use the 2, y. 2 velocity, which included in the environment's state space. For Walker, we use the z, y, world-position, which we add to the environment's 's state space but remove from the policy input. For Jaco, we use the 2, y, z world position.
For our fetch reaching environment, 2016). We set the time limit to 200. For the fetch push environment, we partition the continuous action space into 4 actions, which involve pushing the block forward, backward, left, and right. We set the time limit to 10 for skill learning.
the Gym Robotics Fetch environment (Brockman al,,
We evaluate sequential skill selection 2 environments: Fetch Reach and Fetch Push. For the Fetch Push task, we fix 3 waypoints, depicted in Figure 2 and fix a time horizon of 15 pushes per waypoint. For Fetch Reach, we consider 2 waypoints and a time horizon of 25 for each waypoint.
Our plane environment is 2D world with observations in [-128, 128] x [-128, 128] and continuous actions in [-10, 10] [-10, 10].
A.6 PRETRAINING HYPERPARAMETERS
For the Jaco domain we use a skill dimension of 2 and a discriminator MLP hidden dimension of 64. We use an alpha value of 0 for the entropy weighting in (Laskin et al., 2022). We input the 3D position of the end-effector of the Jaco arm to the discriminator. For the Walker domain we use a skill dimension of 2 and discriminator MLP hidden dimension of 256. We use an alpha value of 0.7 for the entropy weighting. We input the displacement in the 3D position of the torso of the walker to the discriminator. For the Quadruped domain we use skill dimension of 16 and a discriminator MLP hidden dimension of 128. We use alpha value of 0.5 for the entropy weighting. We input the 3D velocity of the body of the quadruped to the discriminator. We use a learning rate lc-4, critic target tau parameter of 0.01, and constant standard deviation exploration schedule of 0.2. The rest of the RL hyperparameters are as in (Laskin et al., 2021).
For the Fetch Push environment, we use a skill dimension of 16 and a discriminator MLP hidden dimension of 16. We use an alpha value of 0 for entropy weighting. For the Fetch Reach environ- skill dimension of 8 and discriminator MLP hidden dimension of 64. We use alpha value of ( for entropy weighting. For all environments, we use a replay buffer size of 100k.
ment,
A.7 INTRINSIC REWARD MATCHING AND ENVIRONMENT ROLLOUT BASELINE HYPEEPARAMETERS
IRM CEM and Env Rollout CEM are trained for 5 iterations with 1000 samples at each iteration and 100 elites selected each iteration. Env Rollout CEM consumes the entire downstream finetuning
13

---

Under review as a conference paper at ICLR 2023
Zero-Shot Performance for Planar Goal Reaching
Planar Goal Reaching
Planar Goal Reaching
Planar Goal Reaching
Zoro shot Average Retura
Zaro- Paro-dhoot Average Return
Zero Awerace Retur
(a) Skill dimension 2
(b) Skill dimension 8
(c) Skill dimension 16
Figure 6: Zero-Shot Returns for Planar Goal Reaching averaged over seeds
budget on just skill selection. For illustrative purposes, we start its plot at 50k steps to show that finetuning still occurs, however, sample-inefficinncy suffers due excessive rollouts for skill selec- tion. This problem only worsens for long time horizons. IRM Gradient Descent is trained for 5000 steps with a learning rate of 5e-3 and initialized at the skill vector of all 0.5s. IRM Random selects 100 random skills. Env Rollout trials 10 random skills for fully episode. Grid Search coarsely trials 10 skills from the skill of all Os to the skill of all Is as in (Laskin et al., 2021).
A.8 PLANAR GOAL REACHING
The planar goal reaching task consists of simple 2D plane with point with a 2D Cartesian state space that can displace the x and y coordinates with a 2D action space. Skills learned tend span the 2D space reaching to diverse locations distributed broadly across the environment. We show some sample zero-shot skill selection results over three different skill dimensions in Figure
A.9 FINETUNING PERFORMANCE ON URLB
In Figure 7 we compare the finetuning sample-efficiency of IRM methods against environment rollout-baselines on the URLB Walker tasks. IRM performs skill selection with 0 environment interactions. The episode length of the URLB environments is 1000, meaning that in order to eval- uate single skill, rollout based methods must exhaust 1000 environment steps (i.e. grid search spends 1000 10 = 10,000 environment steps 10 percent of the available finetuning budget). By contrast, our method immediately uses new environment steps for improving the policy. As a result, the IRM based approaches generally achieve greater sample efficiency, even when initial skill se- lection obtains similar performance to the rollout based methods. For illustrative purposes we have shown Env CEM starting at 50k steps even though it far exceeds the 100k sample budget to select skill before making any RL updates due to having to execute full episode rollouts in the inner loop of optimization. This issue worsens with increasing episode lengths. We plot results over 3 seeds with standard error shading.
A. SEQUENTIAL SKILL SELECTION
For sequential skill selection, we compare IRM Sequential and Environment Sequential skill se- lection. IRM Sequential consists of an iterative process. The first skill chosen entirely free of environment samples, exactly identical to the single-skill tasks. Once the first skill is chosen, we roll out a trajectory with the skills we have chosen so far and use the latter half of the trajectory as the Pearson samples for our EPIC loss. We Gaussian noise with variance for our Canonical samples as described in Appendix A.1 At each step of the skill selection process, we use the corresponding IRM optimization methods.
For our Environment Sequential skill selection method, we select skills iteratively as well. For each waypoint subtask, randomly sample N skills and commit to the best, where N 10/n subtasks.
14

---

Under review as a conference paper at ICLR 2023
Finetuning Performance URLB
IMM Random
IMM Gradient Descent
Rollout
Grid Search
Skill
vide
1000
1000
Figure IRM finetuning results compared rollout- based baselines on Walker URLB tasks.
Skill Dim
IRM CEM
IRM
IRM Rand
Env
Roll.
Env CEM
GS
Rand
8
21.1 0.51
13.7 = 1.61
18.9 0.18
18.4 + 0.18
18,8 # 0.48
17.9
0.101
13.5 = 1.85
16
17.4 1,30
14.6 0.63
18.8 0.26
22.7 0.33
23.1 0.36
14.0 0.19
11.2 2.32
32
20.1 0.54
22.537 025
19.8 0.14
22.2 0:58
21.5 0.67
24.0
19.9
64
21.9
1.68
20.9
22.5
21.6
18.2
13.3
Table 5: IRM methods and environment rollout methods ablated over multiple skill dimensions on Fetch Push
A.11 HIERARCHICAL REINFORCEMENT LEARNING BASELINE
order to validate the benefits IRM's offline skill selection, we compare against a baseline that leverages conventional hierarchical RL algorithm to solve long-horizon, sequential tasks. We instantiate a TD3 manager agent that outputs into skill action space from state input at a temporally abstract timescale. As in the IRM setup, this timescale is fixed to align with the changes in reward to encourage the manager to change its skill prediction according to the change in the reward semantics. The manager's is then inputted to the low-level pretrained skill policy which is rolled out many steps with the skill fixed. Both the manager policy and the low-level policy weights are updated during finetuning. The manager agent is randomly initialized such that its initial skill prediction is random.
A.12
ADDITIONAL ABLATIONS
1.1 SKILL DIMENSION
We ablate skill dimension and evaluate the zero-shot performance of all skill selection methods. IRM's performance generally increases with increased skill dimension despite discriminator over- fitting issues associated with larger skill spaces. The IRM GD learning rate is chosen as 5e-3 for all experiments in this work and is not tuned all. Such likely explains the divergence of the 64 dimensional result.
A. 1.2 PEARSON & CANONICAL. DISTRIBUTIONS
We experiment with many ways to approximate the Pearson and Canonical distributions. We defined Full Random be uniform samples from reasonable estimate of the upper and lower bounds
15

---

Under review as a conference paper at ICLR 2023
for each dimension of the state. For planar environment, the bounds defined explicitly and thus known; for more complex environments, we estimate the bounds. For example, for a tabletop sample 2-dimensional block positions uniformly within the rectangular plane of the table surface. In practice, IRM fairly robust to the distributions, though there are sub- tleties that emerge in the various choices for the Pearson and Canonical distributions. For instance, we also ablate a Uniform(0,1) distribution, which generally performs much worse, due to lack of state coverage for most environments. For the Canonical distribution, also approximate samples by perturbing the Pearson samples by sampled from Gaussian distribution. We experiment with nyperparameters of variance, which may adjusted based on the environment. For our sequential IRM method, we use this Canonical distribution to ablate on-policy samples.
manipulation workspace,
Pearson Distribution
Canonical Distribution
IRM CEM
Full Random
Full Random
20.341 1 0.306
Full Random
Uniform(0.1)
16.343 0.708
Full Random
c <~ NV,
21.191 1: 0.629
Full Random
< (O0 0.1)
21.027 0.419
(niform(0.1) Uniform 001)
e V~,,
5.905 = 3.157
c~ 0.1)
2.851 0.605
Table 6: EPIC Loss Sampling Distribution Ablations.
None of the distributions ablated above require on-policy environment samples. It is possible to use on-policy samples for the state distributions, and we choose do so for our sequential IRM method, as previous skill rollouts may provide useful Pearson samples for the subsequent skill selection. Note that while on-policy Canonical samples are possible, they are incredibly expensive and require access to the environment simulator, we focus on other choices of distributions.
A. 1..3 SPARSE REWARD ABLATION
We ablate our planar EPIC Loss visualizations with sparse rewards. Instead of well-shaped goal- reaching reward, we use sparse rewards based on the tolerance to the goal. We define the tolerance as the radius the agent must be within if our 2d planar environment is scaled to [0, 1] [O, 1]. With very sparse reward, we show that EPIC losses are largely uninformative. However, by slightly relaxing the tolerance, we show much better shaped EPIC loss landscape that bears similarity to that of Figure 5. Thus, while our method is dependent access to extrinsic rewards, and idcally, shaped rewards, we show that the EPIC loss landscape over sparse reward landscapes with sufficient tolerance can be optimized.
EPIC Loss Visualizations
Epic Losses Top Left
Epic Losses Right Goal
Top Left
Epic Losses
Right
0.5
0.8
0.72
0.6
0,70
6.71
0.4
0.68
0.2
Epic Losses for Bettom Left Gool
Epic Losses for Bottom Right Gool
Epic Losses for Bottom Left Soal
Epic Losses for Bottom Right Goal
0.5
0.5
0.70
0.70
0.85
Figure 8: We examine EPIC losses between extrinsic rewards and intrinsic rewards conditioned on the skill vector. We sweep across the 2D skill vector for pretrained planar agent. Left: Sparse goal-reaching reward with tolerance of 0.03. Right: Sparse goal-reaching reward with tolerance of of 0.07.
16


Reviews:
[{"review_id": "q1IdmqSJ8go", "reviewer": "Reviewer_1F21", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the field of unsupervised skill discovery, the authors focus on that the skill discriminator is often used for generating intrinsic reward signals and propose to leverage the skill discriminator to match the intrinsic rewards with extrinsic rewards to solve downstream tasks. They use the EPIC pseudometric to measure distances between intrinsic and extrinsic reward functions, and test a number of methods for minimizing the distances and choosing optimal skills. The empirical evaluation and analysis are done in URLB and the Fetch environment.", "summary_of_the_review": "I like the idea of exploiting the skill discriminator and the reward pseudometric to select optimal skills for downstream tasks and the experiments and analyses presented in the paper. On the other hand, as I mentioned in the Strength And Weaknesses section, to my understanding, it requires the additional information. I believe it would be fairer to state such additional assumptions or difference in the problem settings, because the knowledge of extrinsic reward functions on new tasks is not necessarily cheaper than environment interactions.", "strength_and_weaknesses": "Strengths\n\n- The idea of using distances between intrinsic reward functions induced by the skill discriminator and extrinsic reward functions in downstream tasks to choose one of the pre-trained skills is novel to some degree.\n- Experimentation with the proposed approach is thoroughly done, which provides the empirical evaluation and analysis of the method in multiple aspects.\n\nWeaknesses\n\n- If I'm not mistaken, one important weakness of this work is that it requires more information about the environment and downstream tasks, compared to most prior skill discovery approaches, which makes the underlying problem settings different. The computation of the EPIC loss requires the ranges of state dimensions (or the state distribution) and the extrinsic (i.e., downstream task) reward functions, which, especially the latter, are usually not assumed by prior approaches.\n- In the writing of the paper, the authors should be clear about the distinction in the problem settings. They mention that this approach can \"determine the optimal skill for an unseen task without environment samples\" and is sample-efficient (in the abstract, Sec.3.2, Sec.4.2, etc.), but determining the extrinsic reward function without given knowledge requires or even may not be easily possible (e.g., in sparse-reward environments) with samples from environment interactions.\n- The writing could be improved in some minor ways. For instance, I believe \"invariant on an equivalence class of reward functions that always induce the same optimal policy\" in Sec.2.2 is quoted from the EPIC paper (Gleave et al. (2020)) as-is. Both \"fine-tune\" and \"finetuning\" appear in the same Algorithm 1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pDowZQxImgV", "reviewer": "Reviewer_FyAF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method, IRM, that finds appropriate skills for a given target task by matching the unsupervised-learned intrinsic reward and the extrinsic reward of the target task.\nThe matching procedures don’t require environment samples as in conventional fine-tuning approaches but assume black-box access to the ground truth reward function of the target task.\nEPIC loss is used to correctly measure the behavior similarity between the intrinsic reward and the extrinsic reward.\nIn their evaluation, IRM shows the same order of zero-shot and fine-tuning performance with the baselines.", "summary_of_the_review": "The proposed method is very convincing and supported by analysis results.  \nHowever, provided empirical results less support the effectiveness of the core idea.  \nCurrent results are enough to show that IRM is a working idea,\nbut the benefit of IRM, over any naive approach leveraging extrinsic reward function, is not clearly shown.  \nThus, I vote to reject this submission for now as I think the important comparison is missing,\nbut still willing to raise my score according to the rebuttal discussion.\n\n### Post rebuttal\nI appreciate the authors' efforts in the response.\nThe rebuttal response resolved my main concern about the experimental results.\nI'm convinced that IRM is the only method that can zero-shot select skills without accessing pre-training data.\nStill, given relabeling baseline provides a good intuition of the benefit of accessing the extrinsic reward function and how well EPIC is leveraging it.\nThus, I raised my rating as I agree with sharing the idea of EPIC-based skill matching with the community and leaving the study of consistently working skill matching methods to future work.", "strength_and_weaknesses": "### Strength\n- The proposed method can be a powerful tool that can facilitate zero-shot deployment of unsupervised trained skill policy into actual target tasks.\n- The idea of matching intrinsic reward and extrinsic reward seems novel and the use of EPIC loss is well-motivated. The analysis supports that EPIC is effectively matching the intrinsic reward and the target task reward.\n\n### Weakness\n- Missing baseline\n    - Given access to the extrinsic reward function, one can use it to relabel pre-training time buffer data. Then, the skill that is relabeled with the highest reward can be selected. I think this relabeling-based skill selection is the most convincing apples-to-apples comparison because it doesn’t use the idea of intrinsic and extrinsic reward matching but does use the extrinsic reward function.\n    - HRL method deserves to be a baseline for long-horizon skill sequencing\n- Empirical support is limited. \n    - Most of the comparison is not showing any consistent tendency. It is rather showing that the proposed skill selection can work worse than the grid search method or sampling-based skill selection. (table 1 all rows except Fetch Push, Figure3 first row, Figure 7). \n    - Skill sequencing experiments only show results of IRM random. Considering the inconsistency over IRM optimization methods in Table 1, the result should be presented along with other IRM optimization methods, or a reasonable model selection criteria should be given, to conclude IRM is better.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "9CgP4JZOF3", "reviewer": "Reviewer_qUWm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "There is many works that propose methods for unsupervised skill discovery, e.g. via behavior diversification. This work builds on top of these and proposes an approach for offline skill selection when using the unsupervisedly pre-trained agent for downstream tasks. Specifically, assuming access to a downstream task reward function, they find a skill that aligns the intrinsic reward used during unsupervised pre-training with the downstream task reward under a robust reward metric, EPIC. In experimental evaluations on the URLB benchmark they show that this skill selection is competitive with prior works that require online environment interactions.", "summary_of_the_review": "The paper proposes a novel and interesting approach for a relevant problem. The introduced idea of using the EPIC loss to optimize for reward alignment potentially is useful even outside the proposed application, so I think the paper can be an interesting contribution for the community. I do think that the writing could be substantially improved to make it easier to follow and the experimental results don’t show a substantial improvement over prior work. Thus, in conclusion, I am supporting acceptance but it’s only a weak accept and I am open to change my mind based on the other reviewers’ comments.", "strength_and_weaknesses": "# Strengths\n\n- the problem of skill selection is important for effective fine-tuning of pre-trained agents —> the paper presents a novel approach that can work fully offline and leads to competitive results with prior works that require online interactions —> this is a nice contribution\n\n- the use of the EPIC metric to compare rewards and *optimize* the skill choice is a novel idea and potentially useful even beyond the discussed approach, e.g. similar problems of downstream task reward alignment occur in unsupervised meta-RL approaches and aligning rewards with this robust metric could be an interesting direction to explore there too\n\n- the submission has comprehensive analysis experiments that visualize the EPIC loss for different trajectories and give a good intuition that it correlates well with downstream task success. I also appreciate the ablation studies on other, more naive reward comparison metrics that nicely justifies the added complexity of the EPIC loss\n\n\n# Weaknesses\n\nThe submission’s main weaknesses are in terms of the writing and the experimental evaluation:\n\n(A) **writing hard to follow, too many details in appendix**: the description of the method as well as the experimental evaluation is a bit hard to follow at times. One reason is that many details are pushed to the appendix, which makes It hard to understand the paper from following the main text. For example, the very beginning of the method section should first give a rough intuition for how unsupervised skill discovery methods with intrinsic rewards work and introduce how they train a discriminator to diversify skills. Instead, the paper assumes that readers are already familiar with these concepts. Similarly, in the experimental section, it should be more explicitly stated what the standard skill selection method from prior work is — I had to go through the URLB paper (Laskin’21) to figure out that grid search over skills with the first 4k environment steps is the standard approach — that should be mentioned prominently in the submission. Also baselines, like the SeqEnv are only explained in the appendix, which makes it very hard to understand the comparisons in the experimental evaluation.\n\n(B) **experimental results not very strong**: the presented experimental results show that the proposed method merely matches prior works and on most of the URLB tasks a random skill selection baseline is performing similarly. While prior works do require some environment interactions for skill selection, they use only 4k interaction steps, which is a small fraction of the 100k steps that are used for fine-tuning. Thus, the effective fine-tuning efficiency gains in Fig 3 are marginal.\n\n(C) **discussion of assumptions and limitations not explicit enough**: the paper does not clearly state the implicit assumptions of their method in comparison to prior works. In contrast to prior works, the proposed method requires access to the downstream reward function (as a trade-off vs not requiring online interactions), it assumes that we can sample a representative distribution of states (this would be very hard for image-based observations for example) and for the sequential task evaluations assumes access to per-subtask reward functions. Its effectiveness seems limited in sparse reward cases and, in contrast to prior work, cannot straightforwardly be applied to image-based environments. While it is okay to have tradeoffs like this with respect to prior work, they should be more explicitly discussed to make readers aware of them when choosing what method to use.\n\n\n\n# Questions\n\n- How does the proposed approach perform with sparse rewards? This needs some more discussion (see point on limitations above)\n\n- How does the method perform with shorter offline pre-training? URLB shows performance with 100k, 500k, 1M pertaining steps too — would be good to understand the tradeoffs of the proposed approach vs prior skill selection approaches in this regime too.\n\n- Do you see a way to extend the proposed approach to environments with image-based observations?\n\n- What are the assumptions in the sequential task case? Do you assume access to a reward function for every sub-task?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7has46zDGH", "reviewer": "Reviewer_jnVw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method that learns to choose among a set of skills to solve a downstream task. The key idea presented is that some skill learning methods often learn a discriminator which can be thought of as an intrinsic reward function for the underlying skill. Using an equivalent policy-invariant metric called EPIC, the authors suggest comparing the ‘fitness’ of skills using the corresponding intrinsic rewards to the external task-specific environment reward. This presents a zero-shot method to find a good skill for an unseen task. This method is tested on a number of benchmarks including one that involves composing skills sequentially and the intrinsic reward and EPIC loss are also analyzed.", "summary_of_the_review": "As things stand, I think the paper has too many flaws for it to be published at ICLR. I am happy to reconsider though based on the rebuttal.", "strength_and_weaknesses": "I think the idea and method being proposed in this paper are somewhat interesting and, to my knowledge, novel. However, apart from some methodological issues I have with the idea, my main concern is the quality of writing and communication in the paper. There were multiple instances when reading the paper which made me think that the work was quite rushed and for which I had to spend a lot of time looking through the Appendices in order to make sure I understood the details. While I am willing to ignore some of these, when they repeat so often it makes the paper much less enjoyable to read and, more pertinent to the context of reviewing, much harder to parse and understand. \n\nI will first list out some questions I have regarding the method and will then follow with a list of ways in which I think the communication could be improved, including areas where I think claims need to be backed up with more references to the literature. I’d also like to note that if the writing is improved, I would be more than happy to reconsider my score. \n\nMethodological questions:\n1. In the Appendix, the choices of D_S and D_A are ablated in Table 6. There seems to be quite a stark difference when switching from the ‘Uniform’ distribution for the Pearson distribution to the ‘Full Random’. My understanding was the ‘Full Random’ samples uniformly but with the upper and lower bound for each state fixed. It was not clear to me why changing the bounds would affect the sampling distribution so much unless it was quite a strong assumption. Could the authors clarify what these two distributions mean perhaps with an example?\n\n2. Unfortunately I’m not convinced the method works particularly well given the results of Table 1 and Figure 3. In Table 1, for the JACO task the Env CEM baseline seems to sometimes get a reasonably higher performance (9-10 on Jaco Top and Bottom Right). In the Walker setting a ‘Random’ skill works so well that I am genuinely a bit perplexed as to why other methods are so terrible - for instance in Walker Walk. In the finetuning results of Fetch Reach as well I would not say that IRM is that much more sample efficient. Perhaps understanding this a bit better - for instance why the method does not work so well on Walker could lead to algorithmic improvements that make it more robust. As things stand I don’t think the empirical evidence is compelling enough for this to be used in practice. \nThe Jaco tasks also bring up an interesting point regarding sparse reward tasks. Sparse rewards are easier to define but as the authors note, would be much harder to use as a way to pick skills via matching. In general perhaps the method is sensitive to the nature of the reward landscape? In particular how does the analysis of Figure 5 of the EPIC landscape change for different tasks? Currently I’m not certain if the performance is limited because R_int is not being estimated accurately enough or if the loss landscape itself is not smooth on some tasks and understanding this would be quite interesting I think.\n\n3. In Figure 3, why would we expect the finetuning performance to be different under IRM?  My understanding is that the method would try to pick the ‘most suitable skill’ in terms of how it matches with the task reward. This says nothing about the skills ability to finetune on a new task per se, does it? \n\n4. In Section 7, the EPIC loss is described as a contribution of this work. My understanding was that this loss was used, albeit in a different context, in prior work?\n\nAreas in which presentation can be improved and clarified:\n1. There are multiple places (Section 1, 2.1) with sentences in the vein of - “the intrinsic reward function learned during skill pre-training can..”. First, this notion that when using a particular formulation to learn skills, we can interpret the skill as optimizing an intrinsic reward function is somewhat new. This idea needs to be introduced and at least some context given to the reader to understand what it means. Also I don’t think the claims can be made generically across *all* skill learning methods. At least a gentle introduction of what skill learning methods this applies to would make this part much more readable. Often these sentences do not have references and citations to specific works to back them up as well which is an issue in scientific writing.\n\n2. The equations relating to the EPIC losses - Equations 2 and 3 are hard to parse and not strictly correct I think. Equation 2 includes S, A, S’ on the right-hand-side which should be input to the function. The definition in Equation 3 is also inconsistent with that of (2) and trajectories are defined as (s, s’) which means the action is not used. I understand that the reward functions being considered here do not require actions but this should be specified explicitly in the main text and in either case the mathematical definitions should be consistent.\n \n3. The second point is particularly important in the context of Algorithm 1 since these equations are the only way to understand it. In general, as far as possible, algorithm boxes should be self-contained and express the main jist of the approach. Currently in the box variables N_FT, S_p, A_p, S’_p, S_c, A_c, S’_c are defined but never used and variables s and s’ are used but not properly introduced. I think if the algorithm box just explicitly wrote out the EPIC loss from Equation 2 it would make it much more self contained and easier to parse. \n\nList of typos and other less minor issues in writing:\n1. Paragraph 3, Introduction: paramterized instead of parameterized.\n\n2. Section 2.1 after Equation (1): ‘is’ is missing after ‘where VLB<= I(\\Tau, z)’.\n\n3. Equation (3) and (4) - it equation does not include over what random variables the expectation is being taken.\n\n4. Section 3.1 paragraph 1: ‘specified our intrinsic’ should be ‘specified by our intrinsic’ and ‘As is such’ should be ‘As such’. \n\n5. Section 3.1, paragraph 2: ‘which our pretrained’ should be ‘which of our pretrained’.\n\n6. Section 3.2, paragraph 3: ‘leads improved’ should be ‘leads to improved’. \n\n7. Table 6 v/s 6, Appendix 1 v/s 1 etc. are used interchangeably throughout. It helps when the relevant sections and appendices are explicitly and consistently labeled  throughout the text. As a specific example, the ablation for P_s and P_A at the end of section 3.2 refer to ‘6’ which is unclear.\n\n8. Section 4.1, paragraph 1: ‘is either comparable to our outperforms..’ - ‘our’ should be ‘or’.\n\n9. Section 4.3: ‘We compare an extended versions’ - no ‘on’. Sentence ends with ‘...environment rollout method identifying’. Figure 4.3 Row 2 should be Figure 3?\n\n\nEDIT:\nAfter looking through the changes made during the rebuttal period I think the paper reads much much more clearly and is stronger now. There are still some nagging issues in the presentation (predominantly of Section 3) but I appreciate that the authors also ran experiments for other baselines and it is challenging to satisfy all constraints within a short time limit. I have increased my scores to reflect the updated paper and would like to thank the authors for the great work in addressing concerns from the rebuttal period.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject