Reinforcement Learning with Bayesian Classifiers: Efficient Skill Learning from Outcome Examples
Authors: Kevin Li
Abhishek Gupta
Vitchyr H. Pong
Ashwin Reddy
Aurick Zhou
Justin Yu
Sergey Levine
Venue:  (2021)
Keywords: Reinforcement Learning
Goal Reaching
Bayesian Classification
Reward Inference

Abstract: Exploration in reinforcement learning is, in general, a challenging problem. In this work, we study a more tractable class of reinforcement learning problems defined by data that provides examples of successful outcome states. In this case, the reward function can be obtained automatically by training a classifier to classify states as successful or not. We argue that, with appropriate representation and regularization, such a classifier can guide a reinforcement learning algorithm to an effective solution. However, as we will show, this requires the classifier to make uncertainty-aware predictions that are very difficult with standard deep networks. To address this, we propose a novel mechanism for obtaining calibrated uncertainty based on an amortized technique for computing the normalized maximum likelihood distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions from data, while being able to guide algorithms towards the specified goal more effectively. We show how using amortized normalized maximum likelihood for reward inference is able to provide effective reward guidance for solving a number of challenging navigation and robotic manipulation tasks which prove difficult for other algorithms.

Under review as a conference paper at ICLR 2021
REINFORCEMENT LEARNING WITH BAYESIAN CLASSI- FIERS: EFFICIENT SKILL LEARNING FROM OUTCOME EXAMPLES
Anonymous authors Paper under double-blind review
ABSTRACT
Exploration in reinforcement learning is, in general, a challenging problem. In this work, we study more tractable class of reinforcement learning problems defined by data that provides examples of successful outcome states. In this case, the reward function can be obtained automatically by training a classifier to classify states as successful or not. We argue that, with appropriate representation and regularization, such classifier can guide a reinforcement learning algorithm to effective solution. However, as we will show, this requires the classifier to make uncertainty-awant predictions that are very difficult with standard deep networks. To address this, we propose a novel mechanism for obtaining calibrated uncertainty based on an amortized technique for computing the normalized maximum likeli- hood distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions from data, while also being able to guide algorithms towards the specified goal more effectively. We show how using amortized normal- ized maximum likelihood for reward inference is able to provide effective reward guidance for solving a number of challenging navigation and robotic manipulation tasks which prove difficult for other algorithms.
INTRODUCTION
While reinforcement learning (RI.) has been shown to successfully solve problems with careful reward design (Rajeswaran et al. 2018 OpenAl et 2019). RL in its most general form, with no assumptions on the dynamics or reward function, requires solving a challenging uninformed search problem in which rewards are sparsely observed. Techniques which explicitly provide "reward- shaping" (Ng et al. 1999), or modify the reward function to guide learning, can help take some the burden off of exploration, but shaped rewards can be difficult to obtain without domain knowledge.
In this paper, we aim to reformulate the reinforcement learning problem to make it casier for the user specify the task and provide tractable reinforcement learning objective. Instead of requiring a reward function designed for an objective, our method instead assumes a user-provided set of successful outcome examples: states which the desired task has been accomplished successfully. The algorithm aims to estimate the distribution over these states and maximize the probability of reaching states that are likely under the distribution. Prior work learning from success examples (Fu et al. 201800 8b) Zhu et al. 2020) focused primarily on alleviating the need for manual reward design. In our work, we focus on the potential for this mode of task specification produce more tractable RL problems and solve more challenging classes of tasks. Intuitively, when provided with explicit examples of successful states, the RI. algorithm should be able to direct its exploration, rather than simply hope randomly chance upon high reward states.
The main challenge in instantiating this idea into a practical algorithm is performing appropriate uncertainty quantification in estimating whether given state corresponds to successful outcome. Our approach trains classifier to distinguish successful states, provided by the user, from those generated by the current policy, analogously to generative adversarial networks (Goodfellow et al.) 2014) and previously proposed methods for inverse reinforcement learning (Fu et al.) 20]. In general, such classifier is not guaranteed provide good optimization landscape for learning

---

Under review as a conference paper at ICLR 2021
the policy. We discuss how a particular form of uncertainty quantification based the normalized maximum likelihood (NML) distribution produces better reward guidance for learning. We also connect our approach to count-based exploration methods, showing that classifier with suitable uncertainty estimates reduces to a count-based exploration method in the absence of any generalization across states, while also discussing how it improves over count-based exploration in the presence of good generalization. We then propose a practical algorithm to train success classifiers in a computationally efficient way with NML, and show how this form of reward inference allows us to solve difficult problems more efficiently, providing experimental results which outperform existing algorithms on a number of navigation and robotic manipulation domains.
RELATED WORK
number of techniques have been proposed to improve exploration. These techniques either add
reward bonuses that encourage policy to visit novel states in task-agnostic manner (Wiering and
Schmidhuber [9988 Auer et al., [002]] Schaul et al.) 2011 Houthooft al., [2016] Pathak al.
2017 Tang et al. 2017 Stadie et al. 2015 Bellemare et al. 2016 Burda et al.) 201 Sa
O Donoghue
2018) or perform Thompson sampling or approximate Thompson sampling based on a prior over
value functions (Strens) 2000 Osband et al. 2013 2016). While these techniques are uninformed about the actual task, we consider a constrained set of problems where examples of successes can allow for more task-directed exploration. In real world problems, designing well-shaped reward functions makes exploration easier but often requires significant domain knowledge (Andrychowicz iet al. 2020), access privileged information about the environment (Levine al. 2016) and/or a human in the loop providing rewards (Knox and Stone 2009; Singh et al. 2019b). Prior work
has considered specifying rewards by providing example demonstrations and inferring rewards with
inverse RL (Abbeel and Ng. 2004 Ziebart et al., (2008] Ho and Ermon 2016 Fu et al.
2018a). This
requires expensive expert demonstrations to be provided the agent. In contrast, our work has the minimal requirement of simply providing successful outcome states, which can be done cheaply and more intuitively. This subclass of problems also related to goal conditioned RL (Kaelbling 1993] Schaul et al. 2013] (huu et al.] 2017;; AAnryychoicc et al. [2017] Nair et al. [2018] Veeriah et al. 2018
Rauber et al. 2018; Warde-Farley et al. 20188] Colas et al. | Ghosh et al. [2019] Pong et al. 2020) but is more general, since it allows for more abstract notion of task success.
core idea behind our work using a Bayesian classifier to learn suitable reward function. Bayesian inference with expressive models and high dimensional data can often be intractable, requiring assumptions on the form of the posterior (Hoffman et al. 20133 Blundell et al., 2015 Maddox et al. 2019). In this work, we build on the concept of normalized maximum likelihood (Rissanen) 1996. Shtar' kov 1987). or NML, to learn Bayesian classifiers. Although NML is typically considered from the perspective of optimal coding (Grünwald 2007 Fogel and Feder (018)) we show how it can be used to learn success classifiers, and discuss its connections to exploration and reward shaping in RL.
|
3 PRELIMINARIES
In this paper, we study a modified reinforcement learning problem, where instead of the standard reward function, the agent is provided with successful outcome examples. This reformulation not only provides a modality for task specification that may be more natural for users to provide in some settings (Fu et al. 2018b Zhu et al. 2020 Singh et al.) 2019a), but, as will show, can also make learning easier. We also derive a meta-learned variant of the conditional normalized maximum likelihood (CNML) distribution for representing our reward function, in order to make evaluation tractable. We discuss background on successful outcome examples and CNML in this section.
3.1 REINFORCEMENT LEARNING WITH EXAMPLES OF SUCCESSFUL OUTCOMES
We follow the framework proposed by Fu et al. (2018b) and assume that we are provided with a Markov decision process (MDP) without a reward function, given by M, where M (S,A,T T,Y, Ho), as well as successful outcome examples S_ = {s k 1 which is a set of states in which the desired task has been accomplished. This formalism is casiest to describe in terms of the control as inference framework (Levine 2018). The relevant graphical model in Figure consisss of states and actions, as well as binary success variables e, which represent the occurrence of a particular
2

---

Under review as a conference paper at ICLR 2021
event. The agent's objective is to cause this event occur (e.g., a robot that is cleaning the floor must cause the "floor is clean" event to occur). Formally, we assume that the states in S+ are sampled from the distribution p(syle, True) that is, states where the desired event has taken place. In this work, we focus on efficient methods for solving this reformulation of the RL problem, by utilizing a novel uncertainty quantification method to represent the distribution p(ex(se)-
In practice, prior methods that build on this and similar reformulations of the RL problem (Fu et al. 2018b) derive an algorithm where the reward function in RL is produced by classifier that estimates p ez True 8x). ). Following the adversarial inverse reinforcement learning (AIRL) derivation (Fu al., 2018a Finn et al. 2016). it possible to show that the correct source of negative examples for training this classifier is the state distribution of the policy itself, (s)) This insight results in simple algorithm: at each iteration of the algorithm, the policy is updated to maximize the current reward, given by p(e; True (r)eess), then samples from the policy are added to the set of negative examples S_, and the classifier is retrained on the original positive set S+ and the updated negative set S_.
3.2 CONDITIONAL NORMALIZED MAXIMUM LIKELIHOOD
Our method builds on the principle of conditional normalized maximum likelihood (NML) (Ris- sanen and Roos 2007; Grünwald 2007] Fogel and Feder] 2018), which we review briefly. CNML is method for performing k-way classification, given model class O and dataset D [ITO yo), (41, yI..... (In, }n)}. and has been shown to provide better calibrated predictions and uncertainty estimates with minimax regret guarantees (Bibas et al. 2019). To predict the class of a query point F... CNML constructs & augmented datasets by adding 2g with a different label in each datasets, which write D U (Igg y i). é E (1, 2, ...., k). CNML then defines the class distribution by solving the maximum likelihood estimation problem at query time for each of these augmented datasets convergence, and normalize the likelihoods follows:
PONMI (y= = =1(x,) = pe, (y = ilz0))
0: = arg max ((a.y))-D((,,, Dogpooo (I) 6E
> Pe, (y - jla,)
Intuitively, if xy is close other datapoints in D, then the model will struggle assign high likelihood to labels that differ substantially from other nearby points. However, if x, is far from all datapoints in D, then the different augmented MLE problems can easily classify xg as an arbitrary class, providing us with a likelihood closer to uniform. We refer readers to Grünnalld (2007) for an in-depth discussion. A major limitation of CNML is that it requires training an entire neural network to convergence on the entire augmented dataset every time we want to evaluate test point's class probabilities. We will address this issue Section 5
BAYESIAN SUCCESS CLASSIFIERS FOR REWARD INFERENCE
Ideally, training a classifier with the policy samples as negative examples as described in Section 3.1 should yield a smooth decision boundary be- tween the well-separated negative and positive examples. For example, Fig- ure [ depicts simple 1-D scenario, where the agent starts at the left (sp) and the positive outcomes are the right (8+) side of the environment. Since the positives are on the right and the negatives are on the left, one might expect a classifier to gradually increase its prediction of success as we move to the right (Figure (2a). which would provide dense reward signal for the policy to move to the right. However, this idealized sce- nario rarely happens in practice. With- out suitable regularization, the decision boundary between the positive and negative examples may
Task Eaeolluttoo
Doncd achencc
De Butooorataaasie
Rener inferrce
[x 30
Figure I: BayCRL: Illustration of how Bayesian classifiers learned with normalized maximum likelihood can be used to provide infor- mative learning signal during learning. The human user provides examples of successful outcomes, which are then used in combina- tion with on policy samples to iterate between training classifier with NML and training RL with this reward.
3

---

Under review as a conference paper at ICLR 2021
(a) Ideal rewards
(b) MLE
(c) Tabular CNMI.
(d) CNML. generalization
An idcalized illustration of well-shaped reward and the solutions that various classifier training schemes might provide, Red bars represent visited states near the initial state &y and green pluses represent the example success examples $4. (a) The ideal reward would provide learning signal to encourage the policy to move from the start states to the successful states s... (b) When training success classifier with MLE, the classifier may output zero arbitrary) probabilities when evaluated at states. (c) Tabular CNML will give a prior probability of 0.5 on new states. (d) When using function approximation. CNML with generalization will provide a degree of shaping towards the goal.
Figure 2:
not be smooth. In fact, the decision boundary of an optimal classifier may take on the form of a sharp boundary anywhere between the positive and negative examples in the early stages of training (Figure 2b). As a result, the classifier might provide little to no reward signal for the policy, since can assign arbitrarily small probabilities to the states sampled from the policy. We note that this issue is not pathological: our experiments in Section show that this poor reward signal issue happens in practice and can greatly hinder learning. In this section, we will discuss how an appropriate classifier training method can avoid these uninformative rewards.
4.1 REGULARIZED SUCCESS CLASSIFIESS VIA NORMALIZED MAXIMUM LIKELIHOOD
To create effective shaping, we would like our classifier to provide a more informative reward when evaluated at rarely visited states that lie on the path to successful outcomes. A more in- formative reward function is one that assigns higher rewards to the fringe of the states visited by the policy, be- cause this will encourage the policy to explore and move towards the de- sired states. We can construct such a reward function by imposing the prior that novel states have a non-negligible chance of being a success state. To do so, we train a Bayesian classifier using conditional normalized maximum likelihood (CNML) (Shtar'kor kov 1987). as we de- scribed in Section 3) which corresponds to imposing uniform prior on the output class probabilities.
Algorithm 1 RL with CNML-Based Success Classifiers
1: User provides success examples S.+
2: Initialize policy и, replay buffer S_, and reward classi- fier parameters 0R
3: for iteration i 2,,,... do
4: Add on on-policy examples to S_ by executing т. Sample Res points from S+ (label 1) and New points from S_ (label 0) to construct dataset 'D Assign state rewards as r(s) = PCNMI = 1|8, 7: Train π with RI. algorithm
To use CNML for reward inference, the procedure is similar to the one described in Section We construct a dataset using the provided successful outcomes as positives and the on-policy samples as negatives. However, the label probabilities for RL are then produced by the CNML procedure described in Equation]] [ to obtain the rewards r(s) PCNMI. (e = 1|s). To illustrate how this affects reward assignment during learning, we visualize potential assignment of rewards with a CNML- based classifier the problem described earlier. When the success classifier is trained with CNML instead of standard maximum likelihood, intermediate unseen states would receive non-zero rewards rather than simply having vanishing likelihoods like in Figure 2b The didactic illustrations in Fig 2c and Fig 2d show how the rewards obtained via NML might incentivize exploration. In fact, the CNML likelihood corresponds to form of count-based exploration (as show below), while also providing more directed shaping towards the goal when generalization exists across states.
4.2 RELATIONSHIP TO COUNT-BASHI EXPLORATION
In this section we relate the success likelihoods obtained via CNML to commonly used exploration methods based on counts. Formally, we prove that the success classifier trained with CNML is equivalent to version of count-based exploration with sparse reward function in the absence of any generalization across states (i.e., a fully tabular setting).
4

---

Under review as a conference paper at ICLR 2021
Theorem 4.1. Suppose we are estimating success probabilities p(e 1|s) in the tabular setting,
where we have an independent parameter for each state. Let N(s) denote the number of times state has been visited by the policy, and let G(s) be the number occurrences of state in the set of goal examples. Then the CNML success probability PCNMILe 1[s) is equal to N(S)+ For states
G(a)+1
that are not represented in the goal examples, i.e. G(s) we then recover inverse counts N(+)22'
Refer to Appendix A.7 for a full proof.
4.3 REWARD SHAPING WITH BAYESIAN SUCCESS CLASSIFIERS
While the analysis above suggests that CNML classifier would give us something akin to a sparse reward plus exploration bonus, the structure of the problem and the state space actually provides us more information to guide us towards the goal. most environments (Brockman et al. 2016 Yu et al. 2019) the state space does not consist of independent and uncorrelated categorical variables, and is instead provided in a representation that relates at least roughly to the dynamics structure in the environment. For instance, states close to the goal dynamically in the metric space defined by the states. Indeed, this observation is the basis of many commonly used heuristic reward shaping methods, such as rewards given by Euclidean distance to target states.
also typically close to the goal
In this case, the task specification can actually provide more information than simply performing uninformed count-based exploration. Since the uncertainty-awant classifier described in Section 4. is built on top of features that are correlated with environment dynamics, and is trained with knowledge of the desired outcomes, to CNML without generalization in Fig 2c we expect the intermediate rewards to provide more shaping towards the goal. This phenomenon is illustrated intuitively in Fig 2d] and visualized and demonstrated empirically in our experimental analysis in Section where BayCRL is able to significantly outperform methods for task-agnostic exploration.
able incentivize task-aware directed exploration. As compared
4.4 OVERVIEW
In this section, introduced the idea of Bayesian classifiers trained via CNML means to provide rewards for RL problems specified by examples of successful outcomes. Concretely, a CNML-based scheme has the following advantages:
Natural exploration behavior due to accurate uncertainty estimation in the output suc- cess probabilities. This is explained by the connection between CNML and count-based exploration in the discrete case, and benefits from additional generalization in practical environments,
will see in Section 6
Better reward shaping by utilizing goal examples to guide the agent more quickly and accurately towards the goal. We have established this benefit intuitively, and will validate empirically through extensive visualizations and experiments in Section 6
BAYCRL: TRAINING BAYESIAN SUCCESS CLASSIFIERS FOR OUTCOME DRIVEN RL VIA META-LEARNING AND CNML
In Section 4 we discussed how Bayesian success classifiers can incentivize exploration and provide reward shaping to guide RL. However, the reward inference technique via CNML described in Section I is computationally intractable, as it requires optimizing maximum likelihood estimation problems convergence on every data point we want to query. In this section, we describe novel approximation that allows us to instantiate this method in practice.
5.1 META-LEARNING FOR CNML
We adopt ideas from meta-learning to amortize the cost of obtaining the CNML distribution. As noted in Section 4.1 the computation of the CNML distribution involves repeatedly solving maximum likelihood problems. While computationally daunting, these problems share a significant amount of common structure, which exploit to quickly obtain CNML estimates. One set of techniques that
5

---

Under review as a conference paper at ICLR 2021
directly applicable is meta-learning for few shot classification. Meta-learning uses a distribution of training problems to explicitly learn models that can quickly adapt to new problems.
To apply meta-learning to the CNML problem,
formulate each of the maximum likelihood problems described in Equation T as a separate task for meta-learning, and apply any standard meta-learning technique to obtain a model capable of few-shot adaptation to the MLE problems required for CNML. While any meta-learning algorithm is applicable, we found model agnostic meta-learning (MAMLXFinn
al. (2017)) to be effective choice of algorithm. In short, MAML tries to learn a model that can quickly adapt to new tasks via a few steps of gradient descent.
This procedure is illustrated in Fig 10 and be described follows: given dataset D {( Bo), (I1, v1). (Ix, J. 2n different tasks can be constructed, each corresponding to performing maximum likelihood estimation on the dataset with certain proposed label for rç maxo ((z.))-D((aa y)) log p(y) (z, 0)] or max; ((z.)))u(( [log p(ylar, 0)]. Given these con-
structed tasks (T)), meta-training as described in Finn et al. 2017
max (6),0)))(( 8.1. 0' a a VV
(2)
0
This training procedure gives us parameters 0 that can then be quickly adapted to provide the CNML distribution simply by performing step of gradient descent. The model can be queried for the CNML distribution by starting from 0 and taking one step of gradient descent for the query point augmented dataset, each with a different potential label. These likelihoods are then normalized to provide the CNML distribution follows:
Ps, Po, (y/2²
Pmett-NMLL (2)2:; D)
0- -
a VV (n. (o. (. 0)]. (3)
EvEy po, (y)x)'
This algorithm, which we call meia-NMI, allows us to obtain normalized likelihood estimates without having to retrain maximum likelihood to convergence at every single query point, since the model can now solve maximum likelihood problems of this form very quickly. A complete detailed description and pseudocode of this algorithm are provided in Appendix A.2
Crucially, we find that meta- NML is able to approximate the true NML outputs with just one or few gradient steps. This makes it several orders of magnitude faster than naive CNML, which would normally require multiple passes through the entire dataset on each input point in order to train to convergence.
Feedforward
Meta-NML
Naive CNML
Single input point Epoch of RL
0.0004s
0.0090s
15.19s
23.50s
39.05s
4hr 13min 34s
Table 1: Runtimes for evaluating a single input point and rumning one epoch of RL using feedforward, meta-NML, and naive CNML classifiers. Meta-NML provides over a 1600x speedup compared naive CNML. For full list of runtimes see Appendix A.3.3
5.2 APPLYING META-NML TO SUCCESS CLASSIFICATION
We apply the meta-NML algorithm described above to learning Bayesian
Algorithm 2 BayCRL: Bayesian Classifiers for RL.
success classifiers for providing re-
1: User provides success examples S4
wards for reinforcement learning, in our proposed algorithm, which we term BayCRL- Bayesian classifiers for reinforcement learning. Similarly to Fu al.] (2018h), we can train our Bayesian classifier by first con- structing a dataset for binary clas- sification. This is done by using the provided examples of successful out- comes as positives, and on- on-policy amples collected by the policy as neg- atives, balaneing the number of sam-
2: Initialize policy æ, replay buffer S_, and reward classi- fier parameters 0g
3: for iteration i == 2,.. do
4: Collect on-policy examples to add S_. by executing n.
if iteration i mod k 0 then
6:
Sample Perain states from S_ to create 2naain meta- training tasks
7:
Sample Res total test points equally from S+ (label 1) and S_ (label 0)
8:
Meta-train OR via meta-NML using Equation] 2
9:
Assign state rewards via Equation
10: Train π with RL algorithm

---

Under review as a conference paper at ICLR 2021
pled positives and negatives in the dataset. Given this dataset, the Bayesian classifier parameters ORe can be trained via meta-NML as described in Equation[] The classifier can then be used to directly
and quickly assign rewards to state according to its probabilities r(s) Pmeta NMI. (e 1|s) (via
step of gradient descent, as described in Equation 4 and perform standard reinforcement learning.
po, (e 1[s)
Pmett-NMi (c 1/s)
(4)
Σ (ie(0.1) po. = i|s)
0, = 0R - a ( ( 83, 0)], for E {0,1]
(5)
An overview of this algorithm is provided in Algorithm 2 and full details are in Appendix A.2 The rewards start off at an uninformative value of 0.5 for all unvisited states at the beginning. and close to 1 for successful outcomes. As training progresses, more states are visited, added to the buffer and BayCRL starts to assign them progressively lower reward as they get visited more and more, thereby encouraging visiting of under-visited states. At convergence, all the non successful states will have a reward of close to 0 and states at the goal will have a reward of 0,5, since the numbers of positive and negative labels for successful outcomes will be balanced as described above.
EXPERIMENTAL EVALUATION
In our experimental evaluation we aim to answer the following questions: (I) Do the learning dynamics of prior classifier- based reward learning meth- ods provide informative rewards for RL? (2) Does using BayCRL help ad- dress the exploration challenge when solving RL problems specified by successful outcomes? (3) Does us- ing BayCRL help provide better re- ward shaping than simply performing naïvely uninformed exploration? To evaluate these questions, we evaluate our proposed algorithm BayCRL with the following setup. Fur- ther details and videos can be found at at/tttss://sites http::///iiess..goee.e. wwwpwwww..w///ttt.t...o....w.w.t baycr// //home]
Tambbas
Mala Modal
Figure 3: Diagram of using meta-NML to train classifier. Meta- NML learns initialization that can quickly adapt to new data- points with arbitrary labels. At evaluation time, approximates the NML probabilitics (right) fairly well with a single gradient step
6.1 EXPERIMENTAL SETUP
We start off by understanding the algorithm behavior by evalu- ating it on maze navigation problems, which require avoiding several local optima before truly reaching the goal. Then, to evaluate our method in more complex domains, we consider three robotic manipulation tasks that were previously covered in (Singh et al. (2119) with a Sawyer robot arm: door opening, tabletop object pushing, and 3D object picking. As we show in our re- sults, exploration in these environments is challenging and using naively chosen reward shaping often does not solve the problem at hand. More details on each environment and their associated challenges are available in Appendix AA.41
(a)
Zigzag (b)
Spiral (c)
Sawyer
Maze
Maze
Push
(d) Pick
Sawyer (c)
Sawyer
Door
We compare with a number of prior algorithms and ablations. To provide comparison with a standard previous method which uses success classifiers trained with an IRL-based adversarial method, we include the VICE algorithm (Fu et al. 2018b). Note that this algorithm is quite related to BayCRL, but it uses standard maximum likelihood classifier rather than a Bayesian classifier trained with CNML and meta-leamnin. We also include com- parison with DDL, a recently proposed technique for learning dynamical distances (Hartikainen et al., 2019). We additionally
Figure 4: We evaluate two maze environments and three robotics tasks: (a) the agent must navigate around an S-shaped corridor, (b) the agent must navigate spiral corridor, (c) the robot must push a puck lo- cation, (d) robot must raise a ran- domly placed tennis ball to location, (c) the robot must open the door a specified angle.
7

---

Under review as a conference paper at ICLR 2021
include comparisons to algorithms for uninformed exploration show that BayCRL does more directed form of exploration and reward shaping. To provide an apples-to-apple comparison, we use the same VICE method for training classifiers, but combine it with govelty-basee exploration based on random network distillation (Burda et al. 2018h) for the robotic manipulation tasks, and oracle inverse count bonuses for the maze navigation tasks. Finally, to demonstrate the importance of well-shaped rewards, we compare to running Soft Actor-Critic (Haarnoja et al. (2018), standard RL algorithm for continuous domains, with naive reward functions: sparse reward at the goal, and a heuristically shaped reward which uses L2 distance to the goal state. More details on each algorithm and the hyperparameters used are included in Appendix A.6
6.2 COMPARISONS WITH PRIOR ALGORIHTM
We compare with prior algorithms on the domains described above. As we can see in Fig BayCRL is able to very quickly learn how to solve these challenging exploration tasks, often reaching better asymptotic performance than most prior methods, and doing so more efficiently than VICE (Fu et 2018b) or DDL (Hartikainen et al. 2019). This suggests that BayCRI. is able to provide directed reward shaping and exploration that is substantially better than standard classifier baase methods (e.g., VICE).
Deeaa More
Sensa Mort
BayCRI (DUPS)
VIZE (Pu ot al, 2018)
VICE COunT bonus
VICE (Bunds 2018)
Mlrtticanen 2022)
Sparse reward
distancc
20
Senver Door Coette
Figure 5: BayCRI. outperforms prior goal-reaching methods on all our evaluation environments. BayCRI. also performs better or comparably to heuristically shaped hand-designed reward that uses Fuclidean distance, demonstrating that designing well-shaped reward is not trivial in these domains. Shading indicates standard deviation across seeds. For details the success metrics used, see Appendix
A.442 A.4.2
To isolate whether the benefits purely come from exploration or also from task-aware reward shaping, compare with methods that only perform uninformed, task-agnostic exploration. On the maze environments, where compute ground truth count-based bonuses for exploration. For the higher dimensional robotics tasks, we RND (Burda et al.
Oblec ty ty anal
discretize the state space, we
0.15
20185). From these comparisons, shown in Fig 5 it is clear thar BayCRL significantly outperforms methods that use novelty-seeking exploration, but do not otherwise provide effective reward shaping. In combination with our visualizations in Section 6.4, this suggests that BayCRL. is providing useful task-aware reward shaping more effectively than uniformed exploration methods. We also compare BayCRL to manually heuristically-designn tion, based on Euclidean distance. As shown in Fig 5] BayCRL generally outperforms simple manual shaping in terms of sample complexity and asymptotic performance, indicating that the learned shaping is non-trivial and adapted to the task.
Winsic goll eampas
100
203
300
400
530
600
Figure 6: Ablative analysis of BayCRI. The amortization from meta-learning and access goal examples are hoth important com- ponents for performance.
shaped reward func-
8

---

Under review as a conference paper at ICLR 2021
6.3 ABLATIONS
We first evaluate the importance of meta-learning for estimating the NML distribution. In Figure 6 we see that naively estimating the NML distribution by taking single gradient step and following the same process as evaluating meta-NMI, but without any meta-training, results in much worse performance. Second, we analyze the importance of making the BayCRL classifier aware of the task being solved, to understand whether BayCRL is informed by the success examples or simply approximates count- based exploration. To that end, we modify the training procedure so that the dataset DD consists of only the on-policy negatives, and add the inferred reward from the Bayesian classifier to the reward obtained by standard MLE classifier (similarly to the VICE+RND baseline). We see that this performs poorly, showing that the BayCRL classifier is doing more than just performing count- exploration, and benefits from better reward shaping due to the provided goal examples. Further ablations are available in Appendix A.55
6.4 ANALYSIS OF BAYCRL
BayCRL and Reward Shaping. To better understand how BayCRL provides reward shaping, we visualize the rewards for various slices along the axis on the Sawyer Pick task, environment which presents a significant exploration challenge. In Fig 7 we see that the BayCRL rewards clearly correlate with the distance to the object's goal position, shown as white star, thus guiding the robot to raise the ball to the desired location even if it has never reached the goal before. In contrast, the MLE classifier has sharp, poorly-shaped decision boundary.
BayCRL and Exploration. Next, to illustrate the connec- tion between BayCRL and exploration, we compare the states visited by BayCRL (which uses a meta-NML classi- fier) and by VICE (which uses standard L2-regularized classifier) in Figure [] We see that BayCRL naturally in- centivizes the agent to visit novel states, allowing to navigate around local minima and reach the true goal. In contrast, VICE learns misleading reward function that prioritizes closeness to the goal in zy space, causing the agent to stay on the wrong side of the wall.
BeyCHt
NLE
rewards
Figure 7: Visualization of reward shaping for 3D Pick-and-Place at various z values (heights). BayCRL learns rewards that pro- vide smooth slope toward the goal, adapting to the policy and guiding it to learn the task, while the MLE classifier learns sharp and poorly shaped decision boundary.
Interestingly, despite incentivizing exploration, BayCRL does not simply visit all possible states; at convergence, it has only covered around 70% of the state space. In fact, we see in the scatterplots in Figure g that BayCRI. prioritizes states that bring it closer to the goal and ignores ones that don't, 't thus making of the goal examples provided to it. This suggests that BayCRL benefits from combination of novelty-seeking behavior and effective reward shaping, allowing it to choose new states strategically.
7 DISCUSSION
In this work, we consider a subclass of reinforcement learning problems where examples of successful out- comes specify the task. We analyze how solutions via standard success classifiers suffer from shortcomings, and training Bayesian classifiers allows for better ex- ploration to solve challenging problems. We discuss how the NMI. distribution can provide us a way to train such Bayesian classifiers, providing benefits exploration and reward shaping. To make learning tractable, we propose a novel meta-learning approach to amortize the NML process.
Figure Plots of visitations and state coverage over time for BayCRL vs. VICE. BayCRL ex- plores a significantly larger portion of the state space and is able to avoid local optima.
While this work has shown the effectiveness of Bayesian classifiers for reward inference for tasks in simulation, it would be interesting scale this solution to real world problems. Additionally, obtaining a theoretical understanding of how reward shaping interacts with learning dynamics would be illuminating in designing reward schemes.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-firt international conference on Machine learning, page 1, 2004.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances neural information processing systems, 2017.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20 2020.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 2002.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pages 1471-1479, 2016.
Koby Bibas, Yaniv Fogel, and Meir Feder. Deep pnml: Predictive normalized maximum likelihood for deep neural networks. CoRR, abs/1904.12286, 2019. URL htttp://axxv.cr www. /abs/ 1904. 12286
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. International Conference on Machine Leurning, pages 1613-1622 2015.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, 0ass/1606.011440 2016. URL htlhht:p//arx.v.or
abbs/6066 01540
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018a.
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. CoRR, abs/1810. 1894,, 2018b. URL http:://arrii...rrgaaa///BB
12894
Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre-Yvel Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pages 1331-1340, 2019.
C. Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017.
Chelsea Finn, Paul F. Christiano, Pieter Abbeel, and Sergey Levine A connection between gen- crative adversarial networks, inverse reinforcement learning, and energy- models. CoRR, 0ts/1611.038522 2016. URL hRtp:::///axx wtwpa/iww...o..aaaaoaaa abs 1 1611.1111111111 03852
Yaniv Fogel and Meir Feder. Universal batch learning with los-loss. In 2018 IEEE International Symposium on Information Theory (ISIT), pages 21 21-25. IEEE, 2018
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforce- ment learning. In International Conference on Learning Representations, 2018a.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in Neural Information Processing Systems, pages 8547 8556, 2018b.
Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach. and Sergey Levine. Learning to reach goals without reinforcement lcarning. arXiv preprint 1911::/912.00088, 2019.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1 406.2661,
2014. CtR.ttttp:///// www.oogg/////o.ooo
org /abs/ 1406. 2666
10

---

Under review as a conference paper at ICLR 2021
Peter D Grünwald. The minimum description length principle. MIT press, 2007.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, Abs/1812.00505, 2018. URL
orq/ ottp/ba///aaaa
05905
Kristian Hartikainen, Xinyang Geng, Tuomas Haarojja, and Sergey Levine. Dynamical distance learning for unsupervised and semi-supervised skill discovery. CoRR, abs/1907.08225, 2019. URL
http::/arxiv.et www.aaawwwww..aaa/aaaaaaaaaaaaaa hwtprrggab///111/// 08225
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pages 4565-4573, 2016.
Matthew Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research, (4(1):1303-147) 2013.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1109-1117, 2016.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, 1993.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9-16, 2009.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. CoRR, 0aa//805.00900 2018. URL. htw..htp:///
00909
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. J. Mach. Learn. Res., 17:39:1-39:40, 2016. URL http:t/ttmmm......
ppapersvv1/15-522 52tmmmmmm
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, pages 13153-13164, 2019.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pages 9191-9200, 2018
Andrew Y. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Ivan Bratko and Saso Dzeroski, editors, Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999). Bled, Slovenia, June 27 30, 1999, pages 278-287. Morgan Kaufmann, 1999.
Brendan O' O'Donoguue. Variational bayesian reinforcement learning with regret bounds. arXiv preprint arXi::1807.00647 2018
OpenAl, :,
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning, 2019.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, pages 3003-3011, 2013.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, 2016.

---

Under review as a conference paper at ICLR 2021
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-drive exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition Workshops, pages 16-17, 2017.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018.
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Jürgen Schmidhuber. Hindsight policy gradients. In International Conference Learning Representations, 2018.
Jorma Rissanen and Teemu Roos. Conditional NML universal models. In 2007 Information Theory and Applications Workshop, pages 3
Jorma J Rissanen. Fisher information and stochastic complexity. IEEE transactions on information theory, 42(1))40-47, 1996.
Schaul, Y. Sun, Wierstra, F. Gomez, and J. Schmidhuber. Curiosity- optimization. In 2011 IEEE Congress of Evolutionary Compulation (CEC), 2011.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320, 2015.
Yurii Mikhailovich Shtar' kov. Universal sequential coding of single messages. Problemy Peredachi Informatsii, 23(3):3-17, 1987.
Avi Singh, Larry Yang. Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering, In Antonio Bicchi, Hadas Kress-Gazit, and Seth Hutchinson, editors, Science and Systems XV, University of Freiburg, Freiburg Breisgau, Germany, June htl.htts::/// doi orrg100.... 156677
Robotics:
22-26, 2019, 2019a. doi: 10.15607/RSS.22199X0000
RSS 2019 073
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. In Robotics: Science and Systems (RSS), 2019b.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814. 2015.
Malcolm Strens. A bayesian framework for reinforcement learning. In JCML, 2000.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pages 2753-2762, 2017.
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning. arXiv preprint /806.08000009600 2018.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-maramttric discriminative rewards. In International Conference on Learning Representations, 2018.
Marco Wiering and Jürgen Schmidhuber. Efficient model-based exploration. In Proceedings of the Sixth International Conference on Simulation of Adaptive Behavior: From Animals to Animats.
MIT Press Cambridge, MA, 1998.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2019.
12

---

Under review as a conference paper at ICLR 2021
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real world robotic reinforcement learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL FJe2syrtvS
eww.eeiiwwwww.... ew. net. forum? d=
Y. Zhu, R. Mottaghi, E. Kolve, J. Lim, A. Gupta, Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL. USA. 2008.
13

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.I
GRAPHICAL MODEL FOR CONTROL AS INFERENCE
SO
S1
S2
ST
a0
a1
a2
aT
eo
e1
e
ET
Figure 9: Graphical Model framework for Control as Inference. 6, correspond to auxiliary event variables
representing successfully accomplishing the task
A.2 DETAILED DESCRIPTION OF META-NML
We provide detailed description of the meta-NML algorithm described Section 5 and the details of the practical algorithm.
Taak
T
qqurry
Daliiot
U Dau -
Cenwet
(Raa m
Figure 10: Figure illustrating the meta-training procedure for meta-NML.
Given dataset D {F0, Mo), (I1, y1) un),.., (Xn, yn)), the meta-NML procedure proceeds by first constructing * n tasks from these data points, for a l shot classification problem. We will keep k for simplicity in this description, in RL. Each task т; is constructed by augmenting the dataset with a negative label D U (x,, 0) positive label 'D (I,,, y 1). Now that each task consists of solving the maximum likelihood problem for its augmented dataset, we can directly apply standard meta-learning algorithms to this setting. Building off the ideas in MAML (Finn et al. (017)). we can then train set of model parameters 0 such that after single step of gradient descent it can quickly adapt to the optimal solution for the MLE problem on any of the augmented datasets. This is more formally written as
accordance with the setup of binary success classifiers
max Er-SCr
8.1. 0' -0 -Vo((,
(6)
where L represents standard classification loss function, is the learning rate, and the distribution of tasks p(T) is constructed as described above. For new query point æ, these initial parameters can then quickly be adapted to provide the CNML. distribution by taking gradient step each
14

---

Under review as a conference paper at ICLR 2021
augmented dataset to obtain the approximately optimal MLE solution, and normalizing these as follows:
po, (2)2z)
Nm)te-NML(y/2;2 EvEy po, (y]z))
By ( Vo. 0)]
This algorithm in principle be optimized using any standard stochastic optimization method such as SGD, as described in Fiin et al. (2017). backpropagating through the inner loop gradient update. For the specific problem setting that consider, have employ some optimization tricks in order to enable learning:
A.2.1 IMPORTANCE WEIGHTING ON QUERY POINT
Since only one datapoint augmented to the training set at query time for CNML, can get challenging for stochastic gradient descent pay attention this datapoint with increasing dataset sizes. For example, we train on an augmented dataset of size 2048 by cycling through it in batch sizes of 32, then only in 64 batches would include the query point itself and allow the model adapt to the proposed label, while the others would lead to noise in the optimization process, potentially worsening the model's prediction the query point.
In order to make sure the optimization considers the query point, we include the query point and proposed label (zg..)) y) in every minibatch that sampled, but downweight the loss computed on that point such that the overall objective remains unbiased, This is simply doing importance weighting, with the query point downweighted by a factor of [ where b the desired batch size and N the total number of points in the original dataset.
To see why the optimization objective remains the same, we can consider the overall loss over the dataset. Let fe be our classifier, L be our loss function, D' = {(z,, W U (Zgg y) be our augmented dataset, and By the kth batch seen during training. Using standard SGD training that cycles through batches in the dataset, the overall loss on the augmented dataset would be:
N Σ c(folz)) vi) + C(fe(z.).))
C(D')
we instead included the downweighted query point in every batch, the overall loss would be:
((D')
Σ
Σ
((fe(xs))
C(fe(xa).))
b-1
(BB
N
((fo(z)).81)
((fo(xg).
[b_1]
(EB,
N
((fo(x)) + ((fo(a)))))
which is the same objective as before.
This trick has the effect still optimizing the same max likelihood problem required by CNML, but significantly reducing the variance of the query point predictions at query time. As a concrete example, consider querying meta-CNML classifier on the input shown in Figure [I] If we adapt to the augmented dataset without including the query point in every batch (i.e. without importance weighting), we see that the query point loss is significantly more unstable, requiring to take more gradient steps converge.
take additional gradient steps
15

---

Under review as a conference paper at ICLR 2021
Dataset
Batch losses
Query point losses
with importance weightigg
with importance weighting
13
1.0
[yrd]
Negether
Rocitin
[ym3]
Cierry ooter
(orppoued yral
Number
664) 2010
10000
Number
2003 830
660
800
1000
Figure 11: Comparison of adapting to a query point (pictured on left with the original dataset) at test time for CNML with and without importance weighting. The version without importance weighting is more unstable both in terms of overall batch loss and the individual query point loss, and thus takes longer converge. The spikes the red lines occur when that particular batch happens include the query point, since that point's proposed label (ty 1) different than those nearby points (y = 0). The version with importance weighting does not suffer from this problem because it accounts for the query point in each gradient step, while keeping the optimization objective the same.
A.2.2 KERNEL WEIGHTED TRAINING LOss
The augmented dataset consists of points from the original dataset D and one augmented point (I,,, y). Given that we mostly care about having the proper likelihood on the query point, with an imperfect optimization process, the meta-trainie meta-training can yield solutions that not very accurately representing true likelihoods on the query point. To counter this, we introduce a kernel weighting into the loss function in Equation duuuing meta-training and subsequently meta-testing. The kernel weighting modifies the training loss function as:
max
s.t 8' = - ((o, (7)
where x, is the query point for task 7 and choice kernel. We typically choose exponential kernels centered around Tr. Intuitively, this allows the meta-optimization to mainly consider the datapoints that are copies the query point in dataset, or are similar to the query point, and ensures that they have the correct likelihoods, instead of receiving interfering gradient signals from the many other points in the dataset. To make hyperparameter selection intuitive, we designate the strength of the exponential kernel by parameter Adist, which is the Euclidean distance away from the query point at which the weight becomes 0. Formally, the weight of point x in the loss function for query point 2, is computed as:
2.3 Adist
K(x, xx) exp {-
| x71123
(8)
A.2.3 META-TRAINING AT FIXED INTERVALS
While in principle meta-NML would retrain with every new datapoint, in practice we retrain meta- NML once every epochs. (In all our experiments we set = 1, but we could optionally increase & if we do not expect the meta-task distribution to change much between epochs.) We warm-start the meta-learner parameters from the previous iteration of meta-learning so every instance of meta- training only requires few steps. We find that this periodic training is reasonable enough approximation, as evidenced by the strong performance BayCRL in our experimental results in Section 6
16

---

Under review as a conference paper at ICLR 2021
A.3 META-NML VISUALIZATIONS
A.3.1 META-NML WITH ADDITIONAL GRADIENT STEPS
Below, we show a more detailed visualization of meta-NML outputs on data from the Zigzag Maze task, and how these outputs change with additional gradient steps. For comparison, we also include the idealized NML rewards, which come from discrete count-based classifier.
Meta-NML. is able to resemble the ideal NML rewards fairly well with just gradient step, providing both an approximation of a count-based exploration bonus and better shaping towards the goal due to generalization. By taking additional gradient steps, meta-NML can get arbitrarily close to the true NML outputs, which themselves correspond to inverse counts of 242 explained in Theorem 4.1. While this would give us more accurate NML estimates, in practice we found that taking one gradient step was sufficient to achieve good performance on our RL tasks.
Training set
Idealized NML rewards
Visited examples
08
06
stop)
(5 steps)
Meta-NML (10 steps)
Meta-NML (50 steps)
Figure Comparison of idealized (discrete) NML and meta-NML rewards on data the Zigzag Maze Task. Meta-NML approximates NML reasonably well with just one gradient step at test time, and converges to the true values with additional steps.
Average absolute difference between MLE and meta-CNML
0.30
1-step
0.25
rewards
0.20
0.15
0.00
10
20
40
50
Number finetuning steps
Figure 13: Average absolute difference hetween MI.E and meta-NML. goal prohahilities state space from F We see that meta-NML learns model initialization whose parameters can change significantly in a small number of gradient steps. Additionally, most of this change comes from the first gradient step (indicated by the green arrow), which justifies our choice to use only single gradient step when evaluating meta-NML probabilities for BayCRL.
the entire
17

---

Under review as a conference paper at ICLR 2021
A.3.2 COMPARISON OF REWARD CLASSIFIERS
Training Dataset
MLE Classifier
MLE Classifier p...ng eeely stopping
Ideoliied NML Rewards Awweee counts
Metz-NML Rewards /isioll
(rokenn coneegency
Visted coem examples
Figure 14: comparison the rewards given by various classifier training schemes on the Zigzag maze. From left to right: (1) An MLE classifier when trained to convergence reduces to an uninformative sparse reward: (2) An MLE classifier trained with regularization and early stopping has smoother contours, but does not accurately identify the goal: (3) The idealized NMI. rewards correspond to inverse counts, thus providing natural exploration objective in the absence of generalization; (4) The meta-NMI. rewards approximate the idealized rewards well in visited regions, while also henefitting from hetter shaping towards the goal due to generalization.
A.3.3 RUNTIME COMPARISONS
Below provide the runtimes for feedforward inference, naive CNML, and meta-NML on each of our evaluation domains. We list both the runtimes for evaluating single input, and for completing full epoch of training during RL.
These benchmarks were performed on an NVIDIA Titan X Pascal GPU. Per-input runtimes are averaged across 100 samples, and per-epoch runtimes are averaged across 20 epochs.
Feedforward Inference
Meta-NML
Naive CNML
Mazes (zigzag, spiral)
0.0004s
0.0090s
15.19s
Sawyer 2D Pusher
0.0004s
0.0092s
20.64s
Sawyer Door
0.0004s
0.0094s
20.68s
Sawyer 3D Pick
0.0005s
0.0089s
20.68s
Table 2: Runtimes for evaluating a single imput point using foedforward, meta-NML, and naive CNML classifiers.
Meta-NML provides any unwhhere between a 1600x and 2300x speedup compared to naive CNML, which is crucial to making our NML-based reward classifier scheme feasible on RL problems.
Feedforward Inference
Meta-NML
Naive CNML
Mazes (zigzag, spiral)
23.50s
39.05s
4hr 13min 34s
Sawyer 2D Pusher
24.91s
43.81
Shr 44min 25s
Sawyer Door
19.77s
38.52s
5hr 45min 0Os
Sawyer 3D Pick
20.24s
40.73s
5hr 45min 00s
Table 3: Runtimes for completing single epoch of RL according to Algorithm 2 We collect 1000 samples in the
environment with the current policy for each epoch of training. The naive CNML runtimes are extrapolated based the per-input runtime in the previous table. These times indicate that naive CNML would be computationally infeasible run in an RL algorithm, whereas meta-NML. able achieve performance much closer to that of an ordinary feedforward classifier and make learning possible.
A.4 EXPERIMENTAL DETAILS
A.4.1 ENVIRONMENTS
Zigzag Maze and Spiral Maze: These navigation tasks require moving through long corridors and avoiding several local optima in order to reach the goal. For example, on Spiral Maze, the agent must get stuck on the other side of the inner wall, even though that position would be close in I.2
18

---

Under review as a conference paper at ICLR 2021
distance to the desired goal. On these tasks, sparse reward is not informative enough for learning, while ordinary classifier methods get stuck local optima due to poor shaping near the goal.
Both of these environments have a continuous state space consisting of the (zz, y) coordinates of the agent, ranging from (-4, -4) to (4, 4) inclusive. The action space the desired velocity in the æ and y directions, each ranging from -1 to inclusive.
Sawyer 2D Pusher:
This task involves using a Sawyer arm, constrained to move only in the zy plane, to push a randomly initialized puck to fixed location table. The state space consists of the (I, y, =) coordinates of the robot end effector and the (a, y) coordinates of the puck. The action space the desired x: and y velocities of the arm.
Sawyer Door Opening: In this task, the Sawyer arm is attached to a hook, which must use to open door desired angle of 45 degrees. The door is randomly initialized each time be starting angle of between and 15 degrees. The state space consists of the (x, y, z) coordinates of the end effector and the door angle (in radians); the action space consists of (z, y. velocities.
Sawyer 3D Pick and Place:
The Sawyer robot must pick a ball, which is randomly placed somewhere on the table each time, and raise a fixed (z. y, location high above the table. This represents the biggest exploration challenge out of all the manipulation tasks, as the state space is large and the agent would normally not receive any learning signal unless it happened to pick up the ball and raise it, which is unlikely without careful reward shaping.
The state space consists of the (x, y/ =) coordinates the end effector, the (æ, y, z) coordinates of the ball, and the tightness of the gripper (a continuous value between 0 and 1). The robot can control its (I, y, 2) arm velocity as well as the gripper value.
A.4.2 GROUND TRUTH DISTANCE METRICS
In addition to the success rate plots in Figure 5 we provide plots of each algorithm's 's distance to the goal over time according to environment-specifne distance metries. The distance metries and success thresholds, which were used to compute the success rates in Figure 5 are listed in the table below.
Environment
Distance Metric Used
Success Threshold
Zigzag Maze
Manhattan distance to goal
0.5
Spiral Maze
Manhattan distance to goal
0.5
Sawyer 2D Pusher
Puck L2 distance to goal
0.05
Sawyer Door Opening
Angle difference to goal (radians)
0.035
Sawyer 3D Pick-and-PPace
Ball L2 distance to goal
0.06
Zanc Mart
Sptoe Mect
EayCRL (ours)
20
VICE [Fu 2018)
VICE bonus
Otartikkiienn wi., 2020)
Marhattan
Sparse reward
L2 distance reward
100
200
300
400
300
650
700
1.00
200
370
Semper 20
Semper Podaarshhhee
Seayer Dpents
0.30
3.20
2.0
0:35
0.6
2.00
3.05
9.00
Spochs
Spocht
Figure 15: Performance BayCRL compared to other algorithms according to ground truth distance metries. We note that while other algorithms seem to be making progress according to these distances, they are often actually getting stuck in local minima, as indicated by the success rates in Figure 5 and the visitation plots in Figure
19

---

Under review as a conference paper at ICLR 2021
A.5
ADDITIONAL ABLATIONS
A.5.1 LEARNING IN A DISCRETE, RANDOMIZED ENVIRONMENT
In practice, many continuous RL environments such as the state spaces that are correlated at least roughly with the dynamics. For instance, states that are closer together dynamically
consider in Section have
also typically closer in the metric space defined by the states, This correlation does not need to be perfect, but as long as it exists, BayCRL can in principle learn smoothly shaped reward towards the goal.
However, even in the case where states are unstructured and completely lack identity, such as in discrete gridworld environment, the CNML classifier would still reduce to providing an exploration- centric reward bonus, as indicated by Theorem 4.1 ensuring reasonable worst-case performance.
To demonstrate this, we evaluate BayCRL on variant of the Zigzag Maze task where states are first discretized to 16 x 16 grid, then "shuffled" so that the zy representation of a state does not correspond to its true coordinates and the states are not correlated dynamically, BayCRL manages to solve the task, while standard classifier method (VICE) does not. Still, BayCRL is more effective in the original state space where generalization is possible, suggesting that both the exploration and reward shaping abilities of the CNML classifier crucial to its overall performance.
Zigzag Maze (Shuffied States)
BayCRL (ours) VICE (Fu et al., 2018)
Manhattan distance to goal
15
Sparse Reward (Haarmoja al., 2019)
10
100
300
400
Epochs
Figure 16: Comparison of BayCRL, VICE, and SAC with sparse rewards discrete, randomized variant Zigzag Maze task. BayCRL is still to solve the task on a majority runs to its connection to a count-based exploration bonus, whereas ordinary classifier methods (i.e. VICE) experience significantly degraded performance in the absence of any generalization across states.
A.5.2 FINDING "HIDDEN" REWARDS NOT INDICATED BY SUCCESS EXAMPLES
The intended setup for BayCRL (and classifier based RL algorithms in general) is provide set of success examples to learn from, thus removing the need for a manually specified reward function. However, here instead consider the case where ground truth re- ward function exists which we do not fully know, and can only query through interaction with the environment. In this case, because the human expert has limited knowledge, the provided success examples may not cover all regions of the state space with high reward.
An additional advantage of BayCRL is that it is still capable of finding these "unspecifiedd" goals because of built-in exploration behavior, whereas other classifier methods would operate solely based on the goal examples provided. To see this, we evaluate our algorithm two-sided variant of the Zigzag Maze with multiple goals, visualized in Figure 17 to the right. The agent starts the the far left side of the maze; unknown to it, the right side contains 5 sparse reward regions which actually closer from its initial position.
Ow corde
Rerpeettie
Semsrevet
Figure 17: Visualization Double-Sided Maze environment. Only the goal examples in the bot- tom left corner provided to the algorithm.
middle and is provided with 5 goal examples
20

---

Under review as a conference paper at ICLR 2021
As shown in Figures 18 and 19 BayCRL manages find the sparse rewards while other methods do
not. BayCRL, although initially guided towards the provided goal examples on the left, continues to explore in both directions and eventually finds the "hidden" rewards the right. Meanwhile, VICE focuses solely on the provided goals, and gets stuck in a local optima near the bottom left corner.
Double-Sided Maze with Sparse Rewards
Manhattan distance sperse goal
VICE
20
Sperse Reward SAC (Haammoja
15
examples
10
Epochs
Figure 18: Performance of BayCRL, VICE, and SAC with sparse rewards on double-sided maze where some sparse reward states receiving higher overall reward, whereas ordinary classifier methods (i.c. VICE) move only towards the provided examples and thus are never able for comparison, is generally unable to find the goals. The dashed gray line represents the location of the goal examples initially provided to both BayCRL and VICE.
provided as goal examples. BayCRL is still able to find the sparse rewards, thus
the additional rewards. Standard SAC with sparse rewards, also included
Epoch
Epoch
Epoch 130
Epoch 160
BayCRL (ours)
Epoch 0
Doech
Cooch 130
Epoch 100
VICE
Figure 19: Plot of visitations for BayCRL vs. VICE the double-sided maze task. BayCRL is initially guided towards the provided goals in the bottom left allowing it instead because those rewards are easier to reach. In contrast, VICE moves only towards the (incomplete) set of provided goals on the left, ignoring the right half of the maze entirely and quickly getting stuck in local optima.
as expected, but continues explore both directions, thus hidden sparse rewards as well. Once this happens, it focuses on the right side the maze
21

---

Under review as a conference paper at ICLR 2021
A.6 HYPERPARAMETER AND IMPLEMENTATION DETAILS
We describe the hyperparameter choices and implementation details for our experiments here. We first list the general hyperparameters that were shared across runs, then provide tables of additional hyperparameters we tuned over for each domain and algorithm.
SAC
Learning Rate
3 x 10 0.99 Gaussian (512,512)
Discount Factor 7
Policy Type
Policy Hidden Sizes
Policy Hidden Activa-
ReLUU
tion
RL Batch Size
1024
Reward Scaling
Replay Buffer Size
500, 000
Q Hidden Sizes
512)
Q Hidden Activation
ReLU
Q Weight Decay
Learning Rate
3 x 10
Target Network
5 x 10
BayCRL
Adaptation batch size
64
Meta-training tasks
128
per epoch
Meta-test set size
2048
VICE
Classifier Learning
1 10
Rate
Classifier Batch Size
128
Classifier Optimizer
Adam
RI Algorithm
SAC
RND
Hidden Layer Sizes
(256, (56),256)
Output Units
512
Table General hyperparameters used across all domains.
Goal Examples: For the classifier- methods our experiments (VICE and BayCRL), we provide 150 goal examples for each environment at the start of training. These are used the pool of positive examples when training the success classifier.
We use the version of DDL proposed in Hartikainen et al. (2019) where provide the algorithm with the ground truth goal state g. then run SAC with a reward function of r(s) = -d" (s, g), where d= is the learned dynamical distance function for the policy the current iteration of training.
DDL Reward:
22

---

Under review as a conference paper at ICLR 2021
A.6.1
ZIGZAG MAZE HYPERPARAMETERS
BayCRL
Classifier
Hidden
[(112, 512), (2048, 2048)]
Layers
Adist
0.5, !] 1
Kquery
VICE
7IVICE
1, 2, 10 0, 1
Mixup a
Weight Decay À VICE+Count
0, 5 10-3
Bonus
7IVCCE
2, 10]
Mixup a
0. 1
Classifier
reward
[0.25, 0.5, 1]
scale
Weight Decay
0,5 10 10-3)
DDL
Na
2,
Training frequency
[16, 64]
(every n steps)
Table 5: Hyperparameters we tuned for the Zigzag Maze task. Bolded values are what we use for the final runs Section 6.
A.6.2 SPIRAL MAZE HYPERPARAMETERS
BayCRL Classifier
Hidden
[(512, 512). (2048, 2048)]
Layers
Adist
0.55, 1] 1
Kquery
VICE
7IVCCE Mixup O
1, 2, 10 0,
Weight Decay A VoCE+Count
0, x 10
Bonus
TVICE
1, 2, 10] 00.25, 0.5, 1]
Mixup a
Classifier
reward
scale
Weight Decay A
0,5 10 10-31
DDL
Na
4 16, 64)
Training frequency (every n steps)
Table Hyperparameters tuned Spiral Maze task. Bolded values are we use the final runs
Section 6.
23

---

Under review as a conference paper at ICLR 2021
A.6.3 SAWYER PUSH HYPERPARAMETERS
BayCRL
Classifier
Hidden
[(112, 512), (2048, 2048)]
Layers
Adist
0.2, 0.6,
Kquery
VICE
7IVICE
1, 2, 10) 0, 1
Mixup a
Weight Decay À
0,5 x 10-3
VICE + RND
7IVICE
I, 1, 10] 0,
Mixup
RND reward scale
1,5, 10]
DDL
Nd
4. 10]
Training frequency
16, 64]
(every steps)
Table 7: Hyperparameters we tuned for the Sawyer Push task. Bolded values are what we use for the final runs in Section 6.
A.6.4 SAWYER PICK-AND-PLACE HYPERPARAMETERS
BayCRL Classifier
Hidden
[512, 512), (2048, 2048)]
Layers
Adiet
0.2,0.6,
Kquery
1
VICE
TIVICE
1, 2, 10 0,
Mixup &
Weight Decay À
0,5 x 10-3
VICE + RND
7VICE
1,2, 10
Mixup
0.
RND reward scale
1,5, 10
DDL
Na.
4, 10 [16, 64]
Training frequency
(every steps)
Table 8: Hyperparameters we tuned for the Sawyer Pick-and-Plaas task. Bolded values are what we use for the final runs in Section 6.
24

---

Under review as a conference paper at ICLR 2021
A.6.5 SAWYER DOOR OPENING HYPERPARAMETERS
BayCRL
Classifier
Hidden
[6112, 512), (2048, 2048)]
Layers
Adist
0.05, 0.1, 0.25]
Kquery
[1, 2]
VICE
7IVICE
1,5, 10]
Mixup a
0, 0,5 x 10-3
Weight Decay À
VICE + RND
7IVICE
1,5, 10
Mixup
0,
RND reward scale
1,5, 10]
DDL
Nd
4, 10
Training frequency (every steps)
16, 64
Table 9: Hyperparameters we tuned for the Sawyer Door Opening task. Bolded values are what we use for the final runs in Scction 6.
A.7 PROOF OF THEOREM 1 CONNECTING NML AND INVERSE COUNTS
We provide the proof of Theorem here for completeness.
Theorem A.1. Suppose
estimating success probabilities p(e 1|s) in the tabular setting,
where we have a separate parameter independently for each state. Let (s) denote the number of times state has been visited by the policy, and let G(s) be the number of occurrences of state in
the successful outcomes. Then the CNML. probability PONMI (e = 1[s) is equal to N6))+()))2 For states that never observed to be successful, then recover inverse county N(3+2
G(8)+1
Proof. In the fully tabular setting, our MLE estimates for s[O]s) are simply given by finding the best parameter p. for each state. The proof then proceeds by simple calculation.
For a state with estimate is simply given by g n+t-
Nss) negative occurrences and G(s) positive occurrences, the MLE
Now for evaluating CNML, we consider appending another instance for each class. The new n+11
parameter after appending negative example is then n which then assigns probability T
to the negative class. Similarly, after appending a positive example, the new parameter is _++ we try assign probability n+++1
n+g+T* so
the positive class. Normalizing, we have
PCNML = 1(8) = 9+1
+
When considering states that have only been visited on-policy, and are included in the set of successful outcomes, then the likelihood reduces to
1
PCNML (O = 1[s)
(10)
n1
25


Reviews:
[{"review_id": "mLCde7AjY24", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript aims to solve reinforcement learning problems where the reward is unknown but a set of successful states are available. Iteratively, it trains a classifier using provided successful states as positive and on-policy samples as negative and use its predictions as the reward function to learn RL policy. \n\nIt may be worth clearly explaining the connection and improvement on the work of Fu et al.(2018b, Variational inverse control with events: A general framework for data-driven reward definition) in introduction since both mainly solve the same problems. Fu’s paper first introduced the event framework to generalize inverse RL that also solves the sparse RL problems by iteratively training a classifier to predict the probability on successful states.\n\nCompared to previous work, it seems a major difference in this paper is to change the event classification model, replacing the neural network classifier with CNML classifier. It could enhance the contribution if  good theoretical analysis can be provided. E.g. why the manuscript concludes that CNML performs better than standard neural network clasisifers in terms of uncertainty aware on reward estimation, and how CNML model connects with better exploration and goal-oriented reward shaping.\n\nOther comments are written below.\n\nTheorem 4.1 is defined, but it seems not used in the following sections. It would be helpful if the following can explain whether it is used  to develop the algorithm or explain empirical findings.\n\nIt may need more evidence in Section 4.3 to support the conclusion that the CNML is doing better at reward shaping. It is not easy to make a conclusion based on some specific example in Figure 1. Similarly Section 4.1 also try to show CNML gives reward that can improve exploration and becomes goal oriented using the same example.\n\nSection 5.1 describes how to use meta learning to approximately solve CNML problem so it can reduce computation complexity. It is an interesting idea, but it may need more analysis. Some questions are listed below.\n\n- The distribution of tasks usually has different samples, while the CNML problem has similar data sets for its tasks as each task only adds one sample to the original dataset. Does this difference have influence on learning the model parameter in meta learning?\n\n- It would be interesting to analyze the computation complexity as it is the main reason for applying meta learning in solving CNML.\n\nIn experiments, It may be interesting to test the CNML estimation with full NN convergence on entire augmented dataset every time, to show the trade-off between accuracy and computation complexity when compared with meta learning.\n\nBased on the number of epochs, it seems BayCRL converges slower in some problems, such as zigzag, spiral, sawyer. It may be interesting to give some insights, as good reward may speed the learning process. Moreover, it may be more accurate to consider the time complexity in each epoch when validating the time complexity.\n\nOn page 7, it may need explanation on how to find that BayCRL outputperforms in terms of sample comcplexity as shown in Figure 4.\n\nSome minor comments:\n- On page 5, Appendix Appendix A.5 -> Appendix A.5 \n- It seems Appendix A.2 duplicates Section 5.1, particularly the Eq. 6 and the equations for p_meta-NML and \\theta_y.\n- Is there any reason for line 7 in Algorithm 1 that skips meta learning.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NutPG60XnTO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of learning a policy for an MDP with unspecified reward, given user-provided goal states. To this end, a reward model and a policy are jointly learned: the reward model is the conditional normalized maximum likelihood (CNML) learned from a training set consisting of the example goal states as positive examples, and the policy trajectories as negative examples; the policy is trained to optimize the MDP using the learned reward. Meta-learning is applied to reduce the cost of learning the CNML models.\n\nPros\n+ The idea of using CNML to obtain a smoother reward as compared to a single model (together with the efficient meta-learning approximation) is interesting.\n+ The algorithm is compared with several baselines and seem to perform well. Ablation study suggests that goal examples and meta-learning are important for the proposed approach.\n\nCons\n- The paper is unnecessarily hard to read. A high-level description of the approach early in the paper will be helpful. This is also related to the comments below: it is not clear why the algorithm should work.\n- A claimed contribution of the paper is to \"produce more tractable RL problems and solve more challenging classes of tasks\". The limited feedback provided by the example goal states perhaps make the RL problem more tractable, but how is it possible to solve more challenging classes of problems when less information is available?\n- To learn a useful reward model from the example goal states, the CNML approach alone seems insufficient, and it seems necessary to require a good reward to be a smooth function of feature vectors. For example, if we work in a grid world with random rewards, does the approach still work?\n- Both the paragraph before Sec 3.2 and Alg 1 mention that the set of negative examples keeps growing. This implies that the reward model will become more and more sparser (values closer to zero), even for the goal states? How is such a reward model still useful?\n- Another question about the reward model is that when the policy becomes better, it is more likely to reach the goal states, thus the goal states are more likely to be labeled as both positive and negative. Thus the reward model is more likely to assign lower reward to goal states when more training is done?\n- Fig. 1 seems to be overstating the problem with MLE. What are the features used and what is the classifier model? If the feature is the real-valued position, and a regularized logistic regression model is used, then MLE will not produce such a sparse reward as in (b)?\n- The experiments section should provide more details about the experimental setup: the choice of candidate classifier models, explanation of the baselines (e.g. Sparse Reward seems not mentioned in the text at all), detailed description of the performance evaluation metric. Is Manhattan distance to goal a sensible performance metric for maze navigation?\n\nMinor comments\n- \"OpenAI et al.\": wrong citation format\n- Define L in Eq. (2)\n\nPost-rebuttal\nAfter reading the rebuttal and other reviewers' comments, my score remained the same. The rebuttal helped to clarify some issues, but it is still not clear to me why the algorithm should work. I agree with other reviewers that a more careful revision of the paper, and a further analysis on the algorithm will be beneficial.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KdS-k4Y6b4q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies how to solve RL problems with a set of success states instead of a standard reward function. The central idea is to firstly train a Bayesian classifier from both the input success examples and the on-policy sampling using the conditional normalized maximum likelihood (CNML) and then use the learned classifier as a reward function to guide exploration. It is proved that in a tabular case, the success classifier trained with CNML is equivalent to a version of count-based exploration and it is claimed that with function approximation, the classifier attains non-negligible generalization. Empirically, it is claimed that this approach outperforms existing algorithms on a number of navigation and robotic manipulation domains.\n\nThe novelty of this work lies in the use of CNML to train a more regularized success classifier and further use meta-learning to implement CNML in practice. \n\nThere are several concerns I have:\n1. Can the authors indicate the following information in the experiment:\n- Which algorithms are provided with success examples as prior knowledge in all testing domains? The descriptions in Sec. 6.1 and 6.2 are a little confusing.\n- In Figure 4, I didn’t see the lines for VICE+count-bonus.\n\n\n2. The claim that BayCRL outperforms uninformed, task-agnostic exploration (VICE+count-bonus and VICE + RND ?) is not surprising since the former has prior knowledge.\n\n\n3. The authors claimed that the proposed approach can achieve both effective reward-shaping and exploration. I agree with the first point by comparing it with other IRL methods. But how is the latter true? I think this needs to be further demonstrated in a different setting such as the one in the next point.\n\n\n4. All the tested domains have only one success state. Thus, the example set is informational complete. I wonder about the robustness of the algorithm if the example set does not contain all success states while a reward function (which is surely super sparse) is provided. For example, if there are 10 success ground-truth states and only 5 are provided in the example set and the uncovered states are quite remote from the provided ones (but a reward function is available, i.e., if we reach these hidden success states, a positive reward will be received), then how would this degenerate the performance of the proposed approach comparing with other methods? I think this also testifies the generalization/exploration ability of the algorithm from another perspective.\n\nI vote for weak reject since both CNML and MAML including the reformulation of the problem follow the prior works, which kind of limits the novelty of this paper as applying known algorithms to a defined problem. But I am open to adjusting the score if the rebuttal can address my concerns.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0mkkGx_usM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\n\nThis paper addresses a reinforcement learning problem where the reward\nfunction is learned through a classifier that decides whether states are\nsuccessful or not based on previous examples (i.e. RL after inverse RL).\nThe authors show that this requires uncertainty-aware predictions, which\nare difficult with neural networks. An algorithm, BayCLR, is proposed\nthat uses MAML to meta-learn the conditional normalized maximum\nlikelihood, i.e. the \"maximum likelihood distribution\". Connections to\nthe proposed algorithm and exploration methods are discussed before\nusing the algorithm to solve various robotics tasks.\n\nDecision\n--------\n\nAlthough I liked this paper overall, I am rating it tentatively as\nmarginal below the acceptance threshold. The paper is very well written\nand addresses a relatively clear problem (inverse RL with classifier)\nwith an interesting method (meta-learning CNML). I have some issues with\nunclear statements in the motivation and method that should be\naddressed. While the experiments provide some insight, I think the\nconclusions the authors draw from them are far stronger than the results\nimply.\n\nOriginality\n-----------\n\nI am not very familiar with CNML, but this paper seems very original. In\nparticular, the application of meta-learning to conditional normalized\nseems novel, as well as its application to inverse RL.\n\nQuality and Clarity\n-------------------\n\nThe paper is well-written, most statements are clear and easy to follow.\n\nStrengths\n---------\n\n-   The approach of meta-learning CNML is interesting, and if anything\n    deserves further analysis outside of inverse RL / RL.\n-   Although I have some issues with the motivation (see below), I think\n    the authors do a good job of explaining their rationale for using\n    CNML. In particular, quantifying neural network uncertainty through\n    posterior analysis is quite difficult.\n-   The experiments seem comprehensive. However, I am not very familiar\n    with the environment suite used. Due to the lack of work in this\n    area, there are not many possible baselines, and so the VICE\n    baseline seems like the best choice. The baseline is further kept\n    fair by adding exploration heuristics. I especially like Section 6.2\n    that analyzes BayCRL on the zigzag maze task, which I assume is a\n    tabular environment. In addition, there is a simple ablation but I\n    would prefer more work on this.\n\nWeaknesses\n----------\n\n-   In Section 4, some motivating statements are unclear to me (see\n    Detailed Comments).\n\n-   A single data-point being added to the dataset may not change the\n    distribution much, and this crucial point is only addressed (in an\n    ad-hoc way) in the appendix. It would be good to see an ablation\n    study on this, or perhaps a plot of the average difference between\n    different query points. Perhaps Figure 6 may indirectly explain\n    this, but it should be explicitly addressed.\n\n-   In Section 6, many statements about BayCLR seem stronger than the\n    results imply. Many confidence intervals overlap, and the results\n    themselves are hard to parse with so many lines in each plot.\n    Perhaps some of these baselines which are not competitive can be\n    excluded, or perhaps different linestyles should be used.\n\nDetailed comments\n-----------------\n\n-   Early mentions of exploration (i.e. in the abstract) seem out of\n    place. While you elaborate on the connection to exploration methods,\n    it does not seem like the main point of this paper is to address\n    exploration. It seems to me that you are tackling a novel RL problem\n    where the task is specified through goal states. Specifically, you\n    learn classifier in place of a reward function, and this is\n    exploited to shape an otherwise sparse reward.\n\n-   Section 4.1, \"To create effective shaping, we need to impose a prior\n    on our classifier so that it provides a more informative reward when\n    evaluated at rarely vis- ited states that lie on the path to\n    successful outcomes.\"\n\n    Why is a prior strictly necessary for reward shaping? Unless you\n    mean prior in a very general sense, not a Bayesian prior, I don't\n    see why a prior is strictly necessary.\n\n-   Section 4.1, \"\\[CNML\\].. is essentially imposing a uniform prior\n    over the space of possible outcomes\". This is not obvious to me, and\n    perhaps further explanation is needed.\n\n-   Section 4.2, Theorem 4.1: Perhaps I misunderstand but if $G(s) > 0$,\n    shouldn't $p(e = 1 | s) = 1$? Further, why is that when the agent\n    visit a successful state, i.e. $N(s)$ increases, $p(e = 1 | s)$\n    decreases after each visit.\n\n-   Section 5.1, \"This algorithm, which we call meta-NML, allows us to\n    obtain normalized likelihood estimates without having to retrain\n    maximum likelihood to convergence at every single query point, since\n    the model can now solve maximum likelihood problems of this form\n    very quickly. \"\n\n    Is it correct to say that meta-NML does not need to retrain to\n    convergence at every query point? The second part of the sentence\n    elaborates that metatraining allows you to solve the problem very\n    quickly, but it seems that you still need to solve it at every query\n    point.\n\n-   Section 6.2, Figure 4: I don't think its fair to say that BayCLR\n    performs substantially better. The confidence intervals overlap in\n    all but Spiral Maze and Sawyer 3d Pick-and-Place. Other subtleties\n    are not addressed, such as why RND/count-bonus actually hurt VICE in\n    sawyer 2d push. Other statements such as \"significantly more\n    efficiently\" need explanation as well.\n\n-   Section 6.3, Figure 5: for reproducibility, you should include\n    exactly how many gradient steps are used in the model without\n    meta-learning.\n\n-   Section 6.4, Figure 6: I'm unfamiliar with the environment being\n    used, so some additional details explaining what z means would be\n    helpful.\n\n    \"Furthermore, meta-NML is able to reasonably approximate the\n    idealized NML rewards with just one gradient step…\"\n\n    How is this shown in Figure 6? I don't see anything showing the\n    idealized NML rewards.\n\nMinor Comments\n--------------\n\n-   Section 4, Line 3: missing space: \"ples.For example,\"\n\nPost Rebuttal\n--------------\n\nAfter reading the comments by the other reviewers, I have decided to keep my score at a 5. The authors reply, and the updated manuscript, helped my understanding of the paper. I was considering raising my score, however, the reviewers were nearly unanimous in their confusion regarding the framework or application of CNML. For future iterations of the paper, I suggest that the authors describe CNML, event-based control and their connection more explicitly. If the main contribution is using CNML as the classifier in event-based control, then it would also help to conduct experiments on meta-learning CNML in a supervised learning setting to further elucidate its effectiveness in the reinforcement learning application. I think your paper is very interesting, and I hope that the authors are able to use this feedback to improve their paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:
This paper introduces a method to try to learn in environments where a person specifies successful outcomes  but there is no environmental reward signal.

I'd personally be interested in knowing where people were able to easily provide such successful outcomes instead of, for instance, providing demonstrations or reward feedback. Similarly, I'd be interested in how other methods of providing human prior knowledge compared.

Discussion:
Reviewers agreed the paper was interesting, but none of the 4 thought the paper should be accepted.

Recommendation:
While I do not think this paper should be accepted in its current form, I hope the authors will find the comments and constructive criticism useful.
Decision: Reject