Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation
Authors: Zhizhou Ren
Anji Liu
Yitao Liang
Jian Peng
Jianzhu Ma
Venue: NeurIPS 2022 Accept (2022)
Keywords: Preference-based Reinforcement Learning
Meta Reinforcement Learning

Abstract: Learning new task-specific skills from a few trials is a fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a meta-RL algorithm that enables fast policy adaptation with preference-based feedback. The agent can adapt to new tasks by querying human's preference between behavior trajectories instead of using per-step numeric rewards. By extending techniques from information theory, our approach can design query sequences to maximize the information gain from human interactions while tolerating the inherent error of non-expert human oracle. In experiments, we extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on a variety of meta-RL benchmark tasks and demonstrate substantial improvement over baseline algorithms in terms of both feedback efficiency and error tolerance.

Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation
Zhizhou Ren'2 12, Anji Liu Yitao Liang Jian Peng 126 Jianzhu Ma
Helixon Ltd. University of Illinois at Urbana-Champaiin
3 University of California, Los Angeles
Institute for Artificial Intelligence, Peking University Beijing Institute for General Artificial Intelligence
Institute for AI Industry Research, Tsinghua University zhizhouruhe ixon. com, liuanj i@cs. .ula..a.... edu yitaol@pke @pku edu cn, jianpeng@i 11 liinoii..... edu maj ittiiinnttiiggiiiiii edu.
Abstract
Learning new task-specific skills from few trials is fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL.) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the reinforcement problem of few-shot adaptation in the context of numan-in-the--leee learning. We develop a meta-RL algorithm that enables fast policy adaptation with sefffrrnncebaasee feedback. The agent adapt to new tasks by querying human's 's preference hetween behavior trajectories instead of using per-step numeric rewards. By extending techniques from information theory, design query sequences to maximize the information gain from human interactions while tolerating the inherent error of non-expert human oracle. In experiments, we extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on variety meta-RL benchmark tasks and demonstrate substantial improvement over baseline algorithms in terms of both feedback efficiency and eтror tolerance.
approach
Introduction
Reinforcement learning (RL) has achieved great success in learning complex behaviors and strategies in variety of sequential decision-making problems, including Atari games (Mnih et al. 2015). board game Go (Silver et al. 2016). MOBA games (Berner et al.) 2019), and real-time strategy games (Vinyals et al.) 2019). However, most these breakthrough accomplishments simulation environments due to the sample inefficiency of RL algorithms. Training a policy using deep reinforcement learning usually requires millions of interaction samples with the environment, which is not practicable real-world applications. One promising methodology towards breaking this practical barrier is meta reinforcement learning (meta-RI. Fini al. (017). The goal of meta- RL is to enable fast policy adaptation to unseen tasks with a small amount of samples. Such an ability of few-shot adaptation is supported by meta-training task distribution. Meta-RL algorithms can extract transferable knowledge from the meta-training experiences by exploiting the common structures among the prior training tasks. A series of recent works have been developed to improve the efficiency of meta-RL in several aspects, e.g., using off- off-policy techniques improve the sample efficiency of meta-training (Rakelly et al. 1.01,,Faooo et al. 2020). and refining the cetprratiooneeppiiattioon trade-off during adaptation (Zintgraf et al.
limited in
suite of tasks drawn from prior
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

2020 Liu et al. 2021). There are also theoretical works studying the notion of few-shot adaptation
and knowledge reuse RL problems (Brunskill and Li 2014 Wang al. 2020: Chua et al. (
While recent advances remarkably improve the sample efficiency of meta-RI. algorithms, little work has been done regarding the type of supervision signals adopted for policy adaptation. The adaptation procedure of most meta-RL. RL algorithms is implemented by either fine-tuning policies using new trajectories (Finn al. 2017 Fakoor et al. (020) or inferring task objective from new reward functions (Duan et al. 2016 Rakelly et al. 2019) Zintgral et al. 2020), both of which require the access to the environmental reward function in new tasks to perform adaptation. Such reward-driven adaptation protocol may become impracticable in many application scenarios. For instance, when a meta-trainn tett--rriine robot is designed to provide customizable services for non-expert human users (Prewett) et al. 2010), it is not realistic for the meta-RL. RL algorithm to obtain per-step reward signals directly from the user interface. The design of reward function is a long-lasting challenge for reinforcement learning (Sorg et al. 20TO), and there is no general solution to specify rewards for a particular goal (Amodei et al. 2016; Abel et al., 202T). Existing techniques can support reward-free meta-training by constructing a diverse policy family through unsupervised reward generation (Gupta et al. 2020). and reward-free free few-shot policy adaptation remains a challenge (Liu et al. 2020).
In this paper, we study the interpolation of meta-RL and human-in-the-loop RL, towards expanding the applicability of meta-RL in practical scenarios without environmental rewards. We pursue the few-shot adaptation from human preferences (Fürkrraz et al. (012), where the agent infers the objective of new tasks by querying a human oracle to compare the preference between pairs of behavior trajectories. Such a wreference-based supervision is more intuitive than numeric rewards for human users to instruct the meta policy. e.g., the human user can watch the videos of two behavior trajectories and label a preference order to express the task objective (Christiano et al.) 2017). The primary goal of such a preference- few preference queries to the human oracle. Minimizing the burden of interactions is a central objective of human-in-the-loon learning. In addition, we require the adaptation algorithm to be robust when the oracle feedback carries some noises. The human preference is known to have some extent of inconsistency (Loewenstein and Prelec 1992), and the human user may also make unintentional mistakes when responding to preference queries. The tolerance to feedback error is an important evaluation metric for preference-base based reinforcement learning (Lee et al. 2021a).
adaptation setting is to suit the user's preference with only
To develop an efficient and robust adaptation algorithm, we draw connection with classical problem called Rényí- Caayi--lam'' game (Rényi [96T;; Ulam (976) from information theory. We model the preference-based task inference communicates with the human oracle and infers the task objective from the noisy binary feedback. Through this problem transformation, we extend an information quantification called Berlekamp's 's volume (Berlekamp) 1968) to measure the uncertainty of noisy preference feedback. This powerful toolkit enables to design the query contents to maximize the information gain from the noisy preference oracle. We implement our few-shot adaptation algorithm, called Adaptation with Noisy OracLE (ANOLE), upon advanced framework of probabilistic meta-RL RL (Rakelly et al. 2019) and conduct extensive evaluation on a suite of Meta-RL. benchmark tasks. The experiment results show that our method can effectively infer the task objective from limited interactions with the preference oracle and, meanwhile, demonstrate robust error tolerance against the noisy feedback.
noisy-channel communication problem, where the agent
2 Preliminaries
2.1 Problem Formulation of Meta Reinforcement Learning
Task Distribution. The same as previous meta-RL settings (Rakelly al. 2019). we consider task distribution p(T) where each task instance T induces a Markov Decision Process (MDP). We use tuple My (S. T, R to denote the MDP specified by task T. In this paper, we assume task T does not vary the environment configuration, including state space S, action space A, and transition function T(s' 8, a). Only reward function Rr(- | 88 a) conditions on task T, which defines the agent's 's objectives for task- goals. The meta-RI. algorithm is allowed to sample a suite of training tasks (Ti) N from the task distribution p(T) to support meta-training. In the
meta-testing phase, new set of tasks are drawn from p( P(T to evaluate the adaptation performance.
2

---

Preference-based Adaptation. We study the sho- adaptation with preference- feedback, in which the agent interacts with black-box preference oracle Q rather than directly receiving task-specific rewards from the environment My for meta-testing adaptation. More specifically, the agent can query the preference order of a pair of trajectories (-(1) 7(²) through the black-box oracle Ωy. Each trajectory is a sequence of observed states and agent actions, denoted by T (so; 0o, $1; ... ,81) where L is the trajectory length. For each query trajectory pair (r(2), ,(2)), the preference oracle 52y would return either (1) > -(2) -(1) -(22) according the task specification T, where 7(1) > (2)) means the oracle prefers trajectory 7 to trajectory 7(2) under the context of task T. When the preference orders of 7(1) and 7(2) are equal, both returns are valid. The preference-based adaptation a weak-supervision setting comparison with previous meta-RL. works using per-step reward signals for few-shot adaptation (Finn et al. [217]] Rakelly et al. (219)) The purpose of this adaptation setting is to simulate the practical scenarios with human- -n-th--o- supervision (Wirth et al. 2017: Christiano et al. 2017). We consider two aspects to evaluate the ability of an adaptation algorithm:
Feedback Efficiency. The central goal of RL conducting fast adaptation to unseen tasks with a few task-specific feedback. We consider the adaptation efficiency in terms of the number of preference queries to oracle S2y- This objective expresses the practical demand that we aim to reduce the burden of preference oracle since human feedback is expensive.
Error Tolerance. Another performance metric is on the robustness against the noisy oracle feedback. The preference oracle Ω2 may carry errors in practice, egg. the human oracle may misunderstand the query message and give wrong feedback. Tolerating such oracle errors is a practical challenge for preference-based reinforcement learning.
2.2 Meta Reinforcement Learning with Probabilistic Task Embedding
Latent Task Embedding. We follow the algorithmic framework of Probabilistic Embeddings for Actor-critic RL (FEARL;/Rakell et al.) 2019). The task specification T is modeled by a latent task variable (or latent task embedding) E Z = Rdd where d denotes the dimension of the latent space. With this formulation, the overall paradigm of the meta-training procedure resembles multi-task RL algorithm. Both policy π <a]ss; z) and value function (s,, a; =) condition on the latent task variable so that the representation of z can be end-to-end learned with the RL objective to distinguish different task specifications. During meta-testing, the adaptation is performed in the low-dimensional task embedding space rather than the nigh-dimensional parameter space.
Adaptation via Probabilistic Inference. To infer the task embedding x from the latent task space, PEARL trains an inference network (or context encoder) q(2|c) where is the context information including agent actions, observations, and rewards. The output of ((2(c) is probabilistic, i.e., the agent has probabilistic belief over the latent task space 2 based
its observations and received rewards. We use q(z) to denote the prior distribution for c = 0. The adaptation protocol of PEARL follows the framework of posterior sampling (Strens) (0000. The agent continually updates its belief by interacting with the environment and refine its policy according to the belief state. This algorithmic framework is generalized from Bayesian inference (Thompson) [933) and has solid background in reinforcement learning theory (Agrawal and Goyal Osband et al. 2013 Russo and Van Roy 2014). However, some recent works show that the empirical performance of neural inference networks highly rely on the access to dense reward function (Zhang et al.) | Hua al., 202T). When the task-specific reward signals are sparsely distributed along the agent trajectory. the task inference given by context encoder q(2 c) would suffer from low sample efficiency and cannot accurately decode task specification. This issue may worsen our adaptation setting since only trajectory-wist preference comparisons are available to the agent. It motivates us to explore new methodology for few-shot adaptation beyond classical approaches based on posterior sampling.
3 Preference-based Fast Adaptation with A Noisy Oracle
In this section, we will introduce our method, Adaptation with Noisy OracLE (ANOLE), a novel task inference algorithm for preference-bases fast adaptation. The goal of our approach to achieve both high feedback efficiency and error tolerance.
3

---

3.1 Connecting Preference-based Task Inference with Rényi-Ulam's Game
To give an cntemmation-thhorette view on the task inference problem with preference feedback, we connect our problem setting with classical problem called Rényi- Ulay's 's Game (Rényi 1961 Ulam 1976) from information theory to study the interactive learning procedure with a noisy oracle.
Definition (Rényi-Uland Ulam's Game). There are two players, called A and B. participating the game. Player A thinks of something in predefined element universe, and player B would like to guess it. To extract information, player B can ask some questions to player A, and the answers to these questions are restricted to "yes/no". A given percentage of player A's answers can be wrong, which requires player B's question strategy to be error-tolerant.
In the literature of information theory, Rényi-Ulam" game specified in Definition is developed to study the arror-toleran communication protocol for noisy channels (Shannon] 1956 Rényi and Makkai-Bencsáthh 1984). Most previous works on Rényi-Ulam's 's game focus on the error tolerance of player B's question strategy, i.e., how to design the question sequence maximize the information gain from the noisy feedback (Pelc) 2002). In this paper, we consider the online setting of Rényi- Ulam's game, where player B is allowed continually propose queries based previous feedback.
We draw a connection between Rényi-Ulam's 's game and preference- based task inference. In the con- text of preference-based meta adaptation, the task inference algorithm corresponds to the questioner player B, and the preference oracle 92 plays the role of responder player A. The preference feedback given by oracle Ω is binary signal regarding the comparison between two trajectories, which has the same form as the "yes/no" feedback in Rényi-Ulam 's game. The goal of the task inference algorithm is to search for the true task specification in the task space using minimum number of preference queries while tolerating the errors in oracle feedback. The similarity in problem structures motivates us extend techniques from Rényi-Ulam's game preference-basel task inference.
3.2 An Algorithmic Framework for Preference-based Fast Adaptation
In this section, we discuss how we transform the preference-based task inference problem to Rényi- Ulam's game and introduce the basic algorithmic framework of our approach to perform few-shot adaptation with a preference oracle.
Transformation to Rényi-Ulam's Game The key step of the problem transformation is estab- lishing the correspondence between the preference query preference-based task inference and the natural-language-bssee questions in Rényi-Ulam's ''s game. In classical solutions of Rényi-Ulam' 's game, a general format of player B's questions is to ask whether the element in player A's mind belongs to a certain subset of the element universe (Pele) 2002). whereas the counterpart of question format in preference-basee trajectory to another. To bridge this gap, we use a model-based approach to connect the oracle preference feedback with the latent space of task embeddings. We train preference predictor (+(;;) that predicts the oracle preference according to the task embedding z. This preference predictor can transform each oracle preference feedback to separation
task inference is restricted to querying whether the oracle prefers one
the latent task space, i.e., the preference prediction (-: z) given by task embeddings in subspace of z can match the oracle feedback, and the task embeddings in the complementary subspace lead to wrong predictions. Through this trans- formation, the task inference algorithm can convert the binary preference feedback to the assessment of subspace of latent task embeddings, which works in the same mechanism as previous solutions to Rényi- Ulam's 's game.
Kényi-Ulam's
To realize the problem transformation, we consider a simple and direct implementation of the preference predictor. We train (;:=) on the meta-training tasks by optimizing the preference loss:
Preel (v) = E DxL (I(ra) (2) [T] for (7 > 722) :))]
(1)
where yb denotes the parameters of the preference predictor, (-L(- denotes the Kullback-Leibllr divergence, and [1[ > 7(2) T] is the ground-truth preference order specified by the task specifi- cation T (e.g., specified by the reward function on training tasks). The trajectory pair (2)) is
drawn from the experience buffer, and z is the task embedding vector encoding T. More implementa- tion details are included in Appendix B.2
4

---

Basic Algorithmic Framework. To facilitate discussions. we introduce some notations to model the interaction with the preference oracle Q7. Suppose the adaptation budget supports K online preference queries to oracle Qy, which divides this interactive procedure into K rounds. We define the notation of query context set as Definition to represent the context information extracted from the preference queries during the online oracle interaction.
Definition 2 (Query Context Set). The query context set Qx - denotes the set of preference queries completed at the first rounds, which the task inference protocol queries the preference order between the trajectory pair (T)"), .r)²) at the jth round. To simplify the notations, the trajectory pair (T, "T ) are relabeled according to the oracle feedback so that the preference order given by oracle Q is T. > - for any S S k.
)
The query context set Q2 concludes the context information obtained from the oracle Q in the first rounds. After completing the query round k. the task inference algorithm needs to decide the next-round query trajectory pair (7(1) (Tk+1 7 based on the context information stored in Qk. By leveraging the model-based problem transformation to Rényi-Ulam's game, we can assess the quality of a task embedding z by counting the number of mismatches with respect to oracle preferences, denoted by E(; Qx):
{2; Qx) =
E x < :0)
(1()))(2))E
The overall algorithmic framework of our preference-base few-shot adaptation method, Adaptation with Noisy OracLE (ANOLE), is summarized Algorithm|
Algorithm 1 Adaptation with Noisy OracLE (ANOLE)
1: input: a preference oracle 327, the budget of oracle queries K a prior task distribution ((=), the size of candidate pool M a query generation protocol GENERATEQUERY(()
2: 7 + {zj ~ <(>)}//
>> sample a candidate pool b initialize context information
3:
Co f
4:
for k= to K do
5:
(TR TK
(1) (2)g
+ GEEEERATEQUERY(2, 2x-1) >> critical step: query generation (Eq. (6) request oracle feedback b update context information
(a) Tk (0)) + (T( .7 (2) where u, U E (1,2)
6:
(Th
7: Qk Q1 U(T > > ((1)))
8:
return: arg min {(; Qk)
The first step of our task inference algorithm is sampling a candidate pool of latent task embeddings from the prior distribution g(z). Then we perform K rounds of candidate selection by querying the preference oracle Ωy. The design of the query generation protocol GENERATEQUERY (-) is
critical component of our task inference algorithm and will be introduced section 3.3 The final decision would be the task embedding with minimum mismatch with the oracle preference, i.e., arg min = E(2; Qk).
3.3 Error-Tolerant Task Inference for Noisy Preference Oracle
The problem transformation to Rényi-Ulam's game enables us to leverage techniques from informa- tion theory develop query strategy with both high feedback efficiency and error tolerance.
Binary-Searde Paradigm. The basic idea is conducting a binary-search-ie protocol to leverage the binary structure of preference feedback: After each round of oracle interaction, we shrink the candidate pool Z by removing those task embeddings leading to wrong preference predictions fo z). An ideal implementation of such binary-search protocol with noiseless feedback is expected to roughly eliminate half of candidates using each single oracle preference feedback, which achieves the information-thcortticc lower bound of interaction costs. In practice, we pursue to handle noisy
5

---

feedback, since both the preference oracle Ωy and the preference predictor f& (1:2) may carry errors. An error-tolerant binary-search protocol requires establish an information quantification (e.g.. an uncertainty metric) to evaluate the information gain of each noisy oracle feedback. The goal of oracle interaction to rapidly reduce the uncertainty of task inference rather than simply eliminate the erroneous candidates. In this paper, we extend an rnformation-thhoreed tool called Berlekamp's s volume (Berlekamp) (968) to develop such an uncertainty quantification.
Berlekamp's Volume. One classical tool to deal with erroneous information in search problems Berlekamp's volume, which is first proposed by Berlekamp (1968) and has been explored by subsequent works in numerous variants of Rényi-Ulam s game (Rivest et al., 1980] Pelc 1987
Lawler and Sarkissian 1995 Aigner 1996; Cicalese and Deppe 2007). The primary purpose of Berlekamp's & volume is to mimic the notion of Shannon entropy from information theory (Shannon 1948) and specialize the applications to noisy-channel communication (Shannon 1956) and error-
tolerant search (Rényi 1961). We refer readers to Pele (2002) for comprehensive literature review of the applications of Berlekamp's 's Volume and the solutions to Rényi-Ulam' game.
In Definition S we rearrange the definition of Berlekamp's volume to suit the formulation of preference-basee learning.
Definition 3 (Berlekamp's Volume). Suppose the budget supports K oracle queries in total, and the oracle may have most Kg incorrect feedback feedbbbk among these queries. Berlekamp volume of query context set Qk is defined as follows:
Kg Kg-E((e -(==((;;--)
(K -k
BVolz((H)) = Σ vole(Qun)
(Qs)
Σ Ł=
(3)
8
=EZ
where (K*²)) denotes the binomial coefficient.
As stated in Definition 3 the configuration of Berlekamp's volume has two hyper-parameters: the total number of queries K, and the maximum number of erroneous feedback K z within all queries. This error mode refers to [Bellkkapp (1968)'s noisy-channel model. Berlekamp's 's volume is tool for designing robust query strategy to guarantee the tolerance of bounded number of feedback errors.
What type of uncertainty does Berlekamp's volume characterize? Berlekamp's volume mea- sures the uncertainty of the unknown latent task variable z together with the randomness carried by the noisy oracle feedback. To give more concrete view, we first show how the value of Berlekamp's 's volume BVol 2 @x) changes when receiving the feedback of a new preference query. Depending
(1) (2)), ). the query context set may be on the binary oracle feedback for the query trajectory pair (Tk -k updated to two possible statuses:
e (1)) (2) Ox ( > x( and (2) Qx x (
(4)
where o(2)) (a)) (v) denotes the updated status in the case of receiving oracle feedback Th (a) > +()).
The relation between BVol 2 (Qx-1) and BVol y (Qx) is characterized by Proposition !
which is the
foundation of Berlekamp's volume for developing error-tolerant algorithms.
Proposition 1 (Volume Conservation Law). For any query context set Qa-1 and urbitrary query trajectory pair (TK (1) (21),, the relation between BVol (Qk-1) and (Qx)) satisfies
.Tk
(Qx-)(2x-1) = BVol 3 (2 (2)) + (2 (2)
(5)
The proofs of all statements presented in this section are deferred to Appendix A
As shown by Proposition [ each preference query would partition the volume BVol (Qx-1) into two subsequent branches, 3 ( (0())=22) and = (Q)))())) The selection of preference queries does not alter the volume sum of subsequent branches. Since the values of BVol are non-negative integers, the volume is monotonically decreased with the online query procedure. When the volume is eliminated the unit value, the selection of task embedding with minimum mismatches E(2; Qx) would become deterministic (see Proposition 2
Proposition 2 (Unit of Volume). Given a query context set Qx with BVol (oolzz(O))) 1, there exists
exactly one task embedding candidate z 2 satisfying E(z; Qx) S KE.

---

We can represent the preference- based task inference protocol by decision tree, where each Q with 3 (Q.) = corresponds to a leaf node and each preference query a decision rule. The value of Berlekamp's s volume BVol, & (2) corresponds to the number of leaf nodes remaining in the subtree rooted with context set 2x- The value of z-conditioned volume vol=(Qx: counts the number of leaf nodes with as the final decision. Each path from an ancestor node to a leaf node can be mapped to a valid feedback sequence that does not violate {z; Qk) S Kg. From this perspective, Berlekamp's volume quantifies the uncertainty of task inference by counting the number of valid feedback sequences for the incoming queries.
Error-Tolerant Query Strategy. Given Berlekamp's volume as the uncertainty quantification, the query strategy can be constructed directly by maximizing the worst-case uncertainty reduction:
GENERATEQQEERY(2. Qk-1) = arg min
max BVol (0()))) (Vol=(O)))())) (6)
where GENER ATEQUERY(() refers to the query generation step at line 5 of Algorithm | follows the principle of binary search. If the Berlekamp's volumes of subsequent branches well bal balanced, the uncertainty can be exponentially reduced no matter which feedhack the oracle responds. In implementation, the arg min operator in Eq. (6) is approximated by sampling mini-batch of trajectory pairs from the experience buffer. The query content is determined by finding the best trajectory pair within the sampled mini-batch. More implementation details are included in Appendix B.2
This design
4 Experiments
In this section, we investigate the empirical performance of ANOLE on a suite of Meta-RL benchmark tasks. We compare our method with simple preference-basee adaptation strategies and conduct several ablation studies to demonstrate the effectiveness of our algorithmic designs.
4.1 Experiment Setup
Experiment Setting, We adopt six meta-RL benchmark tasks created by Rothfuss et al. 2011),, which are widely used by meta-RL works to evaluate the performance of few-shot policy adaptation (Rakelly et al. 2019: Zintgral et al. 2020; Fakoor et al., (220)). These environment settings consider four ways to vary the task specification T: forward/backward (-Fwd-Back), random target velocity (-Rand-Vel) Vel), random target direction (-Rand-Dir), and random target location (-Rand-Goal) We simulate the preference oracle 12 by comparing the ground-truth trajectory return given by the MuJoCo-based environment simulator. The adaptation protocol cannot observe these environmental rewards during meta-testing and can only query the preference oracle 12 to extract the task infor- mation. In addition, we impose random noisy perturbation on the oracle feedback. Each binary feedback would be flipped with probability c. We consider such independently distributed errors rather than the bounded-numbon error mode to evaluate the empirical performance, since it is more realistic to simulate human's 's unintended errors. detailed description of the experiment setting is included in Appendix B.I
Implementation of ANOLE. Note that ANOLE is an adaptation module and can be built upon any meta-RL or multi-task RL algorithms with latent policy encoding (Hausman et al., 2018 Eysenbach et al. 2019. Pong et al. 2020; Lynch et al./ 2020 Gupta et al. 2020). To align with the baseline algorithms, we implement a meta-training procedure similar to PEARL (Rakelly et al. 2019 The same as PEARL, our policy optimization module extends soft actor-critic with the latent task embedding 2, i.e., the policy and value functions are represented by T 8; and Qu(s, 2) where 0 and o denote the parameters. One difference from PEARL is the removal of the inference network, since it is no longer used meta-testing. Instead, we set up a bunch of trainable latent task variables [2: J.y to learn the multi-task policy. More implementation details are included in Appendix [B22, The source code of our ANOLE implementation and experiment scripts available at httpp:: // liggttthe tit........ www./twwwwwleeeeeeeeoooool Git Aitpstt/dpppaattiiiio witeeooonwitttttteee withonoo-it----iii--o- OracLE
Haarnoja et al. 2018
7

---

ANOLE (ours)
Greedy Blosy Search
Random Quey
PEARL- AAARR-AAgROooOI
PEARL
PEARL (deuse rowards
MaalCChettah-FwdBBBee
HallCheetah-Rand-Va
Walker2d-Rand-Vel
Average Trajectery Ratarn
2680
2400
1200
2000
803
-150
1080
-300
1280
180
Mets-training Time-steps
Meta-training Time-sfeps
Mets-training Time-steps
Ant-Fwd- Fwd-Back
Ant-Rand-Dir
Ant-Rand-Goal
Avurago Trajectory Return
1300
-240
1003
803
500
-30
693
410
-400
603
300
200
1.M 3,233
272
Metn-training Time-steps
Meta-training Time-stepa
Meta-training Time-steps
Figure 1: Learning curves a suite of MuJoCo-based meta-RL benchmark tasks with preference- based adaptation. All curves plot the average performance from eight runs with random initialization. The shaded region indicates the standard deviation. "PEARL (dense rewards)" denotes the final performance of ordinary PEARL using dense reward signals for meta-testing.
Baseline Algorithms. We compare the performance with four baseline adaptation strategies. Two of these baselines are built upon the same meta- trata-traniigg pre-propee pre-processing as ANOLE but not use Berlekamp's 's volume to construct preference queries.
Greedy Binary Search. We conduct simple and direct implementation of binary-search-like task inference. When generating preference queries, simply ignores all candidates that have made at least one wrong preference predictions, and the query trajectory pair only aims to partition the remaining candidate pool into two balanced separations.
Random Query. We also include the simplest baseline that constructs the preference query by drawing random trajectory pair from the experience buffer. This baseline serves an ablation study to investigate the benefits of removing the inference network.
In addition, we implement two variants of PEARL (Rakelly et al. (019) that use probabilistic context encoder to infer task specification from preference-based feedback.
We modify PEARL's context encoder to handle the preference- based feedback. More specifically, the context encoder is LSTM- network that takes an ordered trajectory pair as inputs to make up the task embedding i.e., the input trajectory pair has been sorted by oracle preference. In meta-training, the preference labeled by the ground-truth rewards from the environment simulator. During adaptation, the PEARL-based agent draws random trajectory pairs from the experience buffer to query the preference oracle, and the oracle feedback is used to update the posterior of task variable z.
PEARL.
PFAARL-+Augmentation We implement a data augmentation method for PEARL's meta- training to pursue the error tolerance of preference-basee inference network. We impose random errors to the preference comparison when training the preference-based context encoder so that the inference network is expected to have some extent of error tolerance.
We include more implementation details of these baselines in Appendix B.3
4.2 Performance Evaluation on MuJoCo-based Meta-RI. Benchmark Tasks
Figure I presents the performance evaluation of ANOLE and baseline algorithms on suite of meta-RT. benchmark tasks with noisy preference oracle. The adaptation algorithms are restricted to use K 10 preference queries, and the noisy oracle would give wrong feedback for each query

---

ANOLK
ANOLE (wlo noise)
Groedy Binary Search
Greedy Binory Search noise)
Ant-Fwd-Back
Ant-Rand-Dir
Ant-Rand-Goal
Average prajectoryy Return
Avorage Trafootory Roturn
-00
-100
Average Trajectory
-140
(00)
-180
693
480
103
-00
-300
-400
Meta-trraining l'mmo-stpps
Meta-training
Tlme-steps
Meta-training T'me-stops
Figure 2: Investigating the impact of oracle noises performance of preference-basee adaptation.
The dotted curves with tag "(w/o noise)" refer to the performance evaluation with a noiseless oracle.
with probability = 0.2. We configure ANOLE with K = 2 to compute Berlekamp's volume. The experiment results indicate two conclusions:
1. Berlekamp's 's volume improves the error tolerance of preference-based task inference. The only difference between ANOLE and the first two baselines, Greedy Binary Search and Random Query, is the utilization of Berlekamp's 's volume, which leads to significant performance gap on benchmark tasks.
2. The con-parametric inference framework of ANOLE improves the scalability of preference- based few-shot adaptation. Note that, using ANOLE's algorithmic framework (see Algo- rithm D random query strategy can also outperform PEARL-based basclines. It may because the inference network used by PEARL cannot effectively translate the binary prefer- ence feedback to the task information.
4.3 Ablation Studies on the Magnitude of Oracle Noise
To demonstrate the functionality of Berlckamp's volume improving error tolerance, conduct an ablation study on the magnitude of oracle noise. In Figure 2 we contrast the performance of ANOLE in adaptation settings with and without oracle noises, i.e., 0.2 vs. = 0.0. When the preference oracle does not carry any error, the simple greedy baseline can achieve the same performance as ANOLE. If impose noises to the oracle feedback, ANOLE would suffer from performance drop, where the drop magnitude is much more moderate than that of the greedy strategy. This result indicates that Berlekamp's 's volume does provide remarkable benefits improving whereas cannot completely eliminate the negative effects of oracle noises. It opens up a problem for future work further improve the error tolerance of preference-based few-shot adaptation. In we conduct more ablation studies to understand the algorithmic functionality ANOLE. Appendix C
tolerance
4.4 Experiments with Human Feedback
We conduct experiments with real human feedback on MuloCo-based meta-RL benchmark tasks. The meta- testing task specifications are generated by the same rule as the experiments in section 4.2 To facilitate human participation, we project the agent trajectory and the goal direction vector to a 2D coordinate sys- tem. The human participant watches the query tra- jectory pair and labels the preference according the assigned task goal vector. The performance of ANOLE with human feedback is presented in Table Each entry is evaluated by 20 runs (i.e., using 20 X 10 = 200 human queries). We note that, in these experiments, the feedback accuracy of the human participant is better than the uniform-noise oracle considered section 4.2 The average error rate of human feedback is 6.2% over all experiments. The evaluation results are thus better than the experiments in Figure
Table 1: Evaluating the performance of ANOLE using human feodback.
Task
ANOLE
Hallfhhettah-Fwd-Bll
1734.3+10.3
Ant-Fwd-Back
931.1438.0
Ant-Rand-Dir 644.8:163.2
We include a visualization of the human-agent interaction interface in
Appendix G

---

5
Related Work
Meta-Learning and Meta-RL. Modeling task specification by a latent task embedding (or latent
task variable) is a widely-applied technique in meta-learning for both supervised learning (Rusu et al. 2019 Gordon al. 2019) and reinforcement learning (Rakelly et al.] 2019). This paper studies the adaptation of latent task embedding based on preference-basee
supervision. One characteristic of our proposed approach is the non-parametric nature of the task inference protocol. Non-parametric adaptation has been studied in supervised meta-learning (Vinyals et al., (016)] [Snell et al., 2017 Allen (et a (219) but is rarely applied to meta-RI.. From this perspective, our algorithmic framework opens up new methodology for few-shot policy adaptation.
Preference-basee RL and Human-ina summan-n--theloop Learning. Learning from human's ss preference ranking is classical problem formulation of
RL (Akrour et al. 2011 2012
Fürkranz et al. 20122] Wilson et al., 2012). When combining with deep RL techniques, recent ad- vances focus on learning an auxiliary reward function that decomposes the trajectory -wise preference
feedback to per-step supervision (Christiano et al. 2017 [barz et al.] 2018). Several methods have been developed to generate informative queries (Lee al. (021b), improve the efficiency of data utilization (Park et al.] 2022), and develop specialized exploration for preference-based RI. (Liang et al. 2022). In this paper, open up new problem for preference-basee based RI,, i.e., meferencc-bbsed few-shot policy adaptation. The potential applications of our problem setting may be similar to that of personalized adaptation (Yu et al. 2021 Wang et 2021), a supervised meta-learning problem for modeling user preference. A future work is considering wider range of human supervision, such
as human attention (Zhang et al.
2020 and human annotation (Guan et al. 2021
Conclusion and Discussions
In this paper, study the problem of few-shot policy adaptation with preference-basee feedback and propose a novel meta-RL algorithm, called Adaptation with Noisy OracLE (ANOLE). Our method leverages classical problem formulation called Rényi-Ulam's 's game to model the task inference problem with a noisy preference oracle. This connection to information theory enable us to extend the technique of Berlekamp's 's volume to establish error-tolerant approach for preference-basee task inference, which is demonstrated as a promising approach on an extensive set of benchmark tasks.
We conclude this paper by discussing limitations, future works, and other relevant aspects that have not been covered.
Adaptation to Environment Dynamics. One limitation of ANOLE is that it does not consider the potential shift of transition dynamics from meta-training environments to meta-testing environments. Our problem formulation assumes only the reward function alters in the adaptation phase (see section 2.I). The current implementation of ANOLE cannot handle the adaptation to the changing environment dynamics. One promising way to address this issue is to integrate ANOLE with classical meta-RL modules to model the probabilistic inference regarding the transition function. In addition, it is critical to investigate the behavior of human preference when the query trajectories may contain unrealistic transitions.
Expressiveness of Preference Partial Ordering. A recent theoretical work indicates that the trajectory-wiss partial preference can define richer agent behaviors than step-wise reward functions (Abel et al. 2021). However, this superior of preference-based learning has rarely been shown in the empirical studies. e.g., in our experiments, the preference feedback simulated by summing step-wise rewards given by the MuJoCo simulator, which is common setting used by most preference-base RL works. In addition, most advanced preference-based RL algorithms are built on methodology that linearly decomposes the trajectory-wise preference supervision step-wise auxiliary reward function, which degrades the expressiveness of the partial ordering preference system. These problems are fundamental challenges to preference- learning.
Acknowledgments and Disclosure of Funding
This work is supported by National Key R&D Program of China No. 2021YFF1200 YFF1201600.
10

---

References
David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of Markov reward. In Advances in Neural Information Processing Systems, volume 34, 2021.
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory, pages 39-1. JMLR Workshop and Conference Proceedings, 2012.
Martin Aigner. Searching with lies. Journal of Combinatorial Theory, Series A, 74(1):43-56, 1996.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 12-27. Springer, 2011.
Riad Akrour, Mare Schoenauer, and Michèle Sebag. APRIL: Active preference learning- based reinforcement learning. In Joint European conference on machine learning and knowledge discovery in databases, pages 1 16-131. Springer, 2012.
Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. International Conference on Machine Learning, pages 232-241. PMLR, 2019.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 2016.
Elwyn R Berlekamp. Block coding for the binary symmetric channel with noiseless, delayless feedback. Error-correcting codes, pages 61-68, 1968.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning, arXiv preprint
2019.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In International conference on machine learning, pages 316-324. PMIR, 2014.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017.
Kurtland Chua, Qi Lei, and Jason D Lee. Provable hierarchy-based meta-reinforreemen learning. arXiv preprint arXi::21 2021.
Ferdinando Cicalese and Christian Deppe. Perfect minimally adaptive q-ary search with unreliable tests. Journal of statistical planning and inference, 137(1):162-175, 2007.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett Ilya Sutskever, and Pieter Abbeel. RI.:: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.661.0000 1.02779, 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-leamnng. In International Conference on Learning Representations, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pages 2052-2062. PMLR, 2019.

---

Johannes Fürmkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang- Baeu- Park. Preference-base reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning. 89(1):123-156, 2012.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference on Learning Represen- lutions, 2019.
Lin Guan, Mudit Verma, Suna Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. In Advances in Neural Information Processing Systems, volume 34, pages 21885-
21897, 2021.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta- learning for reinforcement learning. In International Conference on Learning Representations, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
Yun Hua, Xiangfeng Wang, Bo Jin, Wenhao Li, Junchi Yan, Xiaofeng He, and Hongyuan Zha. Hmrl: Hyper-meta learning for sparse reward reinforcement learning problem. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 637-645, 2021.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, volume 31, 2018
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Eugene L Lawler and Sergei Sarkissian. An algorithm for "Ulam's game" and its application to error correcting codes. Information processing letters, 56(2):89-93, 1995.
Kimin Lee, Laura Smith, Anea Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2021a.
Kimin Lee, Laura Smith, and Pieter Abbeel. PEBBLE: Feedback-efffcient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pages 6152-6163. PMLR, 2021b.
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-baie based reinforcement learning. In International Conference Learning Representations, 2022.
Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcemen learning without sacrifices. In International Conference on Machine Learning, pages 6925-6935. PMLR, 2021.
Evan Zheran Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Explore then execute: Adapting without rewards via factorized meta-reinforcemmett learning. 2020.
George Loewenstein and Drazen Prelec. Anomalies intertemporal choice: Evidence and an interpretation. The Quarterly Journal of Economics, 107(2):533-597, 1992.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. Conference on robot learning, pages 1113-1132. PMLR, 2020.
12

---

Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 (18((750):55995555 -33,, 2015.
lan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. SURF: Semi-supervised reward learning with data augmentation for feedback- meference-basel reinforcement learning. In International Conference on Learning Representations, 2022.
Andrzej Pelc. Solution of Ulam's problem on searching with lie. Journal of Combinatorial Theory, Series A, 44(1):129-144) 1987.
Andrzej Pelc. Searching games with emrors fifty years of coping with liars. Theoretical Computer Science, 270 (70(1-2) 2002.
Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning. In International Conference Machine Learning, pages 7783-7792. PMLR, 2020.
Matthew Š Prewett, Ryan C Johnson, Kristin N Saboe, Linda Elliott, and Michael Coovert. Managing workload in human-rofe robot interaction: A review of empirical studies. Computers in Human Behavior, 26(5):840 85600-5562200 2010.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine Efficient off-policy meta-reinforcemenn learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng. Learning long-term reward redistribution via randomized return decomposition. In International Conference on Learning Representations, 2022.
Alfréd Rényi. On a problem of information theory. MTA Mat. Kut. Int. Kozl. B, 505-516, 1961.
Alfréd Rényi and Zsuzsanna Makkai-Bencsáff A diary on information theory. Akadémiai Kiadó Budapest, 1984.
Ronald L.. Rivest, Albert R. Meyer, Daniel Kleitman, Karl Winklmann, and Joel Spencer. Coping with errors in binary search procedures. Journal of Computer and System Sciences, 20(3):396-404 1980.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. ProMP: Proximal meta-policy search. In International Conference Learning Representations, 2019.
Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4):12211 2014.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
Claude Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):3794423, 1948.
Claude Shannon. The zero error capacity of noisy channel. IRE Transactions on Information Theory, 2(3):8-19, 1956.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panncershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):4844489, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, volume 30, 2017.
13

---

Jonathan Sorg, Richard L Lewis, and Satinder Singh. Reward design via online gradient ascent.
Advances in Neural Information Processing Systems. 23, 2010.
Malcolm JA Strens. bayesian framework for reinforcement learning. Proceedings of the
Seventeenth International Conference Machine Learning, pages 943-950, 2000.
William R Thompson. On the likelihood that unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-25) 2944, 1933.
Stanislaw M Ulam. Adventures a mathematician, New York: Scribner, 1976.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, volume 29, 2016.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350--34, 2019.
Li Wang, Binbin Jin, Zhenya Huang, Hongke Zhao, Defu Lian, Qi Liu, and Enhong Chen. Preference- adaptive meta- meta-learning for cold-start recommendation. In Proceedings of the International Joint Conference on Artificial Intelligence (LJCAI), pages 1607-1614, 2021.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. On the global optimality of model- agnostic meta-learning. In International conference on machine learning, pages 9837-9846. PMLR, 2020.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18:1-46, 2017.
Runsheng Yu, Yu Gong, Xu He, Yu Zhu, Qingwen Liu, Wenwu Ou, and Bo An. Personalized adaptive meta learning for cold-start user preference prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10772-10780, 2021.
Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang Metacure: Meta reinforcement learning with empowerment-drivee exploration. In International Conference on Machine Learning, pages 12600-1261 PMLR, 2021.
Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl Muller, Jake Whritner, Luxin Zhang, Mary Hayhoe, and Dana Ballard. Atari-head: Atari human eye-tracking and demonstration dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 681 6811-6820, 2020.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: A very good method for bayes- adaptive deep rl via meta-learning. In International Conference on Learning Representations, 2020.
14


Reviews:
[{"review_id": "xX9VFbl26g8", "reviewer": "Reviewer_kW96", "summary": "The paper studies the setting of meta-reinforcement learning, but with the constraint that rewards are not observable at adaptation time. Rather, the algorithm must infer the task by querying a noisy oracle (e.g. a human) which indicates its preferences by ranking given pairs of trajectories. The goal is to adapt to the test task with few queries, to minimize burden on the human.\n\nThe authors draw a connection between preference-based adaptation and Rényi-Ulam’s game, an information-theoretic game in which one player tries to guess what the other player is thinking, but they can only communicate via yes/no questions over a noisy channel. Based on this, they propose to use a classical uncertainty quantification tool known as Berlekamp’s volume in order to choose which queries to propose to the oracle. Specifically, they choose the query that maximizes the worst-case uncertainty reduction.\n\nExperimental results indicate that the use of Berlekamp’s volume provides improvement over baselines such as using greedy queries which do not consider the possibility of noise in the oracle’s responses, as well as modifications of PEARL to the preference-based adaptation setting.", "questions": "The query trajectories in Eq. (6) are selected from “the experience buffer”. Based on the fact that you have to subsample this buffer (presumably for computational purposes), I assume this buffer includes trajectories from meta-training time. Is it reasonable to assume that those trajectories will be available at adaptation time?", "limitations": "The authors have addressed limitations, namely that their method cannot completely eliminate the effect of a noisy oracle (which is to be expected).\nI agree with the authors that the risk of negative societal impacts is negligible. If anything, the impact will be positive because it improves the ability of humans to communicate their preferences to deployed AI agents.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The setting of adaptation without dense rewards is useful because it meta-RL more practically applicable.\n* The connection between preference-based adaptation in meta-RL and Rényu-Ulam’s game is novel to my knowledge, and brings information-theoretic tools (specifically, the Berlekamp volume) to bear on the design of the algorithm. This is a useful contribution.\n* The experiments demonstrate improvement due to use of the BVol-based query proposal.\n* The paper is well-written, being organized, clear, and generally well-motivated.\n\nWeaknesses:\n* The paper would benefit from a bit more explanation of the formula for Berlekamp’s volume, since most readers will not be familiar with it.\n* Another useful comparison would be ordinary PEARL using the dense rewards. (I guess you do not even have to re-run it; just use their published results.) This essentially serves as an (unattainable) upper bound on the performance we could expect with a PEARL-like adaptation algorithm, and the gap would show how much is lost by using the preference-based feedback.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gla1QL6vXyS", "reviewer": "Reviewer_rdwj", "summary": "This work considers a new few-shot meta-RL setting, where the agent does not observe rewards during its few shots on a new task, but instead submits pairs of trajectories to a preference oracle, which returns which of the two trajectories is preferred (i.e., would achieve higher returns). The idea is that querying a preference oracle may require less supervision than obtaining reward labels on these trajectories. Then, to solve this setting, this work proposes a new method for task inference, where a preference predictor is learned to imitate the preference oracle, and then the task is inferred by performing an error-tolerant form of binary search with the preference predictor: the idea is to find a task under which the preference predictor most agrees with the preference oracle. This work evaluates the proposed method on standard MuJoCo meta-RL tasks, where only the reward function changes between tasks, and finds that the proposed method outperforms vanilla PEARL (which is not designed to handle this setting). Furthermore, the experiments show that adding error tolerance to the binary search procedure is important when the oracle includes noise.", "questions": "I've condensed the main questions in order of most important to least important I have from my review above here:\n- When is preference-based meta-RL practically better than the standard meta-RL setting (and vice versa)? How does the amount of supervision compare between preference-based meta-RL and standard meta-RL?\n- How do the various methods compare in terms of feedback efficiency? How does performance change as a factor of $K$?\n- Are there reasonable ways to compare with other existing meta-RL algorithms?\n- What happens when $K_E$ is mis-specified?", "limitations": "This work could benefit from more discussion about the limitations of the proposed new setting. Additionally, this work restricts itself to the setting where only the rewards change between tasks, while existing meta-RL works often also consider changing dynamics. It would be useful to see some discussion about whether this restriction is fundamental to the proposed method or setting, or if it can be relaxed in some form.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality\nThis work considers a novel setting, where rewards are not observed in the few shot episodes of meta-RL, and instead, only preferences between pairs of trajectories are given. In addition, and as a result of this novel setting, this work offers a new connection with Renyi-Ulam's games and the notion of Berlekamp's volume, and is overall quite original. However, it is worth noting that the introduction slightly overclaims: Line 48 states that \"reward-free few-shot policy adaptation\" has not been explored in prior work, while [1] does reward-free few-shot policy adaptation, though in quite a different way.\n\n\n# Quality\n- My main concern about the quality is that this work does not obviously validate its claims about feedback efficiency. This work states in several places that it aims to design an algorithm that meets two goals (1) feedback efficiency, and (2) error tolerance. The first goal seems particularly important, since that is a key motivation behind the setting: if it actually requires more supervision than reward labels to provide, then it's unclear why we should consider a preference-based setting. However, the experiments neither test whether the proposed method provides feedback efficiency compared to other methods in this setting, nor whether the preference-based setting requires less supervision than a reward-based setting. I believe that both are critical.\n- Another concern that I have with the experiments is that the experiments consider the case where $K_E$ is correctly specified. However, in practice, it seems unlikely that the noise level of the black-box preference oracle is known a priori. It therefore also seems important to investigate how performance changes if $K_E$ does not match the true noise level.\n- A final concern with the experiments is the set of baselines. Since this work considers a new setting, it is natural that many prior works do not directly apply to the setting, and therefore finding appropriate baselines is tricky. However, while PEARL is an important meta-RL approach, there is a rich set of other meta-RL methods (e.g., RL^2 [2], VariBAD [3]), and it would be helpful to understand whether other existing meta-RL methods also work well in this setting. I note that it's unclear to me how to directly adapt these methods to the new setting in this work, so I do not hold this against this work, but it would also be nice to compare with other meta-RL algorithms beyond PEARL.\n\n# Clarity\nThis work is generally understandable, although the presentation is confusing in some places and requires reading ahead and re-reading. I've noted a few areas below:\n- Lines 163-177 discuss how the preference predictor helps transform the setting to Renyi-Ulam's Game by transforming binary preferences into subset inclusion. However, how the preference predictor achieves this is not clear until later in the binary search discussion. In the introduction to the preference predictor, it's only clear that the preference predictor is learned to match the black-box preference oracle.\n- In addition, while the connection to Renyi-Ulam's Game is interesting, introducing it while introducing the main ideas of the algorithm makes it more challenging to understand what's going on. Perhaps first introducing the main ideas of the algorithm, and later showing how they connect to Renyi-Ulam's Game would be easier to understand.\n- The notation can be quite difficult to parse in some places, with many superscripts. It would be nice if this could be simplified in some areas, if possible.\n\n# Significance\n- My main concern with this work's significance is how practically relevant the setting is. As I alluded to in my section on quality, it's unclear to me that the preference-based setting indeed requires less supervision than providing reward labels. In fact, in the domains included in the experiments, it seems like providing reward labels requires the same amount of supervision. Computing the binary preference seems to require the same computations as the reward labels, e.g., computing the distance from the goal, direction, or desired velocity. I would appreciate if the authors could comment on settings where it is clear that preference-based meta-RL is preferably to the standard setting.\n\n[1] Explore then Execute: Adapting without Rewards via Factorized Meta-Reinforcement Learning. Evan Zheran Liu, Aditi Raghunathan, Percy Liang, Chelsea Finn. https://openreview.net/pdf?id=La1QuucFt8-\n\n[2] RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel. https://arxiv.org/abs/1611.02779\n\n[3] VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, Shimon Whiteson. https://openreview.net/forum?id=Hkl9JlBYvr", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ztes6r2wNR", "reviewer": "Reviewer_L5bT", "summary": "This work studies meta-RL with preference-based reward learning at meta-test time. The authors interpret this problem as classical problem formulation called Rényi-Ulam’s game and design a query selection method based on the technique of Berlekamp’s volume. The authors verified the effectiveness of proposed methods on standard RL benchmarks.", "questions": "* There are various uncertainty-based sampling for improving preference-based RL. For example, one can utilize the uncertainty from reward ensembles. It would be nice if the authors could include more baselines in the experiments.\n\n* Even though experiments on locomotion tasks are interesting, it would also be nice to include robotics tasks. I think Meta-world (https://github.com/meta-world/meta-world.github.io) can be a good candidate\n\n* Som quantitative results which visualize the selected queries from the proposed method and baseline can be interesting to understand the effects of ANOLE.\n\n* Other noisy feedback. In addition to iid noise (flipping with the probability of epsilon), experiments on other noisy feedback can be interesting. For example, authors can consider the Boltzman preference model. ", "limitations": "I'd like to recommend including real-human experiments. Because real humans can show various irrationality and have an inductive bias in decision making, the trends on real humans can be different from synthetic preferences. Therefore, it would be nice if the authors could include real human experiments. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\n\n* Interesting research problem: Meta-RL with preference-based learning is a very interesting research problem, which can be useful for utilizing RL in challenging domains. \n\n* Clear writing: the paper is well-written and easy to follow. \n\n* Interesting connection to Rényi-Ulam’s game & technique of Berlekamp’s volume to generate queries.\n\n# Weakness\n\n* Lack of real human experiments\n\n* Lack of comparison with several baselines\n\n# Overall\n\nThe paper is well-written and the proposed method sounds reasonable. However, more experiments are required to make this work more interesting. My recommendation is \"boardline accept\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3IAE8BssanX", "reviewer": "Reviewer_ZtTs", "summary": "This paper proposes an approach to incorporate preference feedback for meta-RL, as opposed to assuming access to dense environment rewards. This takes the form of querying the oracle for ranking two trajectories, and the method leverages ideas from information theory (berlekamp volume) to deal with noise in oracle feedback. The method is evaluated on 6 gym meta-RL benchmark tasks. \n", "questions": "-> Can this approach actually be used to train agents with a human oracle (without using the mujoco simulator to rank trajectories?). Experiments using human feedback like in section 5.4 of [2] will help demonstrate the efficacy and practicality of the proposed approach. \n\n-> Does the approach scale to more complex meta-learning benchmarks, such as meta-world?\n\n\n[2] : Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (Lee et al.)\n \n", "limitations": "Limitations are adequately discussed", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Significance - \n\n-> The paper addresses an important limitation for practical application of meta-RL, i.e the assumption of dense environment rewards. Querying an oracle (eg: human user) is much more feasible. However, in the included experiments, the number of samples required for meta-training is on the order of a million, and the number of preference queries per step is around 10, putting the total number of queries required on the order of 10 million, so training with this scheme would still be impractical for deployment. It is nonetheless a step in the right direction, since dense rewards are very difficult to specify in the real world for arbitrary tasks, while ranking trajectories can still be done. Further since current approaches to meta-RL don’t commonly use preference feedback, this paper will be valuable to the community in helping to start work in this direction. \n\nOriginality - \n \n-> The method proposed involves bringing preference based learning to meta-RL. The authors leverage ideas from information theory (Berlekamp volume) to account for a noisy oracle, and add this to standard context based meta-RL. This formulation seems novel and while the performance benefit over a simpler greedy binary search baseline is marginal, this paper might inspire subsequent work that borrows from similar ideas. \n\n-> The paper shows advantages of using non-parametric latents (instead of using an inference network like [1]) for the binary feedback problem, as even a random-query strategy can outperform a baseline based on [1]. \n\n-> The related work adequately discusses context based meta-RL and preference based learning. \n\n[1] : Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (Rakelly et al.)\n\n\nClarity - \n\n-> The paper is clearly written, well motivated and easy to follow, and contains ample details (algorithm boxes, hyperparameter details) to aid in reproducibility. \n\nQuality - \n\n-> The preference based meta-RL formulation seems sound. While the experiments only include gym domains these are standard meta-Rl benchmarks. The authors also include ablations where increase in oracle noise leads to worse performance, as expected. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept