Meta-Reinforcement Learning with Adaptation from Human Feedback via Preference-Order-Preserving Task Embedding
Authors: Siyuan Xu
Minghui Zhu
Venue: ICML 2025 poster (2025)
Keywords: Meta-RL
prefrence-based RL
RLHF
Primary area: reinforcement_learning
TL;DR: POEM enables few-shot policy adaptation from human preferences by learning a task embedding space that preserves preference order.

Abstract: This paper studies meta-reinforcement learning with adaptation from human feedback. It aims to pre-train a meta-model that can achieve few-shot adaptation for new tasks from human preference queries without relying on reward signals. To solve the problem, we propose the framework *adaptation via Preference-Order-preserving EMbedding* (POEM). In the meta-training, the framework learns a task encoder, which maps tasks to a preference-order-preserving task embedding space, and a decoder, which maps the embeddings to the task-specific policies. In the adaptation from human feedback, the task encoder facilitates efficient task embedding inference for new tasks from the preference queries and then obtains the task-specific policy. We provide a theoretical guarantee for the convergence of the adaptation process to the task-specific optimal policy and experimentally demonstrate its state-of-the-art performance with substantial improvement over baseline methods.

Meta-Reinforcement Learning with Adaptation from Human Feedback via Preference-OOdde--Preevvig Task Embedding
Siyuan Xu
Minghui Zhu
Abstract
signals. The reward function needs to be properly chosen such that the agent's state-action pairs align well with the task's goal. However, designing such a reward function can be difficult, especially for tasks where the goals are too am- biguous to be directly mapped to specific state-action pairs (Christiano al., 2017; Bai et al., 2022), such as a robot cleaning room preparing meal. Moreover, agents may be required to offer personalized services tailored to human users. Due to lack of domain knowledge, it is hard for the human users to design reward function that reflects their objective. To address the challenge, RL from human feed- back (RLHF) (Casper et al., 2023; Kaufmann et al., 2023) aims solve RL using human feedback, e.g., the binary trajectory comparisons or trajectory rankings, instead of reward signals. Specifically, RLHF learns reward function by aligning the reward signals along trajectories with human preferences and then optimizes the policy by off-the-shelf RL algorithms under the learned reward function. RLHF has achieved state-of-the-ar results in large language model (L.LM) fine-tuning (Ouyang et al., 2022) and preference optimization of text-to-image generation (Lee et al., 2023).
This paper studies meta-reinforrement learning with adaptation from human feedback. It aims to pre-train meta-model that can achieve few-shot adaptation for new tasks from human preference queries without relying on reward signals. To solve the problem, we propose the framework adaptation via Preference Ciffeeence-Oderrrrpe Order- preseevngg EM- bedding (POEM). In the meta-training, the frame- work learns task encoder, which maps tasks to a preference-order preeerrir task embedding space, and a decoder, which maps the embed- dings to the task-specific policies. In the adap- tation from human feedback, the task encoder facilitates efficient task embedding inference for tasks from the preference queries and then obtains the task-specific policy. We provide a theoretical guarantee for the convergence of the adaptation process to the tasc-ipec optimal pol- icy and experimentally demonstrate its state-of- the-art performance with substantial improvement over baseline methods.
Although it addresses the challenge of reward design, RLHF faces the challenge of data inefficiency. First, the large state-action space necessitates substantial preference data to avoid overfitting the learned reward function. However, due to the limitation in annotators' expertise, capabilities, and attention, collecting high-quality human feedback data incurs significant costs and takes considerable time (Casper et al., 2023). Furthermore, optimizing the policy in RLHF relies on conventional RL algorithms, which require exten- sive environment exploration, often amounting to millions of state transition data.
1. Introduction
Reinforcement learning (RL. has achieved significant suc- cess in various sequential decision-makinn tasks, including video games (Mnih et al., 2015; Silver et al., 2016; Lee et al., 2018), robotics (Levine et al., 2016; Lee et al., 2020; Margolis et al., 2021: 2024), and quantitative finance (Liu al., 2021). However, when applied real-world prob- lems, conventional RL methods encounter the difficulty of designing proper reward function (Kaufimann et al., 2023). In RL, an agent seeks to identify the optimal policy that maximizes an accumulative reward function using reward
Meta-reiiforceement learning (meta-RL) (Finn et al., 2017: Beck et al., 2023) aims acquire transferable knowledge by identifying common structures across multiple prior tasks during meta-training, At meta-test time, this knowledge enables the adaptation to unseen tasks using a small amount of data, thereby accelerating the training process and im- proving the data efficiency of RL algorithms. Incorporat- ing meta-RL into human-in- -i-tn--n-theellopp adaptation (Ren et al., 2022; Joey Hejna, 2023) facilitates few-shot adaptation from preference queries without relying on reward signals, and
"School of Electrical Engineering and Computer Science, Pennsylvania State University. University Park, USA. Corre- spondence to: Minghui Zhu <muzl6@pen <muzl6@psu.cduz> Siyuan Xu <spx5032@psu.cdm
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
Table I. Available data for meta-RI. and meta-RI. with adaptation from human feedback
Mcta-RL
Meta-RL with adaptation from human feedback
Meta-training
Trajectories with rewards
Trajectorics with rewards
Meta-test
Few trajectories with rewards
Few trajectories, few preference queries of trajectories
therefore offers promising solution to the data inefficiency problem in RLHE, Paper (Joey Hejna, 2023) applies a super- vised meta- meta-leerning approach, MAML (Finn et al., 2017), to train a meta-reward model, and adapts to a task-specific reward model with few-shot human preference data during the meta-test. However, the meta-test still needs to solve the RL problem to obtain the policy under the learned task- specific reward model, which requires a large amount of state transition data, ANOLE (Ren et al., 2022) adopts the context- based meta-RI. method (Rakelly et al., 2019). Dur- ing the meta-training, ANOLE uses reward signals to learn task encoder, which maps tasks to the task embeddings, and decoder, which maps the task embeddings to the task- specific policies. The meta-test infers the task embedding of new task from its human preference data and then derives its policy. Notice that ANOLE uses the reward signals for the meta-training and the preference data for the meta-test. The mismatch of data types prevents the task encoder from capturing preference-relatee information across tasks, and therefore degrades the performance of task-specific policies.
formance improvement. From the theory perspective, we derive a theoretical result for POEM based on the preference- property of the embedding space, which guarantees the convergence of the policy distribution gener- ated by POEM to the optimal task-specific policy.
Related works. Due to the space limit, related works are included in Appendix A.
2. Problem Statement
MDP and RL task. A Markov decision process (MDP) M & p. the action space A, the discount factor 7, the initial state distribution p over S, the transition probability P(s (s, a) S x x S - [0, 1]. and the reward function r: SSx. x A + [0, [maa). A stochastic policy т : S + P(A) is a map from states to probability distributions over actions, and (als) denotes the probability of selecting action a in state s. The accumulated reward function defined as ((z) & Exg=plEr (<((5,, Oe, (e+1)[=]] m/. An RL task is to identify the optimal policy that can maximize the accumu- lated reward function i.e., T²" A arg max ((x).
is defined by the state space S,
Main contribution. In this paper, we propose the frame- EMbed-
work adaptation via ding (POEM) for meta-RL with adaptation from human feedback. During meta- meta-training, a task encoder is trained with preference data to ensure that the task embedding space maintains the preference- pereer-presriiig property, i.e., the optimal policy for one task is preferred under another task if the embeddings of the two tasks have high similarity. Dur- ing the adaptation from human feedback, when a new task is given by a human, all task embeddings that do not align with the human's preference orders are rejected. Among all the remaining task embeddings, the one with the low- est preference loss on the preference order data is selected to derive the task-specific policy. Our contributions are three-fold. From the algorithm perspective, this paper is the first to propose the preference-order preseeern task en- coder for context- based meta-RL RL training, which establishes a connection between task embeddings and human prefer- ences. This connection facilitates the efficient inference of task embeddings for new tasks during the adaptation from human feedback. From the experiment perspective, conduct experiments on nine continuous control envi- ronments in Mujoco and MetaWorld. The proposed POEM, using few-shot human preference queries, achieves perfor- mance comparable to meta-RL oracle (using reward signals) and demonstrates state-of oftte----he-art results with 20%-50% per-
RL task distribution. Consider a space of RL tasks r, where each task T E I is modeled by an MDP My & {S, A, 7, P. rT), where PT and r" condition on the task T. For task T, the accumulated reward function is denoted as JT and the optimal policy is denoted as ग Assume the RL tasks follow task probability distribution P(T).
Meta-RL with adaptation from human feedback aims to learn a meta-model from the task distribution P(T), which can be adapted to an unseen task Trew ~ PP'))) using few- shot human-prec preference data and few-shot state transition data without the reward signals.
During the vetainraining, a set of tasks (T)) are sampled from P(T),, and the tasks' MDPs {MM,, 1N are explored to generate trajectories. Note that, the reward function rT, of each meta-training task Ty is accessible. Consequently, trajectories with rewards defined as {(s,, az Tt) can be obtained for each task T,.
During the meta-test, a new task Thew is given by a human, and the MDP MT.... can be explored by the agent to obtain trajectories 5 = {(s,, ae st+1))/f However, the rewards along the trajectories cannot be returned. The preference order of a pair of trajectories (71, 72) can be queried. For
2

---

Meta-Reinforcement Learning with Adaptation from Human Feedback
any given query trajectory pair (TL, 72), the preference ora- cle (human) would return Ti >Tnew 72 if the oracle prefers TT over 72 under T; otherwise, Ti <Taee 72 is returned. If the preference oracle does not distinguish 71 and T2, any of T Tnew 72 and Ti -TT..w 72 can be returned. Note that, during meta-testing, both the numbers of sampled trajecto- ries and human preference queries are small. A comparison of available data in conventional meta-RL (Finn et al., 2017: Beck et al., 2023) and meta-RL with adaptation from human feedback is shown in Table 1.
about the task embeddings can be inferred.
In the module of meta-training, we train preference-order preserving task encoder and policy decoder. Specifically, the encoder (=(=7]e [cT)) encodes the task-specific trajecto- ries with rewards (referred to as context c²) into a task embedding vector =" for each training task T, and the decoder policy [T(a]ss = conditions on the task embed- ding 2T obtain the task-specific policy for task T. Two goals are expected for training O and Tq. the preference- order-preserving property holds for the encoder qá and the conditional policy =T( is close to the optimal policy of task T. We incorporate both the preference data and the reward signals to the meta-training to achieve the two goals.
(i) Meta-training Training tooks
qa (r I
Ma(uls.2z
Conditinsl poicy
Encuder
with roward
In the module of adaptation from human feedback, for a given new task Tnew, we sample the trajectory pairs (TI, 72), query the preference oracle to generate the preference or- der of (7,, T2), and then infer the task embedding =T to obtain the task-specific policy ne ne 2 2TTem) ). In particular, for the task embedding inference, we first sample a set of task embedding candidates. Based on the preference-ordee preserving property, we identify the preference order of the trajectories (TI, T2) under each embedding candidate, and remove all embeddings that do not align with the queried preference orders on the new task Tnew- Among all the remaining task embeddings, the one with the lowest prefer- ence loss on the preference query data is selected to derive the task-specific policy.
Preference
Infer embsddiing
Inquctioiis
onea
(ii) Adaptation from Human feedback (meta-test)
Figure 1. Overview of meta-RL. with adaptation from human feed-
back via prefereenee-ordee-preesvvii embedding (POEM)
3. Method Overview
This section presents an overview of our method, adaptation via Preference- Oaaeee--Orrrrpre Order-preserving EMbedding (POEM), to solve the meta-RL. with adaptation from human feedback. We introduce the and design two modules for POEM: (i) meta- meta-training with preference-order peeeeerrrreerinng embedding and (ii) adapta- tion from human feedback by task embedding inference. The structure of the two modules is shown in Figure 1.
ceaeeee-rrdrr--eeeriiii embedding
space
Task embedding
space
fesforward racl
forward mii
optimal polioy RG for7 ochieveo
embedding is to encode each task T to task embedding 2T, where the embed- dings adhere to the The preference-ordderprresevi property refers to that, if the optimal policy of task Ti, denoted as n is pre- ferred over that of another task Tz. denoted as T under a task To. this preference order should be reflected by the similarity order of the task embeddings, i.e., the similarity (inverse measure of vector distance) between the task embed- ding vectors zTo and zTi, denoted as ST .2 "T,),, is higher than the similarity To TT)) between To and Te. For- mally, S(=To, = ZTi) > S( TT)) holds if 7 (T((TT) TTo (T((m)))) where To (Tr) denotes the trajectory sampled by policy on task To. The property is further illustrated by property, the preference order of the task-specific optimal policies can be directly derived from the task embeddings. More im- portantly, when a task is unknown but the preference order under the task is available from human, the information
The wweference-order-preseevve
7g forward 1 nus
The optinnal policy Hj for achioves mis. Tq. policy v5
17/
property.
Figure 2. An example to illustrate the insights of the preference- order-preserving task embedding. Consider three RL tasks To. Ti and To as steering the robot moving forward with 4 m//s 5 m//, and m/ respectively. The task embedding = To is more similar to x Ti than _T= and गं performs better than Tii on task To.
Note that the encoder-decoder structure in the meta-training of Figure is inspired by the context-based meta-RL meth- ods (Rakelly et al., 2019; Zintgraf et al., 2020; Raileanu et al., 2020). As shown in Table the task context (trajec- tories with rewards) can be obtained for both tetaingai and meta-test tasks in meta-RI.. Then, in both meta-training and adaptation (meta-test) of context-based meta-RL meth- ods, the encoder-decoder structure (the blue box) in Figure I is applied to encode tasks and decode task-specific policies. However, for meta-RI. RL with adaptation from human feed-
Figure 2. With the preference-rrder-rreeeve
3

---

Meta-Reinffrremment Learning with Adaptation from Human Feedback
back, reward signals are not available and only preference data can be accessed during the meta-test. ANOLE (Ren et al., 2022) simply borrows the encoder-decoder from the context-based meta-RL, methods and resorts to training an extra reward decoder which generates reward signals during the meta-test. In particular, during the meta-training, the task encoder maps the task-specific trajectory with reward signals to a task embedding, and the task embedding is decoded to both the task-specific policy and reward func- tion by policy decoder and a reward decoder. However, the task encoder cannot capture preference-related features across tasks, and therefore degrades the performance. In contrast, we design the novel task encoder, which extracts task features incorporating preference order information, enabling efficient inference of task embeddings from human preference.
Next, following most existing works of RLHF (Wirth et al., 2017; Kaufmann et al., 2023), the preference order between two trajectories Ti and 72 is modeled as the order of the received (discounted) total rewards, i.e., 71 >T T2 implies the total reward مi 7 rT(s, (1) ,. ,.(41) received from Ti is larger than (or equal to) E/= 7 rT( af²) (2) received from 72- Then, we approximate the expected total reward J. (mī)) in (1) by the average total reward of multi- ple sampled trajectories generated from mt under To, i.e., approximately we have
(1)
To (T 2 JTo (TT ( T To (T Too T To (TTT)) (2)
where -To is the concatenation of multiple trajectories generated by T7. . and TTO (TTT)) is that generated by n Replacing the right- right-hand
hand side of (1) by that of (2), we obtain the following preference-ordee -reeerrreer----innggggg property.
Property 1. For any tasks To. T1. To with their embeddings 2To, 2Ti, TT, we have
4. Preference-Orrer-PPeeeeee Embedding
As mentioned in Section 3, the preferrence-ordee---eeeevviii task embedding connects the relations over the task em- beddings with human preference feedback. The design is motivated by the following two insights.
S(z To 2 ))) 2 S(=T0, = T To (TT, To T To (m7a))
From Property 1, the similarity ordering of task embedding pairs is expected to align with human preference order, i.e., task with a more similar embedding preferred. Lever- aging this property, the peeference-order-peeseee embedding space establishes clear relationship between task embeddings and human preference feedback.
Insight 1. The similarity between tasks should be reflected in the similarity of their embeddings. In specific, if task To is more similar to Ti than T2, the similarity between the embedding vectors of To and Ti denoted as S(2T0, 3(=)).=7T)) should be higher than S(z )T)),, and vice versa. Formally,
task
Finally, we select the similarity metric S used for Property 1. A common selection of the similarity metric between two embeddings is their inner product (cosine similarity), i.e., S(="), =7>) A ( However, if we adopt this similarity metric in this problem, we can prove that it is possible that task embedding space that satisfies (1) does not exist. The formal statement is shown as Proposition in Appendix B. In contrast, in the following theorem, we provide a similarity metric that guarantees the existence of the task embedding space that adheres to (I). Then, Property holds when using the approximation stated in (2).
To is more (or equally) similar to TI than to T2 =T 2 2
This insight is illustrated on the top side of Figure 2 and is widely applied in the design of representation learning meth- ods for images and natural language, such contrastive learning (Chen et al., 2020; Radford et al., 2021).
Insight 2. The second insight comes from an intuition of task similarity: if two tasks are similar, the optimal policy of one task can also perform relatively well on the other task. In specific, consider the optimal policies गテ of Ti and n of T2. If To is more similar Ti than to T2, the optimal policy n7 for task To should be closer to ,,, than to 47.- then ny should achieve better performance on To than Tia Formally,
Theorem 1. For any task space I, there exists task encoder mappings Sr and f : r - R.d, such that for any To, Ti and To E 1, the property in (1) holds when S(2Ti, =(2,7 zz 2T = (T((T) and =T = [T) for i, 0, 1 and 2. In addition, for any task T E I, fT) only depends on rT and (T) only depends on के and PT.
i.e., the total reward JT. ( > JTo (T
To is more (or equally) similar to Ti than to To - JTo ( 2 JTo( (TT;))
The proof of Theorem 1 is shown in Appendix 1.2. As stated in Theorem 1, the task embedding vector 2" can be separated to two vectors: =T - [ , and the similarity between 27i and z zTaa is defined as
This insight is illustrated at the bottom of Figure 2.
Starting from Insights and 2, we can derive the property for the pewperennce-orrr-rreeeeeeee combining Insights 1 and 2, for any tasks To, Tı, and Tz,
embedding. Specifically,
S( Ti T2)
As indicated in Theorem 1, the embedding 27 represents the features of the task-specific reward function rT and 27
S( 2 S(2 To T2) + (Tt)) 2 Jr (- (1)
4

---

Meta-Reinforcement Learning with Adaptation from Human Feedback
represents the features of the task-specific optimal policy r and the transition function Pr. So we denote =x as the policy embedding and af as the reward embedding of task T. Based on Theorem 1, when the similarity metric is se- lected as (3), the task embedding space with the property (1) exists. Note that, the similarity metric in (3) is asymmetric, i.e., S(="1 =²²) S==22 ,=,,,,=),, ). To achieve the property in (I), we design S(2 =2 in (3) to reflect the fitness of n to task Ti. In partieular, (3) employs the cosine similarity between the reward embedding 2!" (represents the reward "T of Ti) and the policy embedding =!Aa (represents the optimal policy T4; of T2) to represent the fitness of Ti; to Ti i.., Jr, (m7a). ). Then, the equivalence in (1) can be estab- lished. As a result, the fitness of T7, to Ti may be different from that of प7. to Tz, which lead to the asymmetry.
We build the encoder based on the variational auto-encoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014; Alemi et al., 2016), which is primarily designed for genera- tive models and has been widely used in context- based meta- RI methods (Rakelly et al., 2019; Zintgraf et al., 2020). Specifically, we train an inference network go. parameter- ized by o, to encode the context cd to a distribution within the embedding space qs (-Ic").. The task embedding =7 is sampled from the distribution qo (=(-)c") and then is mapped to the fiferrenee-order-peeeevvee by two networks Sa. and fe.: parameterized by or and Φn. where the reward embedding 2. = (=7) and the policy embedding 2 = fox (=7). The reward embedding 27 is used in the conditional reward function re, ro, (-12," to recon- struct the reward function r and the policy embedding *! is used in the conditional policy тæ_ (+1=2) ) recover the task-specific optimal policy गH, while 2f and 2I are expected to satisfy Property for any task T.
task embedding space
5. Meta- Meta-Training Training with Preference-Oree Order- Preserving Task Embedding
To achieve the above goal, we design the following opti- mization problem to train the networks in Figure 3 with parameters t Ør, ΦT, 0r, 0z]:
The section introduces the meta-training with the preference- under-preservinn task embedding. Section 5.1 introduces the network structure of module (i) in Figure 1 for the meta- training, which includes the preference-ordee-prresss encoder and the policy network conditional on the task embeddings. Section 5.2 presents the composition of the task context. Scction 5.3 introduces the recursive sampling and training procedure for the meta-training.
min E BT (as (-1 (1cc (O, I)) + (2,, +
+ 3,C, T) + Spreeprr ( T, Tı, T2)].
The objective function includes four loss terms. (i) The KL divergence loss DKL
(0, I)) enables the prior distribution the embedding space, the normal distribution (0, I) to approximate the posterior distribution go(-le"), as used in VAF. (ii) The reward reconstruction loss aims to reconstruct the true reward function rT by the reword network re, under the task reward embedding 2,,, which is defined as (, A
5.1. Encoder-Decodee
As stated in Section 3, during the meta-training, we aim to train an encoder to map the context cT into the task embedding 2" and train a decoder from 2" the task- specific policy, i.e., policy To TT(- that conditions on 2T. Moreover, the embedding space is expected to be preference- order-preserving, i.e., it should satisfy Property 1 shown in Section 4. In this section, we design a network to achieve this goal, which holds a structure shown in Figure 3.
((r (ooo((l ((rraa)))__(((((s - (s, (s, al2,1)) (4)
with x,- = fo( (iii) The policy reconstruction loss aims to reconstruct the task-specific optimal policy क. by the policy network тo, under the task policy embedding =T which is defined as
nhe (s, 2f)
ge (z
le")
fa.
C (2, T) A (ETTa (DKL (1=4))
(5)
Contaxt or
+
with 27 fo- (iv) The preference loss enforces T to satisfy Property I and defined as Cyre ( T, Tí, T2) A
DeclAs le (e"))
Te, (f)
E Targo (o0 [DKL [[ T - (7(((T)) (m+))) Pr|S(= TT > S(x 2 []]] (6)
Figure 3. Encoder-decoder network and imposed loss functions, where using the KI. divergence loss DEL for enforcing the pos- terior distribution go (-le") to (0, I), the reward reconstruction loss L. for recovering the true reward rr by (-, TT), ). the policy loss Cu for recovering optimal policies by Te_ (.=z*, ), and the preference loss Lyve for enforcing 2" to satisfy Property 1.
where TT (T ) and TT (m+tt) are trajectories generated by पे and प&. on T. Here, [l[[" ( >T 77 (T²²) ) is the ground preference and Pr[S(= T, , > S(2T (((T.=Th) [.)
is the preference prediction based on Property Following
5

---

Meta-Reinforcementt Learning with Adaptation from Human Feedback
the Bradley-Tery model (Bradley & Terry, 1952), we use the predicted probability
cedure. In specific, we first replace the policy recon- structive loss in (5) with the policy optimization loss, i.e., La (x 7) A (1
(+1=2) )}}, such that the conditional policy network Te_ (. can be gradually op- timized toward the task-specific optimal policy my. Next, as illustrated in Figure 4, at epoch n of the meta-training. we replace the optimal policy FÝ in the preference loss (6) and the context (8) by the policy T (-, x TT.(n-1)), where both the policy network T and the embedding
Pr[S(E], 1 > S(x 2 ]]
exp (0((5))) T:)
)) + exp (S(2T, =77)))
So, (6) minimizes the distance between the true preference and the preference predictor to achieve Property 1.
(n- are obtained from the last epoch n
5.2. Context design
In this section, we introduce the composition of the task context cT which is subsequently encoded to the task em- bedding =²_ As discussed in Section 5.1 and Figure 3, the task embedding: =" is used to recover both the reward func- tion and the optimal policy. Therefore, we expect the context eT to include two key pieces of information: (i) the map- ping from the state transition the reward, i.e., (s(9) aa(),
Meta-training for opoch 12
77000
Context 7000)
90000
Tepils mm))
Fgg ( (ine)) at spoch nn
((a0),0) (000, M
Explore and sample
g((+1)) + TT (ii) the optimal policy, i.e., s(() + (s(1)) Accordingly, we consider the task context as
Environments
Figure 4. Recursive sampling and training procedure.
C ((s10 a00) s((t1),
(9) rT O7 ((0),(9))
(8)
The recursive procedure continuously optimizes the policy, while using the current policy as the approximation for the optimal policy to train the encoder and sample the context. When the conditional policy network ne, (,,²z) converges to the task-specific optimal policy ny. the sampled con- text cT is the optimal trajectory in (8) and the employed preference loss is exactly (6).
where ((")) ~ T7 (-15(9) and Q7 (s(1) a()) is the optimal value on (s(1), ). Then, the information about the opti-
(TT(s())
mal policy is incorporated in at') and 077
In existing context-based meta-RL. methods, such as PEARL (Rakelly et al., 2019), the context of PEARL is defined as cT = {(s(t) (t) s ((++1), ,(4)) )) where (t)) is not sam- pled from the optimal policy (-ss())) As result, does not include information about the optimal policy and thus performs worse than (8) (As tested in Section 7). More im- portantly, the task context in (8) is particularly designed for the meta-RL with adaptation from human feedback. Specif- ically, (8) cannot be used to meta-RL. RL, because the optimal action (1) and the optimal value Q7 (s(1), a())) cannot be obtained for a new given task and therefore cannot be en- coded to =T. In contrast, as indicated in Figure I, after the meta-training stage of the proposed method POEM, i.e., the done, the encoder
Finally, we summarize the meta- veta-training algorithm in this section as Algorithm 1. In Algorithm 1, we apply the re- cursive sampling (line 16) and training (lines 4-14) for the meta- mreta-tranning Following the suggestion in context-based meta-RL RL methods (Rakelly et al., 2019; Fu et al., 2021; Yu et al., 2024), we employ the soft actor-critic (Haarnoja et al., 2018) for policy optimization. For the encoder qá. we employ the cormutation-invariann probabilistic network architecture used in PEARL (Rakelly et al., 2019).
training of the networks in Figure ds and the context cT will be not used during the meta-test phase. Instead, the human feedback data is used infer the task embedding 2T to obtain the policy. Therefore, we can use the context in (8) to be as informative as possible to train& a well-structured embedding space.
6. Adaptation from Human Feedback by Task Embedding Inference
In this section, we introduce the adaptation from human feedback algorithm, which is applied after the meta-training described in Section 5.
After the preference-ordde--reeeevii
task embedding en- coder qa- fe.. for and the policy network ne, that shown in Section 5 are trained, for a given task Taem E PP()), if we can obtain its task embedding ZTne, the task-specific policy can be derived as To,, (1, =T ). As a result, the adap- tation from human feedback is to obtain TTee More im- portantly, the task embedding 2 Tnew should satisfy Property
5.3. Recursive sampling and training procedure
When using the network structure and the loss functions in Section 5.1 and the context in Section 5.2 for the meta- training, an issue is that the task-specific optimal policy क in (5), (6) and (8) is not available. To address this issue, we employ a recursive sampling and training pro-
6

---

Meta- Learning with Adaptation from Human Feedback
Algorithm 1 Meta-training algorithm
Algorithm 2 Adaptation from human feedback
Require: Training tasks (7:)1=1 from P(T'; initial net-
Require: Learned parameters Φ { Φr, Un Or, 0x }. maximal preference queries K, error tolerance model e defined in (9).
work parameters ((()- = (ol0), ( 100 0f 0 Initialize replay buffers B¹ for each training task T.
2: Sample set of data (s, (s,,,'',, (1,, a)) by the initial
1: Preference query set 2o < 0
policy " (-|zz where =TT.00) ~ NN 1) and ((s.a) set to 0, and add them to B for each T3
2: for k = 1,... K do
3: The candidate embedding set 2x 0
4: while |2,r| - N₂ do
3: for epoch n 1,... N do
5: Sample 2 ~ (0,
4: for step = 1, MM do
6: if S2, >. S for all (z', E 2x then
5: for each T, do
Add 2 to 2g
Sample context cTil))) ~ Bi 7: Sample 2T.i(n)) ~ Ig(nn (-1c TT(n))
6:
end if
end while
8:
Sample RL batch b' B
10:
Select (2', 2n) from Z by (10)
9:
LKL (T:) DKL (gacns (-(cTT())))
11:
Query the preference oracle on the new task Tnew
10:
Set L. (Ti) the policy optimization loss of SAC under the policy Tegn (-|=7 (T)( on b'
for trajectories 7' = T -T (Te, (-1²)) and 7"
J Tnow (To, (To. (1227)
11:
Set (T.) as (4) under the embedding 2 2T.(m) Sample tasks T T.". and set Epre T( as (6) under the embeddings TT(()), TT,)m) TT,'.ou)
12:
if > TTee then
12:
13:
Qx Qx-1 U (2',
14:
else
15:
2s - (3",
13:
end for
16:
end if
14:
Optimize (in) by minimizing (k)( (T)) (T)+ Spre (To, T(")
17:
Compute the policy Te, (1124) for task Trew where
z² = arg max z E(s, ae,, log Pr(S(2zz') >
15:
end for
S(2,z")
16:
Clear B and sample set of data (s, a, s r, ((s a)) by TI Togn (-127 (1))) and add them to B for each T
18: end for
17:
((a+1) <- (())
Property to incorporate a preference error tolerance model e. Specifically, if the policy on 2' is preferred than the policy z", then S(z,z') >. ((z,="), which is defined as
18:
end for
i.e., for the task Tnew: the task specifcc policy To, (1. 2T) of ' is preferred if the similarity between 2Tnew and 2T is higher. Leveraging this property, we deploy two policies Te, (-. 2T)t) and Te, (,,z7a) on Tnew, query the preference oracle to obtain the preference ranking, and thereby deter- mine whether zTnew is more similar to 2Ti or 2T², which positions 2 Tnew in a half-space. Repeat the above procedure with different policy pairs, we can progressively position the z Them more precisely in the task embedding space. The algorithm of inferring the task embedding 2 Tnew is stated in Algorithm 2.
exp (S(z, 2²))
1 -
2
(9)
exp (S(z, + exp (S(z, z))
2
The condition in (9) includes all with S(2, 2¹) > S(2, 2")
and 2 with that S(2, 2') is close to S(z, =²). It can be seen as the preference probability defined in (7), Pr[S(=T , =T)) > S(2 , ]]] Z 1/2 - e. In line 17, we apply the maximum likelihood over all the embedding candidates to determine the output task embedding y² and policy Te_ (-|=²).
Before querying the preference oracle, need to decide which two trajectories are queried. In line 11, to maximize the query efficiency, we select the query trajectory pairs that can eliminate the task embeddings as many as possible, i.e.,
In Algorithm 2. we continuously query the preference oracle on Tnew then add the preference query set 2s (lines 11-16). Simultaneously, in lines 4-9, we sample the task embedding 2 from (o, and only keep the task embedding candi- dates that align with all preference queries in the query set, i.., 2 with S(2, z/)= > S(2, 5(2,2") for that the policy on 2' is preferred than the policy on z". Note that the human pref- erence queries during the adaptation from human feedback may include errors, and Property| 1 may not be strictly sat- isfied for the learned preference- ordere---orrrrreeee presererigg encoder, especially when two query policies are similar. So, we relax
(E', "",=")== arg min ,'' (max (max{12(0) (11(211122211
(10)
where z = f= Z S(2,2') >. S(z, 2")} and ( {2 2, S(2, >. S(2, )²)) Here, 2 is the set of remaining embeddings when z' is preferred and 2(2) is that when z" is preferred. Considering the worse case, i.c., one of z(1) and Z(2) with the larger size will be remaining, (10) first takes the set max max [ to pick the larger set (1) and 222), and then minimizes its size.
7

---

Meta- Meta-Reiiforcemment Learning with Adaptation from Human Feedback
POEM (ours)
ANOLE ANCLE
ANOLE greedy
1eWard
Cracie
Haltherigh Noleera-
Volning
Humnnng 20Werittiirgg
Ansragge 4wD. romaan))
200
700
125
250
100
475
130
Mole buining ing tie skeps (1e5)
Mete vaning dipseaop (Tes)
Mclo-urnigg MPD 20ps (166)
Made Aigabanne ning L0n 2Aps (166)
HolfChooter Welidaattion from humar bodtook
Azt-tle Uedestetion Seachork
Hkmmnco440 20ddeeeeee from human Swathhrk
Mepp 10Mdupssion
125
475
Nurbbe of pretennos querae
Numibe preforeno quolis
Number of preforence cuorics
Number of profernnc suoies
Figure 5. Performance on HalfChectah- Vel, Ant- Vel, Humanoid Dir, and MetaWorld- Average accumulated reward on test tasks v.s. samples collected during meta-training (Top) and during adaptation from human feedback (Bottom). "Oracle" denotes the performance when the reward signals are available during the meta-test.
Theorem Assume that (i) the task encoder f :
that the probability density of Zu on =Tee increases mono- tonically. If the preference oracle has no error, i.e., c = 0, then P(Zx 2²now) O which increases exponen- tially, and the candidate distribution Zμ converges to 2 Tnew in distribution. To the best of our knowledge, Theorem 2 is the first to provide a performance guarantee for meta-RL with adaptation from human feedback.
Rdd holds that, for T P(T), 2T = [(T) follows the normal distribution (0, 1); (ii) fo and fn Rt + Rd mappings such that 277 = fz) and = f= satisfy Property| 1; (ūi)) the policy network with 0z is optimal, i.e., To, (-12T) = गे.. Suppose the preference oracle in Algorithm 2 on task Tnew holds the error at most e, i.e. the condition in (9) is satisfied. Let Zy be the random variable sampled as same as Z& in Algorithm Let Z be random variable following (0, I). Under certain mild conditions on the mappings I, and f we have
7. Experiment
In the experiments, we evaluate the performance of the proposed approach, POEM, by addressing the following re- search questions: (i) How does POEM perform compared to baseline methods of meta-RL with adaptation from human feedback? (ii) Can POEM effectively handle errors in the preference oracle? (iii) How do individual components of PEOM impact its performance?
(a) The probability density function (PDF) of Zy at 2 Taem
has P(Zz- = ==Teex) (1) 1426)
P(Z==Tnew Tnew)
(b) When == 0, Zy 4 Tnew, i.e., Z& converges to Tnow in distribution.
Experiment settings. We conduct experiments on nine scenarios in two benchmarks centered around robotic con- tinuous control tasks. The first benchmark is MuJoCo, involving robotic locomotion tasks simulated by the Mu- JoCo physics engine (Todorov et al., 2012), which is a widely adopted platform in meta-RL works (Finn et al., 2017: Rothfuss et al., 2019; Rajeswaran et al., 2019) to test the performance of few-shot policy adaptation. The sec- ond is World (Yu et al., 2020), benchmark suite for robotic tabletop manipulation, which includes diverse set of motion patterns for robotic arms and is recognized as a more challenging benchmark for meta-RL. RL. Detailed descrip- tions for the two benchmarks can be found in Appendix C. Across the nine scenarios, each scenario consists of a task
Here, C1 and C2 are the constants, and x 2Tuww is the embed- ding of Tnew such that JT... (To, (-I2Tnew)) JTee (T
The full statements of the assumption on fr and fx. the con- stants C1 and C2 in Theorem 2, and the proofs of Theorem 2 are provided in Appendix B.2. In Theorem 2, we derive the performance guarantee for Algorithm 2. Theorem first assumes the encoder-decoder network shown in Figure is well-trained. Specifically, (i) states that the posterior distri- bution is the normal distribution; (ii) requires that Property holds, and (iii) ensures that the optimal policy is accurately reconstructed. Theorem defines the candidate embedding distribution Zg. from which the candidate embedding set 2 Algorithm 2 is sampled, i.e., 2z Zg. It shown
8

---

embedding (POEM) Meta- RectoReeiffrreement Learning with Adaptation from Human Feedback
POEM
ANOLE
ANCLE
ANOLE greedd
newaad
Cracie
HalCChotshh
haining
Avonag 300um (0wee//ee roaaded
100
Netu-buring Emn 4apa (100)
Mate Emn slapalappaaooooooo (1o5)
Nate-zaning are-skpa (160)
mun--innngg Bro--kopa (166)
Figure 6. Performance evaluation under a noisy preference oracle. Average accumulated reward on test tasks v.s. samples collected during meta-training HalfChectah- Vel, Ant-Vel, Humanoid- and MetaWorld MLL10. "Oracle" denotes the performance when the reward signals are available during the meta-test.
family, where each task within the family is characterized by its dynamic system and reward function. The details of the task settings can be found in Appendix C.2.
comparable to that observed with an accurate preference oracle, as shown in Figure 5, highlighting its robustness to preference errors. Notably, (Ren et al., 2022) has demon- strated ANOLE's strong performance in handling preference noises, and the proposed method significantly outperforms ANOLE in these scenarios. Detailed descriptions of the reward model, comparison results for more scenarios, and detailed analysis are shown in Appendix C.S.
Baseline algorithms. We compare our method, POEM, with four state-of-the-art baseline methods for meta-RL with adaptation from human feedback: (a) probabilistic context- based meta-RL RL with three strategies for the adapta- tion from human feedback: ANOLE, Greedy Binary Search, and Random Query, which are introduced in (Ren et al., 2022); and (b) MAML-reward (Joey Hejna, 2023), which applies MAML (Finn et al., 2017) to train meta reward model to adapt the task-specific reward model. Moreover, we also compare with (c) oracle, which is obtained when the reward signals are available during the adaptation. Since MAML-reward only trains the reward model during the meta-training, there is no performance curve for the meta- training policies. To ensure fair comparison, the data requirements for all the methods are the same during both the meta- trataatraiiing and the adaptation from human feedback. More details of the experimental settings are introduced in Appendix C.3.
HaltChontn Mothe
Aeerige soenggoccuumain aoc.m www.dddddddddddddddd roward reward
Aourago accumukazo soowardd
100
-10
Tsovensoder sackeen 1020
Usig certed PCEM (oun)
C5 volning line
Messttuii Siee
Figure 7. Ablation studies on Halfchehta- Vel.
Ablation studies. We conduct ablation studies to test the impact of components designed for POEM, including (i) the two-encoder structure, i.e., the policy/reward embedding encoders, with distance metric in (3) (Section 5. and (ii) the context design (Section 5.2). Detailed descriptions of the comparisons between the components designed for POEM and those used in previous works are shown in Appendix C.7. Figures 7, 12, and 13 demonstrate the effectiveness of the components (i) and (ii) designed for POEM.
Performance evaluation. Figure 5 compares the perfor- mance of POEM with the baseline methods under an accu- rate preference oracle four scenarios. The results for the remaining five scenarios are shown in Appendix C.5. It is shown that the proposed method, POEM, significantly out- performs all the baseline methods, i.c. 20%-50% improve- ment over the best baselines in terms of the accumulated rewards during both the meta-training and the meta-test. More detailed comparisons between POEM and the base- lines are shown in Appendix C.5.
8. Conclusion
In this paper, we propose novel meta-RL algorithm, adap- tation via preference-order-preeee for meta-RL RL with adaptation from human feedback. The key idea of POEM is to train task encoder such that the task em- bedding space holds the preference-orde perereeeerrrr----iiingggg prop- erty. We provide a theoretical analysis regarding the conver- gence of the adaptation process and empirically demonstrate its superior effectiveness.
Performance under noisy preference oracle. We evaluate POEM when the preference oracle holds noises during the adaptation from human feedback. We use the Boltzmann model, introduced in (Lee et al., 2021a), to model the human preference errors. Figure illustrates the performance of POEM under noisy preference oracle across four scenar- ios. The results indicate that POEM achieves performance

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
Acknowledgements
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
This work is partially supported by National Science Foundation through grants ECCS 1846706 and ECCS 2140175. We would like to thank the reviewers for their constructive and insightful suggestions.
Duan, Y., Schulman, J., Chen, X., Bartlett, L., Sutskever, I. and Abbeel, RL2: Fast reinforcement learning via slow reinforcement learning. In International Conference Learning Representations, 2017.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Finn, C., Abbeel, P., and Levine, Model-agnostic meta-
learning for fast adaptation of deep networks. Interna- tional conference on machine learning, pp. 1126-1135, 2017.
References
Fu, H., Tang, H., Hao, J., Chen, C., Feng, X., Li, D., and Liu, W. Towards effective context for metu-reinforcement learning: an approach based on contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pp. 7457-7465, 2021.
Alemi, A. A., Fischer, Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv: 161.2.0010000 2016.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870, 2018.
Beck, J.. Vuorio, R., Liu, E. Z., Xiong, Z., Zintgraf, L., Finn, C., and Whiteson, S. survey of meta-reinforcement learning. arXiv preprint arXiv:2301. 08028, 2023.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Biyik, E. and Sadigh, D. Batch active preference-based learning of reward functions. In Conference on robot learning, pp. 519-528. PMLR, 2018.
Ji, K., Yang, J., and Liang. Y. Bilevel optimization: Con- vergence analysis and enhanced design. In International Conference on Machine Learning, pp. 4882-4892, 2021.
Biyik, E., Huynh, N., Kochenderfer, M., and Sadigh, D. Active preference-based gaussian process regression for reward learning. In Robotics: Science and Systems, 2020.
Joey Hejna, D. S. Few-shot preference learning for human- in-the-loop rl. In Conference on Robot Learning, pp. 2014-2023 PMLR, 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 32443333344344 1952.
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.1492. 2023.
Casper, S., Davies, X., Shi, C., Krendl Gilbert, T., Scheurer, J., Rando Ramirez, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., al. Open problems and fundamental limi- tations of reinforcement learning from human feedback. Transactions Machine Learning Research, 2023.
Kingma. D. P. and Ba, J. Adam: A method for stochastic
optimization. arXiv preprint arXiv: 2014.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint 131 2013.
Chakraborty, S., Bedi, A., Koppel, A., Wang. H., Manocha, D., Wang, M., and Huang, F. Parl: A unified framework for policy alignment in reinforcement learning from hu- man feedback. In The Twelfth International Conference on Learning Representations, 2024.
Lee, D., Tang, H., Zhang, J., Xu, H., Darrell, T., and Abbeel, P. Modular architecture for stareraft with deep rein- forcement learning. Proceedings of the AAAI Con- ference on Artificial Intelligence and Interactive Digital Entertainment, volume 14, pp. 187-193, 2018
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., and Hutter, M. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47):5986, 2020.
10

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
Lee, K., Smith, L., Dragan, A., and Abbeel, P. B-pref: Benchmarking meference-based reinforcement learning- In fiith Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track, 2021a.
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, a 518(7540): 529-533, 2015.
Lee, K., Smith, L. M., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Interna- tional Conference on Machine Learning, pp. 6152-6163. PMLR. 2021b.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 0 2022.
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C.,, Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text- to-image models using human feedback. arXiv preprint arXiv:2302. 1212 2023.
Radford, A., Kim, J. W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763 PMLR, 2021.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Raileanu, R., Goldstein, M., Szlam, A., and Fergus, R. Fast adaptation to new environments via policy-dynamics value functions. In Proceedings of International Confer- ence on Machine Learning, pp. 7920-7931 2020.
Liang, X.,
Shu, K., Lee. K., and Abbeel, P. Reward uncer- tainty for exploration in preference-based reinforcement learning. In International Conference on Learning Rep- resentations, 2022.
Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S.
Meta-learning with implicit gradients. Advances neural information processing systems, 32, 2019.
Liu, H., Socher, R., and Xiong, C. Taming maml: Efficient unbiased meta-reinforcement learning. In International conference on machine learning, pp. 4061-4071 2019.
Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, Efficient off-policy meta-reinforcement learning via prob- abilistic context variables. In International Conference Machine Learning, pp. 5331-5340. PMLR, 2019.
Liu, S. and Zhu, M. In-trajectory inverse reinforcement learning: Learn incrementally before an ongoing trajec- tory terminates. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Ren, Z., Liu, A., Liang, Y., Peng. J., and Ma, J. Efficient meta reinforcement learning for preference-based fast adaptation. Advances in Neural Information Processing Systems, 35:15502-15515, 2022.
Liu, S. and Zhu, M. Learning multi-agent behaviors from distributed and streaming demonstrations. Advances in Neural Information Processing Systems, 36:53552- 53564, 2023a.
Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep gen- erative models. In International conference on machine learning. pp. 1278-1286. PMLR, 2014.
Liu, S. and Zhu, M. Meta inverse constrained reinforce- ment learning: Convergence guarantee and generaliza- tion analysis. In The Twelfth International Conference Learning Representations, 2023b.
Rothfuss, J., Lee, D., Clavera, I., Asfour, T., and Abbeel, P. Promp: Proximal meta-policy search. In International Conference on Learning Representations, 2019.
Liu, X.-Y,, Yang, H., Gao, J., and Wang, D. Finrl: Deep reinforcement learning framework to automate trading in quantitative finance. In Proceedings of the second ACM international conference on Al in finance, pp. 1-9, 2021.
Silver, D., Huang. A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., and Lanctot, M. Mastering the game of go with deep neural networks and tree search. nature, 589 489, 2016.
Margolis, G. B., Chen, T., Paigwar, K., Fu, X., Kim, D., Bae Kim, S., and Agrawal, P. Learning jump from pixels. In Annual Conference on Robot Learning, 2021.
Margolis, G. B., Yang, G., Paigwar, K., Chen, T., and Agrawal, P. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 43(4): 572-587, 2024.
Stadie, B. C., Yang, G., Houthooft, R., Chen, X., Duan, Y., Wu, Y., Abbeel, P., and Sutskever, I. Some considerations on learning to explore via renforeeiforreementt learning. arXiv preprint / 0// 2018

---

Meta- Meta-Reinforeementt Learning with Adaptation from Human Feedback
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ inter- national conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z.., Munos, R., Blundell, C.,, Kumaran, D., and Botvinick, M. Learning to reinforcement learn. arXiv preprint arXiv:/61..0576. 2016.
Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, A survey of preference-based reinforcement learning meth- ods. Journal of Machine Learning Research, 18(136):
1 46,6,22 2017.
Xu, S. and Zhu, M. Meta value learning for fast policy- centric optimal motion planning. In Robotics science and systems, 2022.
Xu, S. and Zhu, M. Efficient gradient approximation method for constrained bilevel optimization. Proceedings of the AAAI Conference Artificial Intelligence, 32510):12509- 12517, 2023.
Xu, S. and Zhu, M. Meta-reinforcemeett learning with universal policy adaptation: Provable near-optimality under all-task optimum comparator. arXiv preprint arXiv:2410.09728. 2024.
Xu, Y., Wang. R., Yang, L., Singh, A., and Dubrawski, A. seeference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 33:18784-18798 2020.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference robot learning, pp. 1094-1 100. PMLR, 2020.
Yu, X., Dunion, M., Li, X., and Albrecht, V. Skill-aware mutual information optimisation for generalisation in re- inforcement learning. arXiv preprint arXiv:2406.0483 2024.
Zintgraf, L.,, Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hof- mann, K., and Whiteson, S. Varibad: very good method for bayes-adaptive deep rl via meta-learning. Proceedings of International Conference on Learning Representations, 2020.
12

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
Appendix for "Meta-Reinforcement Learning with Adaptation from Human Feedback via Seeeerence-Orddee-PPeevvi Task Embedding"
A. Related works
From RLHF to meta-RL, with adaptation from human feedback. RL from human feedback (RLHF) (Wirth et al., 2017; Kaufmann et al., 2023; Casper et al., 2023) has been extensively studied to align the objective of an RL task with human intentions, while the reward function of the task is not directly accessible. A mainstream of the existing methods continuously inquires humans to obtain human feedback data, e.g., the binary trajectory comparisons or trajectory rankings, and use the data to infer the reward function, then optimize the policy based on the inferred reward function (Christiano et al., 2017; Ibarz et al., 2018; Xu et al., 2020; Chakraborty et al., 2024). Several methods have been developed to improve the efficiency of data utilization, including optimizing the query selections (Biyik & Sadigh, 2018; Biyik et al., 2020; Chakraborty al., 2024), initializing with a better policy (Ibarz al., 2018; Lee et al., 2021b), and improving exploration efficiency (Liang et al., 2022). Although these methods enhance data efficiency, the number of feedback queries remains substantial, often ranging from Ik to 10k (Liang et al., 2022). This paper addresses the data inefficiency issue in RLHF through meta-RL with adaptation from human feedback. Specifically, we propose pre-training a meta-model by extracting knowledge from multiple existing RL tasks. This enables the meta-model to leverage few-shot human preference data from a new task to derive a high-quality policy. By utilizing knowledge from the pre-training tasks, the number of required feedback queries for a new task can be significantly reduced to as few as to 10.
Categorization of meta-RL. Meta-RL methods can be generally categorized into (i) optimization--ased. (ii) black-box, and (iii) context-based methods (Beck et al., 2023). (i) The optimization- based meta-RL, including MAML (Finn et al., 2017) and its variants (Stadie et al., 2018; Liu et al., 2019; Xu & Zhu, 2024; 2022), has bilevel optimization (Ji et al., 2021; Xu & Zhu, 2023) structure. In the lower-level optimization, the task-specific policy is adapted from a shared meta-policy using a policy optimization algorithm (e.g.. one-step policy gradient in MAML). In the upper-level optimization, the meta-policy is optimized to maximize the performance of the adapted policy starting from the meta- meta-policy over training tasks. (ii) Black-box meta-RI. (Duan et al., 2017; Wang et al., 2016) learn an end-to-end neural network model. The model has fixed parameters during the policy adaptation, and takes the history samples of the task as the input and the task-specific policy as the output. (iii) Context- context-based meta-RL (Rakelly et al., 2019; Raileanu et al., 2020; Zintgraf et al., 2020) encode the samples of the tasks to the task embedding vector and decode the embedding vector to the task-specific policy. In RL with adaptation from human feedback, the reward is accessbsle However, it is indispensable in policy optimization algorithms and also provides important information within the history samples of a task. Therefore, the optimization based and black-box methods cannot be directly applied in this setting. In contrast, in context- basee meta-RL, although the encoding process disrupted due to missing reward signals, the task embedding vector can be inferred from human preference data. Both ANOLE (Ren et al., 2022) and this paper belong this category. However, ANOLE borrows the task embedding encoder from an existing context-based meta-RI. model, and the task embedding space is not designed for preference data. In this paper, we derive preference-ordering. prreringggeeeiinnggg embedding space mapped from the task space, which is more suitable for task embedding inference from human preference data.
Comparisons of methods for meta-RL, with adaptation from human feedback. Paper (Joey Hejna, 2023) applies a supervised meta-learning approach, MAML (Finn et al., 2017), to train a meta-reward model, and adapt it to the task-specific reward model with few-shot human preference data during the meta-test. However, the method still needs to solve the RL problem obtain the task task-specific policy under the learned task-specific reward model, which requires large amount of data from environmental exploration. To address scenarios of the meta-RL where both human feedback data and environment exploration data are few-shot, ANOLE (Ren et al., 2022) infers the embedding vector of the task from the human preference data and derives the task-specitt policy from a well-trained context-based meta-RL model. In particular, ANOLE first employs a context- hased me model trained by PEARL (Rakelly et al., 2019), which provides an encoder that maps the task space to a task embedding vector space and a decoder that maps the embedding vectors to the task-specific policies. Next, ANOLE trains a conditional reward model that takes the task embedding vector as the input and the task-specific reward as the output. During the meta-test, ANOLE matches the human preference queries and the conditional reward model to infer the task embedding vector, and then decodes the vector to obtain the task-specific policy. An issue of ANOLE is that the meta-training and meta-test data for the task embedding decoder are inconsistent. Specifically, During the meta-training of PEARL,, task-specific samples with reward signals are used to encode tasks to embedding vectors. However, human preference data is used to infer the embeddings during the meta-test, but is not involved during the meta-training, As result, the task embedding space is not specifically designed for preference data, which may lead to sub-optimal performance. In
13

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
contrast, the proposed method, POEM, trains a preference-ordering-reeee prestrring task embedding encoder, which establishes connection between the task embeddings and preference data. During the meta-test, the few-shot preference comparisons the new task can be directly used to obtain the task embedding.
B. Analysis and Proof
B.1. Selection of similarity metric
B.1.1. INEFFECTIVENESS OF THE SIMPLE SIMILARITY METRIC
As claimed in Section 4, the similarity metric S(=71 ) - (=77,,77) is not valid for the preference-orddeiing-eeeeenr preserving embedding space. The formal statement is shown in Proposition 1.
There exists task space T such that, for any task encoder mapping there exist tasks To, To and T2 I along their embeddings zTo =(To), æTi = =(TT), and Ta f(22) with that both (1)
Proposition 1. Consider S(=71, 22 ,
and Property I are violated, i... the following property is violated:
, 22 2 5(220,277
T To (
>To TTo (
Proof. Assume that given any task space r such that, for any task encoder mapping f, for any tasks To, Tr, and To T along their embeddings To f(To), zh f(T),, and z = f(T2) where the following property is satisfied:
(f(T)): ((T)) 2 (f(T)).
To (Tt,)) To T To (TTT.)
Consider the task distribution that, for any tasks To. Tı, and Tz E with that (i) the initial states of To. Tı, and Ta is fixed, (ii) the optimal policy ... के. T.. for To. Ti and T2 I are deterministic. Then, have
( 2 (f(To). = (T 2 JTa (TTZ) for any To Tı, and To r.
Also, we have
(U(TO), ( ((T2)) < (To())))) S JTo (T7,) for any To, Tí, and To P.
They imply that
(J(Ta), = (U()))
= ( = JTo (T77) for any To, Tı, and To
tasks Tí and Ty have that the reward function rT! F rT; but कंं. = -.द.- Then, we have
JT JTO (mंz) for any To T.
Then,
(TTo). = (f(To), f(T)) for any To E P.
Consider task To Ti, then = (f then we have f(Ti') = f(T2').
Next, consider the optimal policies of tasks To and To are evaluated under tasks Tí and Ty, we have
(f(T)) 2 (f(T)), = (TT.) JT. (TT)) for any To. Ty E T.
(((T)), 2 (I(T)) = ()5( 2 Jog (TTa) for any To, To e.
Since =(Ti') = ((T2), we have
(((T)) 2 (J(TO). = ((TO).((T)) > ((TT)).((T))
Then, have
(T JTq 2 (Tr)) = (Tا())))) 2 (TT for any To. To r.
However, when the reward function !T: * TTz. we can easily to find tasks To and Tg such that the above statement is violated.
14

---

Meta- Rentoreiiffrremment Learning with Adaptation from Human Feedback
B. 1.2. EFFECTIVENESS OF THE PROPOSED SIMILARITY METRIC In this subsection, we prove Theorem I.
Proof. Consider the mapping J. with that
(x(T) = (17((1 (771, "r (11
and the mapping fx with that
(T) (1)))))
(02)))()))))) (1) (4
Then, we have that, for any tasks To and Ti, we have
(e(()) JTo (mT7))
Then we have
(2 T (2 -
= ( 2 (TT2)
where 277h = (Ti and 2 = (T) for = 0, and 2.
B.2. Performance guarantee
We first show and justify the complete assumptions for Theorem 2.
Assumption 1. Assume that the following networks applied in Algorithm 2.
(i) The task encoder f [ - Rd holds that, for T ~ TP(T), 2 = ((T follows the normal distribution N(0, I).
-
(ii) The following properties hold for the mappings J. and RRd Rd
(ii.a When 2द = =f{==" and 2 fn (= Property hold for
(ii.b) The mapping fo is a bijection from IRd to RR,, and fr is -ipisccciizz continuous.
(ii.c) The mapping fx is La Lipschitt continuous.
(ii)) the policy network with 0z is optimal, ie., To- (+127) n...
Assumption Assume the preference oracle on task Tnew used in Algorithm 2 holds the error at most e, i.e., the condition in (9) satisfied.
In Assumption 1, we assume the encoder-decoder network shown in Figure 3 is well-trained. Specifically, assumption (i) states that the posterior distribution is a normal distribution, which is enforced by the KI. divergence loss DD PxL; assumption (ii) requires that Property 1 holds, which is encouraged by the preference loss Lprei and assumption (iii) ensures that the optimal policy is accurately reconstructed, as enforced by the policy loss E, Moreover, require the continuity property for the mappings Jr and f=. Note that, for different tasks, the reward functions are different, and vice versa. So, well-trained mapping fr is a bijection, and then f. exists. In Assumption 1, we assume the maximum error of the preference oracle on task Tnew is e.
Next, we provide the complete statement of Theorem 2, which is shown in Theorem 3.
Theorem 3. Suppose Assumptions and hold for Algorithm 2. Let Zx be the random variable sampled as same as 2μ in Algorithm 2. Let Z be a random variable following. (0, Under mild conditions on fr and fm. we have
(a) The probubility density function (PDF) of Zu at z Taow has
P(Z [Tew))
P(Zu =new) VEL, log VT
142c 1-2c
(11
VDLr log
1+2c
1-2c
(b) When 0, Zy converges to Tnew in distribution, i.e., Za - Tnew Here, 2 TTew is the embedding Tneso such that JT.ee (TO, (-12Tnow)) J (T
15

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
Proof (a). Consider the space of define by:
exp exp (S(2,2)))) exp (x(, + exp (2 z"))
1
2
the space of can be simplified as
1 2E
S (12,z') - -5(,,=") 2 log
1--2 2e
Consider the distance metric is 5(21,²") = (2/, 2"",, we can derive the space for the task reward embedding 2, as a half-space specified by the following linear inequality:
1 + 2e
(Er, 2 2x4 2 log
(11)
1- 2c
We denote the half-space specified in (1I) with 24. = and as H (z t).
Let Z, (Z) where Z is random variable following /O I). Define the function g( = Pr(z, e H = 0)]. Since f.1 is Lipschitz continuous and the probability density of Z on any point is finite, then the probability density of Zr on any point is finite, then 0('x2, 2=) is continuous.
Next, we have g = = - 24), 0 S - S 1, and 0 S g(=,, S 1. Without loss of generality, we
assume that ( x²) 2 2 and g( 2 S 1. From the mean value theorem, since g(2n, =²) is continuous, there exists a
pair (z,, > such that g x))
and g (2" )=== = 3. We choose (z/ = to generate the half-space H # c). and
aim derive the upper bound for Pr(Z, H H - )) and Pr[Z, { ")].
Since the mapping f. is Lipsschit continuous and Z is random variable following (O, I), we have
Lr
1 2c
Pr[Z, H H(,, 2 e) H 2" 0)] S
log
Vn
1- 2E
Then,
1
Ly log
1 2c - 2e
Pr|Z, H( 2", c)
+
2
2t
Similarly, we have
L log
1 1 2c
Pr|Z, E H(= 2z c) -
+
2
2m
Therefore,
by log
1 2e
max(PrrZ, E H( = c). Pr E HH ",,,,
0]}
S 2
22T
1 2c
In Algorithm 2, the pair (²) " generated by the following optimization problem,
(E', = arg min (max (max([
where 2(1) (= E Zz= S(2,²2) >, S(z, and 2 {= E Ze S(2, =") >. S(2,z')) This means that (2',2") = arg (max (== 24 e) PriZ, E
Therefore, we have
1 Lr
1 1 2c
max H( Pr,, H(* 2/ e]]}
log 1 2e
2
22
Therefore, after we generate the pair ("', ) for one time to sample 21, there is at least probability of 1 eliminated. From Assumption 2, the z,Tae will be included in (2/, 24/ e) when é' is preferred and included in H (2 En c) when g" is preferred. Therefore, 2.new will not be eliminated.
بr log /2न
1+2c
1-
16

---

Meta-Reinforeementt Learning with Adaptation from Human Feedback
Denote the left probability as Pk. then Po = 1 and P1
S 2
+
L-
log
1+2c 1-2c
Repeat the above procedure to sample 22.
V25
Zz.
Zk. we have
1 Ps - Ps
Lr log
[1+ 2E
+
2
1 2
Then, we have
V2Lr Tx
1 2c
V2L, log
1 2e 1 26
Pk S
1
log
1 2e
2
Then,
P(Z zJnew)
P(Zk = =Nuww)
2
(1-
V
عااو log
1-2c
1+2c (2) (4)*
)
+
12h
/x
log
1+2c 1-2e
Proof of (b). Consider O. Consider any embedding 26 such that 22 o xTnow i.e, Tnow (To (-12g)) Jnew (T))
Assume that the angle between the vectors z/7 and 2, is 8, then 5 > 0.
Consider that the pair (3', (") generated in Algorithm 2, the probability of generating (z', such that
(2
Tnes
O and (zf, - > 00
is non-zero, until = is eliminated from the candidate space. Therefore, we have that, if 26 + Tnem then
lim P P(Z = =0) 0. 10000
Therefore, for any closed set S, if 2,new E S, lim P P(Z E S) = 1; else lime- P(z E S) 0. Then, Ze converges to 2 Thew in distribution, i.c., Zn 4 2 Taee
C. Experimental Supplements
All experiments executed on computer with a 5.20 GHz Intel Core i12 CPU and an NVIDIA RTX 4090 GPU.
C.1. Benchmarks
MuJoCo. The first benchmark is the robotic locomotion benchmarks simulated using the MuloCo simulator (Todorov et al., 2012). The benchmarks are created by (Finn et al., 2017; Rothfuss et al., 2019) and are widely adopted platform in meta-RI. (Rajeswaran et al., 2019; Raileanu et al., 2020; Xu & Zhu, 2024; Liu & Zhu, 2023b; Liu & Zhu; 2023a) and meta-RL with adaptation from human feedback (Joey Hejna, 2023; Ren et al., 2022) to test the performance of few-shot policy adaptation. Visualizations of the benchmark is shown at the top of Figure
Meta World. The second benchmark is MetaWorld (Yu et al., 2020), a benchmark suite for robotic tabletop manipulation. Meta World includes diverse of motion patterns and interactions with different objects for robotic arms and is recognized as more challenging benchmark for the few-shot policy adaptation in meta-RL. Visualizations of the benchmark is shown at the bottom of Figure 8.
C.2. Task settings
We conduct experiments totally nine scenarios in two benchmarks centered around robotic continuous control tasks.
We consider seven scenarios on the benchmark of MuJoCo. The dynamic systems of the agents in the seven scenarios include three different robots: HalfCheetah, Ant, and Humanoid. Visualizations of these agents in their respective environments are provided at the top of Figure& 8, where Half-Cheetal (Figure 8.a) has 17-dimensional state space and 6-dimensional action space; Humanoid (Figure 8.b) has a 376-dimensional observation space and a 17-dimensional action space; Ant (Figure 8.c) has 27-dimensional observation space and an 8-dimensional action space. The task settings include forward/backward
17

---

Learning with Adaptation from Human Feedback
Half Chef-Cheetah
(b) Humanoid
(c) Ant
assembly
basketboll
button press topdown
button press topdown wall
button press
button press wall
coffee button
coffee pull
coffee push
dial turn
disassemble
door close
coor
drower close
drower
foucet open
foucel close
hommer
press
press
pull
peg
unplug side
pick
side
side
hole
(d) Meta World
Figure 8. Visualization of robotic locomotion environments, including Half-Cheetah, Humanoid, and Ant, simulated by Mujoco and the MetaWorld environments.
motion (-Fud-Back), various target velocities (-Vel), various target directions (-Dir-2D), and various target positions (-Goal). We show the detailed setting of the seven scenarios as follows.
Seff-Cheetah-Ve.. In the experiment of Half-Cheetah-Ver the agent is Half-Cheetah, and the reward is the negative absolute value between the agent's 's current velocity and a goal velocity, where the goal velocity characterizes the task. The task distribution is defined by the distribution of the goal velocity, which a uniform distribution from 0.0 to 3.0.
In the experiment of Half-Cheetah-FwwdlD includes tasks. The reward in the first task is the agent's current velocity. The reward in the second task is the negative of the agent's s current velocity.
Seer-Chhetah--wwle
the agent is Half-Cheetah, and the task family
Ant-Vel Vel. In the experiment of Ant- Vel, the agent is Ant, and the reward is the negative absolute value between the agent's s current velocity and a goal velocity, where the goal velocity characterizes the task. The task distribution is defined by the distribution the goal velocity, which is a uniform distribution from 0.0 to 3.0.
Ant-Fwd-Back. the experiment of Ant-Fwd-BBade the agent Ant. and the task family includes tasks. The reward in the first task is the agent's current velocity. The reward in the second task is the negative of the agent's current velocity.
Ant-Dir-22 Dir-2iD In the experiment of Ant-Dir-2D, the agent is Ant, and the reward is set vy sin 0 + cos 0, where and by are the velocities along the z-axis and y-axis, and 0 is the walking direction of the humanoid. So the reward is the velocity along the direction The task characterized by the walking direction which is sampled uniformly from xx /2 to x x//2.
Ant-Goal. In the experiment of Ant-Goal, the agent is Ant, and the reward is set as negative absolute value between the agent's current location and goal, where the goal characterizes the task. The task distribution is defined by the distribution of the goal location, which is uniform distribution from circle with a radius of 3.0.
Humanoid-Dir-211 In the experiment of Ant-Dir-2D, the agent is Humanoid, and the reward is set as uy sin 0 + cos O, where and iú are the velocities along the z-axis and y-axis, and 0 is the walking direction of the humanoid. So the reward is the velocity along the direction 0. The task is characterized by the walking direction 0 which is sampled uniformly from - π 7/2 to x/2.
Two scenarios are conducted on MetaWorld. The scenarios in MetaWorld include task families that hold different
18

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
manipulation types (ML10) and varying operation positions (ML1). We show the detailed setting of the scenarios as follows.
MetaWorld-MLL. In the experiment of MetaWorid-MLJI the tasks family has a single type of manipulation task with varying target operation positions.
Meta ML01 In the experiment MetaWorld-ML10, the task is characterized by the type of manipulation, which shown in (d) of The tasks family has 10 types manipulation tasks where each type manipulation task has varying target operation positions. Note that the state transition functions in different types of manipulation tasks are different.
C.3. Experimental setting
We sample 120 training tasks for meta-training of scenarios including Half-Cheetah- Vel, Ant-Vel, Ant-Dir-2D, Ant-Goal, Sommanidd-Dir211 MetaWorld-Min Back and Ant-Fwd-Back. We simulate the preference oracle by comparing the ground-truth trajectory return from the environments. The adaptation cannot observe the environmental rewards during meta-test and can only query the preference oracle. At the meta-test time, the meta- meta-policy is adapted by at most 10 preference queries for each task and at most 20 trajectories without reward signals (two trajectories for single preference query). The performance of the adapted policy is tested on 30 tasks and we evaluated 10 episodes in each task. To ensure a fair comparison, we keep all the above settings the same for all methods during both the meta-training and the adaptation from human feedback.
MetaWorld-MIL and sample tasks for the meta-training Half-Cheetah Ca---Chhethh-eee
C.4. Hyper-parameters
We summarize major experiments. The hyper-parametert in Table can be divided into two parts. The first part includes the that follow the existing meta-RL (Rakelly et al., 2019) and RL (Haarnoja al., 2018) algorithms. So we directly use the hyper-parameters in these works in our proposed method. The second part is new and specific to the proposed method, including the coefficients of the three loss terms and the error tolerance constant We select the coefficients such that all these three loss terms have similar scales. For the error tolerance constant we tune to deal with the noise of the preference oracle while not deteriorating the performance of POEM when the preference oracle is accurate. We show the ablation study on the hyperparameter, the error tolerance constant 6, in Section C.7.
in the following table. We use this set of hyper-parameters for all POEM's
19

---

Meta-Reinforremment Learning with Adaptation from Human Feedback
Table Default Hyper-Parameters and Configurations
Default Configuration
Dimension of latent embedding d
5
Discount factor
0.99
Optimizer (all losses)
Adam (Kingma & Ba, 2014)
Learning rate of policy network
1 10
Learning rate of Q function
1 10-1
Learning rate of Value function
3 10
Learning rate of encoder
1 1-10-4
Adam- 3)
(0.9, 0.999, 10 *)
3r coefficient of reward loss fy
0.2
3 :rre: coefficient of preference loss Lyre
6.0
3,: coefficient of policy loss &
0.3
Error tolerance constant e
0.1
Gradient steps per environment step
1/5
# Gradient steps per target update
1
# Transitions in replay buffer (for each task T) # Tasks in each mini-batch for training SAC Transitions in each task batch for training SAC Trajectories each task of each mini-batch
le5
32
256
10
Transitions in each context # Preference queries K # Sample embeddings N₂ # Trajectory horizon
128
10
100
200
20

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
Table Comparison of average accumulated reward + standard deviation between POEM and baseline methods in the adaptation from human feedback (after querying 1, 5, and 10 times preference oracle, respectively) on MuJoCo and MetaWorid.
Hal/Cheetah- Vel
HallCheetah--Fwd-BBee
1 query
5 queries
10 queries
query
5 queries
10 queries
MAML-reward
-201+13
-193+11
1182=11
-300+110
-40 138
-41142
ANOLE
-173:112
-1201110
-112:114
503=149
1012:1121
1198:189
ANOLE-random
-172:114
-125113
-115113
12361:203
14191108
1483:145
ANOLE-greedy
-169+12
-128+18
-115#12
1367=155
1482++91
1493+89
POEM
-65110
-5018
-6616
1689_54
16841332
1709123
Ant-Goal
Anl-Fwd-Back
query
queries
10 queries
query
5 queries
10 queries
MeMa-reward
-610+88
-603+88
-602-899
101+11
105+
105=111
ANOLE
-550121
-405129
1980119
618160
830.100
802.138
ANOLE-random
-551+22
2419+26
16
808
810#222
812+299
ANOLE- greedy
-560:25
-412:338
-401=51
788+339
806:+23
820==32
POEM
-351119
-203:112
-202.17
1030
1013156
1015
Anl-Dir-2D
Ant- Vel
query
queries
10 queries
query
queries
10 queries
Meta-reward
102+:20
101+23
112+:30
-3199
-1+10
6=88
ANOLE
308.145
515178
588-167
36.15
80::10
83:17
ANOLE-random
181=300
432++45
498=335
20+5
60+:10
100+:6
ANOLE-greeed
172.120
4431:30
414.225
30.55
70.110
11045
POEM
671=15
732+:45
749=44
95::
108+8
11244
Humanoid-Dir-2)
MctaWorld-MLL1
query
queries
10 queries
query
queries
10 queries
MAML-reward
504=:52
5294:56
523=588
153+18
164+119
166=15
ANOLE
754-:59
7034:55
722-662
2384:31
192-1:22
240+:21
ANOLE-randon
232-+54
780:164
796=
242+:31
192#:39
229+229
ANOLE-greedd
231-:30
7814:59
798-1:54
2394:34
1754:26
183
POEM
812+445
1462#114
1451=67
326117
324::13
34947
MetaWorld-MIL
1 query
5 queries
10 queries
MAML-reward
680+-36
691+-40
692-+49
ANOLE
1086-1167
1042+70
1018+82
ANOLE-randone
1117-644
1045+:80
1145+:52
ANOL.F-greedy
1091+:47
1011777
1122-776
POEM
1435+112
1588132
1601+13
21

---

Meta- RectaReeinforcement Learning with Adaptation from Human Feedback
POEM Meta-Wootd-ML (Mala-training
ANCLE
ANCLE-random
ANOLE-greody MAMil-roward
Mett-Worid-MI 1/Adaptation from human feedback
Average accumulated rewaad
1800
Average jocc/multteed reward
1600
-400
1000
1000
800
000
800
10
Mets-training time-steps (1e6)
Number of proference queries
Figure 9. Performance on MetaWolrd-ML11 Left: Average accumulated reward on test tasks v.s. samples collected during meta-training. Right: Average accumulated reward on tasks v.s. preference queries during adaptation from human feedhack. "Oracle" denotes the performance when the reward signals are available during the meta-test.
POEM (ours)
ANOLE
ANCLE-random
ANOLE-greedy
MAMA-rrward
Oracie
HrlChhcrahh
Oool
Average 8220 m 006e WeRERE
200
+000
0/00
600
02 D0
70
25
Art.Ped BockiSdusttion foom human foodbook
Ant Dil 2DWdde. frrm heman bodtbok
Art Coal 2DWduasrinn from humen feeebook
Averagg: pow.mm/stdd
400
Aoorage
preferrnnc queriss
quatics
Figure 10. Performance HallCheetah-Fwwd-BBe Ant-Fwd-Back, Ant-Dir-2D, and Ant-Goal. Top: Average accumulated reward
test tasks v.s. samples collected during meta-training. Bottom: Average accumulated reward on test tasks v.s. preference queries during
adaptation from human feedback. "Oracle" denotes the performance when the reward signals are available during the meta-test.
C.5. Supplemental results for performance evaluation
Figures 9 and 10 show the performance evaluation of POEM and baseline methods with an accurate preference oracle on MetaWolrd-Mil HalfCheetah- Fwd- BBckk, Ont-Fdd-Back Ant-Dir-2D, and Ant-Vel. It is shown that the proposed method, POEM, significantly outperforms all the baseline methods during both the meta-training and the meta-test.
We further compare the proposed method POEM with the baseline methods in Table It is shown that the proposed method, POEM, significantly outperforms all the baseline methods. Moreover, single query for the adaptation from human feedback can achieve the best performance on HalfCheetah- Fwd-hBac- and Ant-Fwd-BBan Fadk-ackk, because these two scenarios are simple and each scenario only includes two tasks, and single query is sufficient to distinguish the task embedding of a given new task. In the other seven scenarios, queries for the adaptation from human feedback almost achieve the best performance. Moreover, in these harder scenarios, the proposed method, POEM, exhibits greater performance advantages over baseline methods compared to that in the two simpler scenarios.
22

---

Meta-Reinforeemennt Learning with Adaptation from Human Feedback
C.6. Supplemental results under noisy preference oracle
We evaluate POEM when the preference oracle holds noises during the adaptation from human feedback. We use the Boltzmann model model the human preference errors. In partieular, the oracle answers 7(1) 7(2) with the probability
(B B. (10)
exp (3 Return (r(1))
exp
exp (B Return (r())) exp (B Return (7(2)))
(1) [ exp r(
exp
where denotes the temperature parameter. This error mode is commonly considered by recent preference-based RL works (Ren et al., 2022; Lee al., 2021a). To normalize the total reward, we select the temperature parameter
E. EAEPHE,,,
Figure 11 illustrates the performance of POEM under noisy preference oracle MetaWolrd-MLI, HalfCheetah-FFw--Bee hat-Fwd-Backe Ant-Dir-2D, and Vnt-Vel. The results indicate that POEM achieves performance comparable to that observed with an accurate preference oracle, as shown in Figures 9 and 10, highlighting its robustness to preference errors. Notably, while (Ren et al., 2022) has demonstrated ANOLE's strong performance in handling preference noises, the proposed method significantly outperforms ANOLE these scenarios.
POEM (ours)
ANGLE
ANCLE-random
ANOLE INOLE-graedy
MAWa-deward
Oracie
HalfCheetthh-Fii Fard Back/Weta-rrmiig
Art-Fud- training
Gaal-Gal-- Meit traniigninnng
Averaga ecoumuiited rewaard
1000
Averge acc.amu bsted recoerd
3200
1900
1250
1000
600
R30
-400
000
280
20
Mala-triinin limesttpps (106)
Mela-triining Binesdhapa (166)
Mata-tishhnng Sma-stops (108)
Mett-Worid-ke ML LiMets-triiing
1.01
1600
700
400
3003
1000
200
800
0.
0.5
3.0 Meta training tio-staps (1e5]
1.0
2.0
25
3.5
05
02
03
0.4
05
38
077
03
0.5
Meta-trahing lime-sopp (106)
Figure II. Performance evaluation under a noisy preference oracle. Average accumulated reward test tasks samples collected during meta-training on HalfCheetah- -wad---Fddd.--..------- BBck, Ant-Fwd-Baan Backk Ant-Goal, Ant-Dir-2D, and Meta 1..ld-..
C.7. Supplemental results for ablation studies
We conduct ablation studies to test the impact of the individual design components in POEM, including (i) two-encoder structure, i.e., the policy/reward embedding encoders (Section 5.1) and (ii) context design (Section 5.2).
Two-encoder structure with the new distance metric. We first consider the impact of the two-encoder structure. In Section 5.1 and Figure 3, we employ embedding encoder to map the task context cd to the task embedding =T, and then employ two mappings the task embedding 2" to the task-specific policy embedding = (by føz) and task-specific reward embedding 2! (by ). Moreover, we consider the distance metric S A ;,?), where =T fe. (2T) and 2, (zT). Here, we consider a naive option to define the task embedding and the distance metric, i.c. employing a single embedding encoder to map the task context cT to the task embedding 2T and then define the distance metric S(= (T & (2Th, TTT). In Proposition 1 of Appendix 1.1.1, we theoretically show the ineffectiveness of the naive distance metric. Here, we experimentally compare the performance between the two designs.
23

---

Meta-Reinforeementt Learning with Adaptation from Human Feedback
Figure 12 shows the performance comparison between the two-encoder structure with the distance metric in 3 designed for POEM and the single encoder with naive distance metric. It is shown that using the single encoder with naive distance metric hurts performance.
Art-Dir- toMiii
Art-Gool 2DiMets training
Humanoic- Oir Mii---
-150
100
Two-ensoder snadare IPEM (ours)
Average 3ocamn loee revarrd
-200
Encodo stacare PEAR.
710)
-200
800
800
400
-100
800
-00
-20
Two encoder duckure in POEM (ears
Two-onooder stucture in POEM you
200
Er Eroudor saxtara in PEARL
-500
Brooder stuckus in PEARL
05
0.5
1.0
15
2.
24
30
35
40
23
15
1.
25
25
30 35
4.
0.5
10
Meta training time-stegs| (1e6)
Metartraning simesteps (1c6)
Meta traintrainnn time steps (te6)
Figure 12. Ablation studies on the encoder structures in Ant-Dir-2D, Ant-Goal, and Humanoid- Dir. Average accumulated reward on test
tasks v.s. samples collected during meta-training.
Context design and recursive training. We examine our choice of context, i.e., the input of the encoder. As introduced in Section 5.2, we consider the task context as cT {(s(t) a(tt), ,((++)), ,(t)),((++)) , .r4), Q7 (s((), ) where a(t) ~ TT (-Is(t) and 27 (s(*), a(t)) is the optimal value on (s(*), ) As discussed in Section 5.3, since the optimal policy क is not available during the meta-training, we approximate the optimal policy क. by the policy aain--(-,- x I=.(n-1)), where both the policy network TØi= 1) and the embedding 2T=((n-1) are obtained from the last epoch n - 1. Then, we use the recursive training to update the parameters of the policy network d!d) and the embedding z =Ta.(n)) in each epoch n.
In PEARL (Rakelly et al., 2019), the context is c {(s((), g(t+1), .,/())// sampling. This enables the trajectory sampling for the new task during the meta-test. However, in the meta-RL with adaptation from human feedback, the trajectory with reward signals is not available for the new task, and the proposed method does not need to sample the trajectory for the context. Therefore, we do not need to use the posterior sampling in PEARL. Instead, we design the context which incorporates more information about the optimal policy in ( and Q
,(t+1),,,()) JE= where (1)) is sampled by posterior
We experimentally verify the benefits of the context design in POEM over that in PEARL. Figure 12 shows the performance comparison between the context design in POEM and that designed for PEARL. It is shown that using the context designed for PEARL holds certain performance loss. This could be attributed to the fact that the optimal policy-related information embedded within the designed context aids the encoder in learning more informative and effective task embeddings.
Ant-Goal 2D/Mtt
Humanoic Cin wwt.eta---aannteee
500
Liiing contedt PIEM (oun))
-000
Using camred in FOEM comnet PEARI
PEARL
Avarage socumalated reward
7000
500
500
4003
-400
700
3000
2000
Awerage
200
550
100
Meta training limesttos (1e58
Metairaanin time steps (1c6)
Meta training (1e6)
Figure 13. Ablation studies on the context design in Ant-Dir-2D, Ant-Goal, and Humanoid- Dir, Average accumulated reward on test tasks v.s. samples collected during meta-training.
Ablation study selecting the error tolerance constant c. We conduct the ablation study on the hyperparameter, the error tolerance constant under the preference oracle and the preference oracle with noise set Appendix C.6. Figure 14 shows the results of different eтror tolerance constants e in the adaptation from human feedback (meta-test) under the preference oracle. Figure 15 shows the results of different error tolerance constants e in the adaptation from human feedback (meta-test) under the noisy preference oracle.
24

---

Meta-Reinforeement Learning with Adaptation from Human Feedback
Humanoic-DD- Oir-20 Di-pp/Aaapiiiiin from huren feedhock
from humen fewdbockk
Humanoid- Distiondeppaationn from human feedback
Average acccmmuuaa Santdddddddd roward
2000
PG2EM
-003
FEEM
Averoge
POEM
POEM =
FOEM c
POEM, co1.1
POEM 02
POBM c
POEM C=02
POEM c- 0.3
FOBM C=
POEM. C-0.3
Number preference queries
Number of preference queries
Number of preference queries
Figure 14, Ablation studies on selecting the error tolerance constant c, in Ant-Dir-2D Ant- Goal, and Humanoid Dir. Average accumulated reward test tasks v.s. samples collected during meta-train veta-training. The human feedback is the oracle.
HAmanoi--D--2Dlddpppie from human feadback
VVl-Goal-2D/Addppleiin from human feedback
Humanois- Humannié-Di-2DAa Di-DDiiddiiiiiiin from human feadback
302
roward
Average accumuatedd reward
800
Auerage accumretee
202
700
-500
Averiigo
POSM
660
POEM 0-2
FOBM 0
POEM, 0=0
300
POEM 0.3
FOBM C
500
POEM, C=0.3
Number of proferonco quorles
Number of preference quaries
Number of proferance querics
Figure 15. Ablation studies on selecting the error tolerance constant e, Ant-Dir-2D, Ant-Goal, and Humanoid-Dir Average accumulated reward test tasks v.s. samples collected during meta-training. The human feedback is noisy oracle.
25


Reviews:
[{"review_id": "fe9T4IeZur", "reviewer": "Reviewer_Pj9r", "summary": "The authors present the adaption via Preference-Order-preserving EMbedding (POEM) framework. Their key insights that are if the trajectory of a task is distilled into an embedding, the similarities between tasks should be evident in these embeddings and that the optimal policy on one task should do sufficiently well on another if there is sufficient enough similarity. They leverage these properties to create an algorithm that allows a human in the loop to do a preference-based selection between two policies to progressively move closer to the already-known policy that works best for the new task. This is essential due to the new task not providing the model with an environmental reward. They further include a relaxation on their initial insight (trajectory similarity being captured in embeddings) to account for human error in the preference selection.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful and indebted for the time and effort invested in evaluating our manuscript and for all the suggestions to make our manuscript a better and stronger contribution.\n\n>**Methods And Evaluation Criteria 1: I could see issues with Property 1 being called into question in scenarios where similarity between task embeddings may not be sufficient to ensure policy transferability, such as long-horizon dependencies, however I believe the relaxation in equation (8), while explained to be used to account for noise in human preference selection, should also clarify this.**\n\n**Answer:** In this manuscript, we do **not** claim that, Property 1 always holds, and the similarity between task embeddings is sufficient to ensure the policy preference, for any encoder. \n\nInstead, we claim and prove that in Section 3, under the MDP (including the case of long-horizon dependencies), **there exists** a task encoder such that Property 1 holds, i.e., there exists an encoder such that the similarity ordering of task embedding pairs is expected to align with human preference order. Next, in Section 4, we train an encoder that enforces that Property 1 holds under the encoder.\nTo achieve this, we design a loss term for the encoder network, the preference loss term (equation (6) in line 241), which penalizes violations of Property 1. \nTherefore, although Property 1 may not be naturally satisfied without the training, once the encoder is trained under the supervision of the preference loss term, Property 1 holds approximately. \n\nFurthermore, as you kindly pointed out, we also incorporate the relaxation in Equation (8) to account for noise in Property 1, enhancing the algorithm’s robustness.\n\n>**Methods And Evaluation Criteria 2: I would appreciate more detail about how the human-in-the-loop was given the trajectories**\n\n**Answer:** During the meta-test (human-in-the-loop adaptation), a new task ${\\mathcal{T}}_{new}$ is given. The agent explores the environment and provides two trajectories (the rewards along the trajectories are unknown) to the human. Then, the human will tell the agent which one is better, and the agent will adapt the policy according to this human feedback.\nOne time of the human preference query is denoted as one iteration of the human-in-the-loop adaptation.\nThe detail of the human feedback is introduced in Section 2, lines 95-109.\n\n>**Experimental Designs or Analyses: The primary concern is not understanding how these environments were modified.**\n\n>**Question 2: Could the authors please explain the difference between the base Mujoco and their modified version?**\n\n**Answer:** Thanks for pointing out the confusion. In this paper, we do not modify the environment of Mujoco. Instead, we directly use the environment in Mujoco and design the reward functions for multiple tasks for the meta-RL setting. \nThe details of the reward design are shown in Appendix C.2.\nTo avoid the confusion, we will modify the \"Modified Mujoco\" to \"Mujoco\".\n\n>**Suggestions: Despite the technical strength of this paper, I take a very strong issue with the related work being relegated to the appendix. Even if the authors only use half of a column to address it in the main paper, I believe it is critically important for previous and related work to be given its proper acknowledgment by the scientific community. This is the primary motivating factor behind my score.**\n\n**Answer:** Thanks for pointing it out. In the modified version, we will move the related work section in Appendix A (lines 608 to 662) to the main body of the paper. \n\nIn the related work section, we comprehensively review the works related to (i) meta-RL methods, (ii) RL from human feedback (RLHF), and (iii) methods for meta-RL with human-in-the-loop adaptation, and provide details of the comparisons between the problem settings, the problem formulations, and the algorithm designs in this paper and those in the existing works. Specifically, in (i), we discuss three categories of meta-RL methods and whether they can be applied to the problem of meta-RL with human-in-the-loop adaptation. In (ii), we discuss the existing methods for RLHF and the motivation of studying meta-RL with human-in-the-loop adaptation based on RLHF. In (iii), we discuss the existing methods for meta-RL with human-in-the-loop adaptation, including ANONLE and meta-reward, and discuss the differences between their algorithm designs and this paper.\n\n>**Question 1: Could the authors explain how the policies were presented to the human-in-the-loop during evaluation? How many iterations of this were required per task?**\n\n**Answer:** Please refer to the answer to **Methods And Evaluation Criteria 2** for the explanation of the human-in-the-loop adaptation.\n\nAs shown in Figures 5 and 10, we conduct at most 10 iterations for human-in-the-loop adaptation for each task, and it usually requires about 5 iterations to achieve the near-optimal performance per task.", "claims_and_evidence": "The authors claim a three-fold contribution in being the \"first to propose the preference-order-preserving task encoder for context-based meta-RL training, which establishes a connection between task embeddings and human preferences\"; experiments with their new framework conducted in a modified Mujoco and MetaWorld; and a proof for the theoretical result for their algorithm which guarantees convergence to the optimal task-specific policy. \n\nNo claims are problematic aside from the second, primarily because I am unclear how the environments were modified despite section D in the appendix.", "methods_and_evaluation_criteria": "The proposed methods follow clearly from Insights 1 and 2. I could see issues with Property 1 being called into question in scenarios where similarity between task embeddings may not be sufficient to ensure policy transferability, such as long-horizon dependencies, however I believe the relaxation in equation (8), while explained to be used to account for noise in human preference selection, should also clarify this. The environments also are well-suited to testing the proposed algorithm, though I would appreciate more detail about how the human-in-the-loop was given the trajectories (ie, text description, final reward, video?)", "theoretical_claims": "I have read through and do not see any issue with the proofs provided in Appendix B.", "experimental_designs_or_analyses": "I've read through the task descriptions of how the authors evaluated their methods in the corresponding environments and do not see any issue with these tasks, nor do I feel the environment to be invalid for the proposed algorithm. The primary concern is not understanding how these environments were modified. I did not see this explicitly outlined in D1 or D.2.", "relation_to_broader_scientific_literature": "The key contributions of this paper are very related to the fields of RL, specifically meta-RL, continuous control and preference ordering. I believe these contributions to be relevant to the field of Machine Learning as a whole.", "essential_references_not_discussed": "I am not aware of any essential references not discussed.", "other_strengths_and_weaknesses": "This is a very strong paper. Technically, outside of lacking a few details that I may have just missed, I cannot find any fault with the paper. The proposed method is very clearly motivated and the only issues I could think of regarding their methods were already addressed. They show strong results in their evaluation and justify their approach with a rigorous proof.", "other_comments_or_suggestions": "Despite the technical strength of this paper, I take very strong issue with the related work being relegated to the appendix. Even if the authors only use half of a column to address it in the main paper, I believe it is critically important for previous and related work to be given its proper acknowledgement by the scientific community. This is the primary motivating factor behind my score.", "questions_for_authors": "1.) Could the authors explain how the policies were presented to the human-in-the-loop during evaluation? How many iterations of this were required per task?\n\n2.) Could the authors please explain the difference between the base Mujoco and their modified version?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qsiu7Hpdii", "reviewer": "Reviewer_d1QQ", "summary": "The paper presents a framework for meta-reinforcement learning (meta-RL) called Preference-Order-preserving Embedding (POEM), which aims to facilitate few-shot adaptation to new tasks using human preference queries instead of traditional reward signals. The framework comprises a task encoder that maps tasks into a preference-order-preserving embedding space and a decoder that generates task-specific policies from these embeddings. During the adaptation process, the encoder efficiently infers embeddings for new tasks based on human preferences, ensuring that task embeddings reflect similarity in task performance. The authors provide theoretical guarantees for the convergence of the adaptation process to optimal task-specific policies and demonstrate through experiments that POEM significantly outperforms existing methods, achieving a 20%-50% improvement in performance on various continuous control tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Claims and Evidence: The approach is quite similar to the work proposed by ANOLE.**\n\n**Answer:** The partitioning of the embedding into reward embedding and policy embedding spaces is only an initial and minor design of the paper. The main contribution of the proposed method is that we train a preference-order-preserving task encoder, which establishes a connection between task embeddings and human preferences. This connection facilitates the efficient inference of task embeddings for new tasks during human-in-the-loop adaptation.\n\nIn terms of the algorithm design, as you pointed out, this paper first designs an encoder that partitions the embedding space into reward embedding space and policy embedding space. Second, based on the embedding space, we prove that an encoder exists under the embedding space partition, which holds the preference-order-preserving property (Property 1 in Section 4). Third, in Section 5, we train an encoder that enforces that Property 1 holds by using the preference loss term (equation (6) in line 241), which penalizes violations of Property 1. Fourth, in Section 6, the preference-order-preserving property of the encoder enables the task embedding inference from human preferences for the human-in-the-loop adaptation.\n\nNote that all the above four steps of the algorithm design are different from ANOLE, which addresses the issue of ANOLE that the task encoder cannot capture preference-related features across tasks. \n\n>**Theoretical Claims: The definition of \"well-trained\" is unclear. The policy reconstruction loss is not used. How to ensure the encoder-decoder is properly trained?**\n\n**Answer:** The definition of \"well-trained\" is given by the three assumptions of Theorem 1. Specifically, the assumption (i) (lines 375-377, Theorem 1) states that the posterior distribution is the normal distribution; the assumption (ii) (lines 377-379) requires that Property 1 holds, i.e., the preference-order-preserving property of the encoder holds; and the assumption (iii) (lines 379-381) requires that the optimal policy is accurately reconstructed. \n\nTo achieve the above three requirements, i.e., the encoder-decoder network is well-trained, we use the KL divergence loss term in line 273 to achieve assumption (i), use the preference loss in Equation (6) (line 240) to achieve assumption (ii), use the optimal reconstruction loss in Equation (5) (line 235) to achieve assumption (iii). Therefore, including the policy reconstruction loss, all the losses included in Section 4 are used to support Theorem 2.\n\n>**Experimental Designs or Analyses a)**\n\n**Answer:** In the conducted experiment, MetaWorld-ML10, the tasks family has multiple types of manipulation tasks. As shown in Figure 8 (page 17), the interactions between the robot and the environment are different for different types of manipulation tasks, and therefore their state transition functions are different and change. For example, it is easy to see that the state transition in the \"assembly\" task and that in the \"basketball\" task are different. As shown in Figure 5, the proposed method achieves 50\\% improvement over baselines on MetaWorld-ML10. \n\n>**Experimental Designs or Analyses b)**\n\n**Answer:** There are several works, such as [1,2], addressing the issue of imbalance in task distribution in meta-learning/meta-RL. Their approaches are agnostic to the meta-RL methods and can also be used to address the issue in this paper. However, this paper primarily focuses on a new problem of meat-RL with human-in-the-loop adaptation. Addressing the issue of imbalance in task distribution is an interesting future work.\n\n[1] \"Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks\", 2020.\n\n[2] \"Improving Generalization of Meta Reinforcement Learning via Explanation\", 2024.\n\n>**Weakness 1**\n\n**Answer:** The numbers of steps between this paper and ANOLE are exactly the same for the ant task and the cheetah task. In both this paper and ANOLE (Ren et al., 2022), cheetah-vel uses 1M steps (check the first figure in Figure 5 in this paper and the second figure of Figure 1 in ANOLE paper), cheetah-fwd-back uses 2M steps (check the first figure in Figure 9 in this paper and the first figure of Figure 1 in ANOLE paper). In this paper, we also use 4M steps on the ant task (check the second and third figures in Figure 9 in this paper).\n\n>**Weakness 3**\n\n**Answer:** Meta-RL with human-in-the-loop adaptation is to train a meta-model from the training tasks (T1, T2, T3), such that it can be adapted to new tasks with limited human preference queries. The learned meta-model is fixed during the meta-test phase. When a new task T4 arises, the meta-model learned from (T1, T2, T3) will be adapted to solve T4. Similarly, when a new task T5 arises, the meta-model will be adapted to solve T5. If any of the training tasks T1, T2, T3 arises in the meta-test, the meta-model is adapted to solve the task, which is easier than solving a new task T4.", "claims_and_evidence": "No. The approach is quite similar to the work proposed by ANOLE. The only difference is the partitioning of the embedding into reward and policy spaces, which seems like a relatively incremental improvement.", "methods_and_evaluation_criteria": "Yes . The method tries to solve very relevant problem in RL.", "theoretical_claims": "Yes. However, Theorem 2 requires the encoder-decoder network to be well-trained. However, the definition of \"well-trained\" is unclear, as the policy reconstruction loss is not used initially, and it is not mentioned when it starts being used. How can we ensure that the encoder-decoder is properly trained?", "experimental_designs_or_analyses": "The authors compute extensive results, demonstrating impressive improvements over the state-of-the-art.  However, a) it is unclear what would happen if the environment configuration changes across tasks.  b) If there is an imbalance in tasks distribution, it could affect π-θ, which in turn might introduce an inductive bias in the encoder. This issue is not addressed or discussed in the paper.", "relation_to_broader_scientific_literature": "The approach of learning composite features that consist of both reward and policy in the embedding is non-trivial. The using of human feedback data to infer the task embedding instead of using encoder and context in meta-test phase is innovative", "essential_references_not_discussed": "SoA is satisfactory", "other_strengths_and_weaknesses": "Strength\n\nThe paper formulates the theorem and its proof in a satisfactory manner.\nThe authors also compute extensive results, demonstrating impressive improvements over the state-of-the-art.\n\nWeakness\nThe comparison with ANOLE might not be entirely fair. In their paper, they meta-train for 4M steps on the ant task and 2M steps on the cheetah task, while this paper only trains for 1M steps. It is unclear what would happen if they trained for a similar number of steps, making direct comparisons of the results problematic.\n\n\nLimited ablation study: The paper lacks an ablation study on the selection of hyperparameters.\n\n\nThe training details are also limited. During the meta-test phase, if multiple new tasks (T4, T5) arise, what will happen to the previously learned tasks (T1, T2, T3)? Will the learned meta-tasks perform the same way?\n\n\nThere are several notational and spelling errors throughout the paper, which make it difficult to understand the concepts. For instance, in lines 196-200.", "other_comments_or_suggestions": "Nothing", "questions_for_authors": "Refer weakness section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nif0QTwO2a", "reviewer": "Reviewer_oXJH", "summary": "This paper presents a novel meta-reinforcement learning (meta-RL) framework called Preference-Order-Preserving EMbedding (POEM), which enables test-time preference-based human-in-the-loop adaptation of the meta-RL policy. The main research problem is how to meta-train a policy when there exists a discrepancy between the reward feedback available at training time, and human preference-only feedback at test time. The paper proposes a preference-order-preserving task embedding algorithm that maps tasks into a latent space where task embedding similarity preserves preference orderings. At test-time, the method infers task embeddings from human preference queries and finds the best task-specific policy. Both theoretical and empirical results are presented in the paper. Theoretical results include a provably preference-order preserving similarity metric that addresses an issue with standard cosine similarity, and convergence guarantee for the adapted policy at test time. Empirical results compare POEM with baseline meta-RL methods such as ANOLE and MAML on standard simulated robotic continuous-control domains, showing significant improvement.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful and indebted for the time and effort invested in evaluating our manuscript. Thanks for the typo reminders and the suggestions to make our manuscript a better and stronger contribution. \n\n>**Methods And Evaluation Criteria: One minor weakness in the evaluation is the experiments are limited to continuous control robotics tasks, and the task spaces are mostly target locations. In these setups the level of difficulty in generalizing to different goals may not be very high. How would POEM perform in more complex task spaces?**\n\n**Answer:** In the conducted experiment, MetaWorld-ML10, the task is defined by the type of manipulation rather than the target location. The task family includes multiple types of manipulation tasks. As shown in Figure 8 (page 17), the interactions between the robot and the environment vary across tasks, encompassing different types of manipulation such as ‘assembly,’ ‘basketball,’ and ‘door opening.’ The training task set in this benchmark is highly heterogeneous, making the task space complex. As shown in Figure 5, the proposed method achieves a 50\\% improvement over the baselines on MetaWorld-ML10.", "claims_and_evidence": "1. The paper presents a preference-order preserving task encoder for context-based meta-RL learning that enables human-preference adaptation at test-time.\n\nThis claim is strongly supported by the description of POEM, theoretical derivation of similarity and the resulting encoder implementation, theoretical convergence guarantee, and details on the meta-train and adaptation algorithms.\n\n2. POEM achieves performance comparable to meta-RL oracle, with a 20%-50% improvement over baselines on standard continuous control tasks.\n\nThis claim is strongly supported by experiment results on nine continuous control environments using standard benchmarks (MuJoCo and MetaWorld) and comparisons against multiple baselines (ANOLE, MAML-reward).", "methods_and_evaluation_criteria": "The methods and evaluation criteria used in the paper are mostly appropriate for the problem.\n\nOne minor weakness in the evaluation is the experiments are limited to continuous control robotics tasks, and the task spaces are mostly target locations. In these setups the level of difficulty in generalizing to different goals may not be very high. How would POEM perform in more complex task spaces?", "theoretical_claims": "The paper makes 2 theoretical claims, both supported by proofs.\n\nTheorem 1: Proves that an embedding space can be constructed where task similarity aligns with preference order.\nTheorem 2: Shows that the adapted task embedding converges to the true task embedding with sufficient preference queries.", "experimental_designs_or_analyses": "The experimental design is well done, with ablations for different components of POEM. Details of the experiment sets, including evaluation tasks, hyperparameters, are given. Experiments also study how noise in the preference oracle affects the learned policy.", "relation_to_broader_scientific_literature": "The paper is well-placed within existing works on context-based meta-RL.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Presentation of the paper is a clear strength. From motivation to the theoretical results to implementation choices are all clearly explained.", "other_comments_or_suggestions": "L196, right column “the embedding \\tau_{\\pi}” -> “the embedding \\tau_{r}”\n\nL324, left column “PREAL” -> “PEARL”\n\nL347, right column “mile” -> “mild”\n\nL383, right column “APACE” -> “POEM”\n\nL760, “mile” -> “mild”", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GJKabp1Obu", "reviewer": "Reviewer_Fiat", "summary": "This paper focuses on meta-reinforcement learning with human in the loop adaptation scenario and proposes the Preference-Order-preserving EMbedding framework. The core idea of this framework is that if the optimal strategy of the environment achieves better performance in the other task, then the two tasks are more similar and the task embeddings are also closer. During the training phase, the framework trains an encoder and aligns the task embeddings with the preferences. In the human in the loop adaptation phase, the framework employs a task embedding inference method.\n\n## update after rebuttal\nMy concern was resolved, and I raised my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful and indebted for the time and effort invested in evaluating our manuscript and for all the suggestions to make our manuscript better.\n\n>**Weakness 1 and Q1**\n\n**Answer:** Thanks for pointing out the important observation. In both the existing context-based meta-RL methods, such as PEARL (Rakelly et al., 2019), and this paper, the ideal case of the meta-training can be using the optimal policy to reconstruct the decoder policy, and using the optimal policy to train the encoder. As you kindly mentioned, this can help avoid the error due to the current policy being not optimal. However, the optimal policies are not accessible, we have to replace the optimal policy reconstruction loss by the policy optimization loss.\n\nIn PEARL, the authors also use the policy optimization loss to train the output conditional policy (the decoder policy) and train the encoder simultaneously at the beginning of training. Although the policies during the training for the encoder are not optimal, the method is shown to be effective and efficient. One reason could be that although the policies are not optimal, they could provide an effective approximation to guild the optimization during the training to the correct direction. Therefore, we follow the same encoder-decoder training pattern in this paper, which keeps the algorithm statement concise.\n\n>**Weakness 2 and Q2**\n \n**Answer:** Explanation of Equation (9), i.e., $(\\hat{z}^{\\prime}, \\hat{z}^{\\prime\\prime})={\\arg\\min}_{z^{\\prime},z^{\\prime\\prime} \\in \\mathcal{Z}_k}( \\max \\lbrace|\\mathcal{Z}^{(1)}|,|\\mathcal{Z}^{(2)}|  \\rbrace )$:\n\nIn Equation (9), $\\mathcal{Z}^{(1)}=\\lbrace z\\in \\mathcal{Z}\\_k: S(z,z^{\\prime} ) >\\_\\epsilon S(z,z^{\\prime\\prime} )\\rbrace$ and $\\mathcal{Z}^{(2)}=\\lbrace z \\in \\mathcal{Z}\\_k: S(z,z^{\\prime\\prime} ) >\\_\\epsilon S(z,z^{\\prime})\\rbrace$.\nWhen $z^{\\prime}$ is preferred over $z^{\\prime\\prime}$, i.e., $S(z,z^{\\prime} ) >\\_\\epsilon S(z,z^{\\prime\\prime} )$, any embedding in $\\mathcal{Z}^{(1)}$ will satisfy the preference condition and remain to the valid embedding candidate. Similarly, when $z^{\\prime\\prime}$ is preferred over $z^{\\prime}$, i.e., $S(z,z^{\\prime\\prime} ) >\\_\\epsilon S(z,z^{\\prime} )$, any embedding in $\\mathcal{Z}^{(2)}$ will remain as the valid embedding candidate. If the number of the remaining valid embedding candidates is smaller, the range of the valid embeddings will be quickly narrowed, and then the query efficiency will be higher.\nTherefore, the goal of Equation (9) is to make the number of the remaining valid candidates as small as possible.\nAs we do not know what the queried preference is and which set of $\\mathcal{Z}^{(1)}$ and $\\mathcal{Z}^{(2)}$ will be remaining, we have to consider the worst case, i.e., the one between $\\mathcal{Z}^{(1)}$ and $\\mathcal{Z}^{(2)}$ with the larger size\nwill be remaining. Therefore, Equation (9) first takes the set $\\max \\lbrace |\\mathcal{Z}^{(1)}|,|\\mathcal{Z}^{(2)}|\\rbrace$ to pick the larger set in $\\mathcal{Z}^{(1)}$ and $\\mathcal{Z}^{(2)}$, and then minimizes its size. \n\nExplanation of line 17 in Algorithm 2, i.e., $z^k=\n\\mathop{\\arg\\max}\\_{z \\in \\mathcal{Z}_k} {\\sum}\\_{(z^{\\prime},z^{\\prime\\prime}) \\in \\mathcal{Q}\\_k} \\log \\mathrm{Pr} [S(z,z^{\\prime} )>S(z,z^{\\prime\\prime})]$:\n\nIn the $k$-th iteration of the human-in-the-loop adaptation, the candidate embedding set $\\mathcal{Z}_k$ includes multiple valid embedding candidates. However, we need to pick one as the output for the $k$-th iteration. Therefore, line 17 is to use the maximum likelihood over all the valid embedding candidates in $\\mathcal{Z}_k$ to determine the output task embedding $z^k$.\n\n>**Q3**\n\n**Answer:** After query the preferences of $(z_1,z_2)$ and $(z_3,z_4)$, it will be beneficial from querying $(z_1,z_3)$. However, $(z_1,z_3)$ is not the most query-effective pair to be queried. As mentioned in the **Answer to Q2**, we use Equation (9) to determine which pair is the most query-effective.\n\nNote that, in Algorithm 2, sampling two new embeddings other than $z_1, z_2, z_3, z_4$ in $Q$, incurs almost no cost, as it merely involves sampling from a normal distribution. However, the human preference is the most expensive step in the human-in-the-loop adaptation. Therefore, $(z_1,z_3)$ in $Q$ will not be used to query the human preference if it is not the most query-effective pair computed by Equation (9).\n\n>**Weakness 3 and Q4**\n\n**Answer:** Our method cannot handle tasks in which the dynamics change over time within a single task. If the dynamics is changing over time within any single task. When the task is revealed to the agent, the agent needs to query a human for preference comparison gradually and optimize the policy based on all the historical preference queries. However, when the dynamics change over time, the historical preference queries from humans may become invalid, which leads to a wrong direction for the policy optimization. Solving tasks with dynamics changing over time is an interesting future work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The authors propose a novel method building upon the broader scientific literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:1.  \n1.This paper focuses on meta-RL with human-in-the-loop adaptation, which I believe is an important topic.  \n2.The authors perform an evaluation on MuJoCo and Meta-World, demonstrate better performance compared to SOTA methods.  \n3.The paper is well-written, clearly explaining its contributions and well-grounded in prior literature.  \n\nWeaknesses:  \nThe use of the policy optimzation loss function to replace the optimal policy in Algorithm 1 requires further discussion.   \n2.The motivation behind Algorithm 2 needs to be explained in more detail.   \n3.Additionally, experiments on dynamic changing tasks should be included.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1.In Algorithm 1, the policy optimization loss is used to replace the optimal policy, while the preference loss requires the policy to be optimal. However, both losses are used simultaneously at the beginning of training. Would it be better to use only the policy optimization loss initially and introduce the preference loss after n epochs?   \n2.I still cannot fully understand why Equation (9) and line 17 of Algorithm 2 are designed in this way. Could the authors provide further explanation in plain language as to why this formulation leads to high query efficiency?  \n3.In Algorithm 2, would it be beneficial to further query the preferences of z in Q? For example, based on the pairs ($z_{1}$,$z_{2}$) and ($z_{3}$,$z_{4}$) in Q, could querying ($z_{1}$,$z_{3}$) lead to better results.  \n4.The method should also be applicable to tasks with dynamic changing. How does the method perform in such environments?\nIf the authors address the concerns I have raised, I'm willing to increase my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper propose, Preference-Order-preserving Embedding (POEM), a new framework for meta-reinforcement learning (meta-RL). It aims to improve few-shot adaptation to new tasks using human preference queries. In POEM, the trajectories of different tasks are encoded into an embedding, with the similarities between tasks reflected in these embeddings. Given a new task, the task encoder facilitates efficient task embedding inference for new tasks from the preference queries and then obtains the task specific policy. 

Reviewers generally agree that this paper has strong experimental results and the author has address most of the reviewers concern.
Decision: Accept (poster)