The AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI (R2 HCAI)

Exploiting Unlabeled Data for Feedback Efficient Human Preference based
Reinforcement Learning
Mudit Verma1

Siddhant Bhambri 1

Subbarao Kambhampati 1

1

SCAI, Arizona State University
muditverma@asu.edu, siddhantbhambri@asu.edu, rao@asu.edu

Abstract2022). In recent literature, a potential solution to such issues has been to allow a human in the loop (HiL) to specify their preferences as feedback on queried trajectory pairs(Wilson, Fern, and Tadepalli 2012; Christiano et al. 2017;
Verma and Metcalf 2022) instead of feeding hand-designed
reward functions to the system.

Preference Based Reinforcement Learning has shown muchpromise for utilizing human binary feedback on queried trajectory pairs to recover the underlying reward model of the
Human in the Loop (HiL). While works have attempted tobetter utilize the queries made to the human, in this work wemake two observations about the unlabeled trajectories collected by the agent and propose two corresponding loss functions that ensure participation of unlabeled trajectories in the
reward learning process, and structure the embedding spaceof the reward model such that it reflects the structure of state
space with respect to action distances. We validate the proposed method on one locomotion domain and one robotic manipulation task and compare with the state-of-the-art baseline
PEBBLE. We further present an ablation of the proposed loss
components across both the domains and find that not only
each of the loss components perform better than the baseline,
but the synergic combination of the two has much better reward recovery and human feedback sample efficiency.

Recent Preference-based Reinforcement Learning(PbRL) methods like (Lee, Smith, and Abbeel 2021; Parket al. 2022; Verma and Metcalf 2022; Christiano et al.
2017; Soni et al. 2022) can efficiently utilize the queries
made to the human in the loop via several key ideas likepre-training (Lee, Smith, and Abbeel 2021), improved
query sampling strategy (Lee, Smith, and Abbeel 2021),
data augmentation (Park et al. 2022; Guan et al. 2021; Guan,
Verma, and Kambhampati 2020), or priors specified overreward function (Verma and Metcalf 2022). In this work, we
present a complementary approach by which we can exploit
unlabeled trajectories for improved reward recovery of theunderlying human reward function. Our method relies on
two main observations: the first is that an extremely large
population of trajectories lie in the agent’s buffer (collected
over training episodes) that are not used in the reward
learning process and that the representation space for the
reward function being learnt is not reflective of how thestate space is structured. Specifically, our first observation
reinforces the fact that much of the explored trajectories
do not actually participate in the reward learning process
and in fact their best change to affect the reward function
is once they get sampled and queried to the human in theloop. We posit that this untapped data source can greatly
improve reward recovery and reduce the feedback sample
complexity. Our second observation notes that the reward
function being learnt may not conform to the structure of
the state space simply because it doesn’t get exposed to as
many data points (in comparison to, say, the policy approximation function). We utilize our observations to improve
performance of RL agents in recovering the underlying
reward function and learn a good policy by exploiting therich unlabeled trajectory data. Although works like SURF
(Park et al. 2022) have proposed a semi-supervised learning
approach to utilize unlabeled trajectory data, they would
generate labels for unlabeled trajectories and use thesedata points as if they were given by the human in the loop.
We argue that this is still an indirect way of updating theIntroduction
Reinforcement Learning (RL), especially Deep Reinforcement Learning has gained immense popularity with significant leaps in allowing agents to learn complex behaviors, inhigh dimensional state and action spaces (Mnih et al. 2015;
Arulkumaran et al. 2017). However, much of the successes
have also been attributed to well specified reward functions
which ground the agent’s behavior and subsequent task inthe expected manner. As prior works have argued, the specification of low level reward functions for seemingly easytasks could be quite difficult and may still result in inexplicable and unexpected results (Verma et al. 2019, 2021;
Gopalakrishnan, Verma, and Kambhampati 2021a,b) potentially affecting trust between Human-AI (Zahedi et al. 2021,2022). For example, works like (Krakovna et al. 2020; Vamplew et al. 2018) have raised the issues of reward hacking
and reward exploitation where the RL agents would discover behaviors that seems to be “cheating” or incorrect andyet maximize the expected cumulative reward. This has also
gotten attention from the explainable AI community where
they attempt to analyze whether the agent is actually behaving in the intended manner (Verma, Kharkwal, and Kambhampati 2022; Sreedharan et al. 2020; Kambhampati et al.

1

reward function via these unlabeled trajectories since all
the information about the predicted reward values is lostwhen providing a hard decision label on which trajectory
was preferred. Having said this, the proposed triplet loss, inpart, attempts to perform semi-supervised (Zha, Bhambri,
and Guan 2021; Park et al. 2022) learning and builds over
SURF for PbRL. Finally, empirically we found that our
method easily outperforms SURF on both the locomotion
and robotic manipulation task.

implications are not limited to PEBBLE and can be applied
to any PbRL method that attempts to approximate the underlying human reward function. In fact, our method is complementary to existing methods that typically improve theagent’s performance or reward recovery by improved query
sampling strategy or data augmentation to name a few.

MethodWe operationalize our observations as mentioned in the Introduction section by utilizing the unlabeled trajectory dataand propose a solution to leverage each of the two observations. The Result section shows that these two insights are
actually complementary. For our first observation, there exists a rich source of unlabeled data and that it could be helpful to ensure that the reward function being learnt is affectedby it, we make the following assumption :Background
Reinforcement learning allows for agents interacting inan environment E where at each discrete timestep t, theagent receives an observation ot from the environment andchooses an action at based on its policy π. As in conventional RL frameworks we assume that the underlying
system is a Markov Decision Process, i.e. the tuple <S, T , A, R̃h , γ > describing the state space S, agent’s action space A, the underlying environment transition dynamics T , the discountP
factor γ where the agent’s goal is to max∞k
imize the return
k=0 γ R̃h (st+k , at+k ) computed over
the reward system Rh in concern. In the preference based
reinforcement learning setup we are interested in, the goalof the agent is two fold, first to infer the human’s underlying
reward model R̃h via binary feedback over trajectory pairs
and further use the learnt reward model Rh to compute a
policy πϕ parameterized by ϕ to maximize discounted cumulative return over Rh .
We utilize the formulation presented in (Wilson, Fern,
and Tadepalli 2012) for the preference based reinforcement learning problem where the agent queries the human in the loop with a trajectory pair τ0 , τ1 , τi ={(sk , ak ), (sk+1 , ak+1 · · · (sk+H , ak+H ))} for a binary
feedback y ∈ {0, 1} indicating their preferred trajectory.
Such feedbacks along with the queried trajectories are stored
in a dataset Dτ as tuples (τ0 , τ1 , y). Following the Bradley
Terry model (Bradley and Terry 1952) to compute probability of one trajectory be preferred over another, recent
line of works like (Lee, Smith, and Abbeel 2021; Christiano
et al. 2017) approximates the human reward function as Rh ,parameterized by, say, ψ, by solving a supervised learning
problem where the returns computed over the learnt reward
function are higher for trajectories that were preferred by
the human in the loop than the returns computed on the nonpreferred trajectory. This is done by minimizing the crossentropy between the predictions and ground truth human labels as follows:

Assumption 1 A trajectory τ , sampled under a policy πϕ ,that has not been queried to the human in the loop is assumed to be preferred by the human.
Since there exist a large bank of trajectories that has notbeen queried to the HiL, Assumption 1 makes a paternalistic
choice about whether those trajectories would be preferredby the HiL over some other trajectory. Moreover, we can use
this assumption to ensure that the reward model can now use
these unlabeled trajectories. In contrast to prior works like
(Park et al. 2022) that extract labels via learnt reward modelover the unlabeled trajectory data, we use our assumption
1 and propose a triplet loss that directly updates the reward
model as follows,X
1
Lt (τ ; Dh ) =max(0, ||R(τ ) − R(τg )||2
|Dh |τg ,τb ∼Dh− ||R(τ ) − R(τb )||+m)(3)

where probabilities Pψ are computed using the approximated reward function Rh as :

P
0 0
exp
t Rh (st , at )PPψ [τ0 ≻ τ1 ] = P(2)i ii∈{0,1} exp
t Rh (st , at )where m is the margin hyperparameter. We overload the notation for reward to reflect the rewards
for the trajectory states as a vector, i.e. R(ø) =T[R(s0 ) R(s1 ) · · · R(sT −1 ) R(sT ))] for a trajectory τ of length T . Dτ is the bank of trajectories sampled
by the agent, a common element used by off-policy RL algorithms as the replay buffer. Dh is the dataset of preferencelabels over the queried trajectory pairs where we use τg todenote the trajectory that was preferred over τb . The triplet
loss uses the preferred trajectories as the positive samples,the dis-preferred trajectories as the negative sample and theunlabeled trajectory τ as the anchor.
For our second observation, we propose an action based
loss (variants of which have been seen in works like (Venkattaramanujam et al. 2019; Eysenbach, Salakhutdinov, andLevine 2019)) for the reward model to impose a soft constraint on the state embedding being learnt by the reward
model in an attempt to ensure that the reward model also
reflects the structure of state space (with respect to action
distances).

For our experiments we use the PEBBLE (Lee, Smith, andAbbeel 2021) as the backbone, however the proposed work’s

Definition 1 Action distance Ad between two states undersome policy πϕ (s) and transition dynamics T (s, a, s′ ) is

LCE = −

E[y(0)logPψ [τ0(τ0 ,τ1 ,y)∼D(1)≻ τ1 ] + y(1)logPψ [τ1 ≻ τ0 ]]2

given by the expected number of action steps taken to reacha state s2 from s1 .
We propose to enforce such a soft constraint in the embedding space of the reward model, Re (s) computes the embedding of the state s, by ensuring that the euclidean distance
between the embedding of two states s1 and s2 reflects theaction distance Ad (s1 , s2 ). This can be achieved by minimizing the Mean Squared Error (MSE) between the computed distance in the embedding space and the action distance as follows :X
1
La (Dp ) =(||Re (si ) − Re (sj )||2 −dy )2|Dp |(a) Success Ratesi ,sj ,dy ∼Dp(4)
where for a si , sj are pair of states in the dataset Dp =(si , sj , dy ) which consists of the computed ground truth action distances between them as dy . What remains is to create
this dataset Dp . We utilize the trajectory bank Dτ (and also
include trajectories in Dh ) to obtain Dp . The key idea is thatsince the action distance ground truth that we want is an expectation over number of actions taken to reach sj from si ,we can approximate this action distance by sampling a statesi , sj ∈ τ where j > i, τ ∈ Dτ and use the number of action
steps taken in the trajectory from s1 to s2 as the ground truth
distance dy = j − i. An important note is that the distances
dy in the dataset Dp should be from the agent’s current policy πϕ . For off-policy RL algorithms where the replay buffer,Dτ , would contain trajectories sampled from a stale policy,
we emulate the required behavior of the dataset Dp by ensuring that only the last k trajectories added to the datasetDτ are used to compute Dp .
We finally utilize a linear combination of the two proposed loss functions Lt , triplet loss, and La , action distanceloss, with the cross entropy loss LCE (see Section ) to update the reward model as :Lreward = λCE LCE (Dh ) + λt Lt (Dτ )+ λa La (Dp (Dτ [k..T ]))(b) Return of learnt πϕ on ground truth reward Rh

Figure 1: Evaluation curves on the robotic manipulation
task of Sweep-Into as measured on the success rate and theground truth human reward Rh .
(Quadruped) and the other is a robot manipulation task
(Sweep-Into). Recent literature (Lee, Smith, and Abbeel
2021; Park et al. 2022) on Preference-based Reinforcement
Learning has showcased results particularly for continuous
control tasks like locomotion and robot manipulation. Future work involves more extensive evaluation on other tasks
like Walker, Cheetah, Drawer Open, Window Open etc.,
and further investigation of the method’s benefits on explicit
knowledge tasks in discrete action space domains like Montezuma’s Revenge and MS-Pacman.
In order to systematically compare our work with thebaselines we use a synthetic oracle that has a fixed reward
model for the agent (Rh ). A good PbRL algorithm should be
able to recover this reward model, Rψ , and subsequently apolicy learnt on the recovered reward model, πϕ , when evaluated on the human’s reward model, should yield high expected return. We follow existing literature and assume theenvironment’s original reward function as the oracle’s underlying ground truth reward model Rh and provide a feedback label as follows :PP0Rh (τ0 ) > i Rh (τ1 )iPPy(τ0 , τ1 ) =(6)
1i Rh (τ0 ) <i Rh (τ1 )(5)where LCE is computed over Dh that contains the queried
trajectory pairs with human binary feedbacks (mean overthe samples), Lt is computed over the unlabeled trajectory
buffer (mean over all the trajectories) Dτ and La is computed over the dataset of state pairs with action distance Dp(mean over all the tuples) created from the k most recent
trajectories added to Dτ .

Experiments
We wanted to investigate the following two questions via
empirical evaluations :1. Do the proposed losses improve the existing state of theart in preference based RL in terms of reward recovery,
feedback efficiency and performance of the learnt policy?
2. Are the two losses proposed in this work complementary,and more so synergic?

Note that there can exist a third case in equation 6, where
the human has exactly equal preference over the trajectory
pairs, and does not occur frequently after a few updates tothe reward model. Even though prior works have highlighted
the use of soft cross entropy loss (instead of LCE ) to handle

We validated our proposed method via experimentson two domains, of which one is a locomotion task
3such situations it is not central to the problem of PbRL withsparse equal preferences over a large number of randomly
sampled trajectory pairs, and we do not expect the use of
LCE to limit the presented losses Lt , La and concepts in any
way. We do, however, plan to investigate the implicationsof the proposed techniques when the human in the loop isallowed to mark trajectory queries as equally preferred.

Implementation Details :The underlying RL algorithm used to train the policy is SAC(Haarnoja et al. 2018) for baseline PEBBLE and ours. For
all our experiments we query feedback over trajectories ofa fixed length of 50 and borrow other hyperparameters used
for the baseline algorithm, SAC, and (for the hyperparameters that are common) ours from (Lee et al. 2021). We use
λC E = 1, λt = 0.5, λa = 3 to ensure that all the computed
loss values have a similar scale. To compare with the original
baseline results, we set the max feedback threshold to 1000
for Quadruped-Walk and 10000 for Sweep-Into. We use thesame architecture for the policy and the reward models asin (Lee, Smith, and Abbeel 2021), and use the penultimate
layer in the reward model as the embedding space requiredby the action distance loss.

Figure 2: Evaluation curves on the locomotion task of
Quadruped as measured on the ground truth human reward
Rh .

Finally, to ascertain that the two losses presented in thiswork that utilize unlabeled trajectory data (triplet loss Lt andaction distance loss La ) work complementary to each other,
we perform an ablation comparing the two losses together
against only the triplet loss and only the action distance loss.
We find that, although, in both the environments even only
one of the losses easily outperform the baseline PEBBLE,
the synergic combination of the two increases the performance measures substantially with much fewer human feedback samples.

Results
For both of the environments we compare our method
against two baselines, PEBBLE (Lee, Smith, and Abbeel2021) that follows a similar training paradigm but only uses
the Cross Entropy Loss and an RL (SAC) baseline that hasaccess to the underlying ground truth reward Rh . Our results
for the baseline PEBBLE and SAC performance on these domains can be corroborated as shown by (Park et al. 2022).
For the task of Quadruped-Walk, PEBBLE (see Fig. 2)
only achieves performance levels of a return of ∼400 which
is much less than that of our SAC baseline ∼980, in contrast, Our work (Triplet Loss, Ad Loss) performs significantly better than the PEBBLE baseline and reaches performance levels of SAC (even outperforming it for a while,that has potentially been attributed to better exploration in
PbRL setups than SAC) with significantly fewer feedback
samples. Similar gains are seen for the task of Sweep-Into(see Fig. 1) where PEBBLE utilizes a very high number of
feedback samples to show any improvements in the successrate (Fig. 1a) and the return over ground truth reward (Fig.
1b). The combination of triplet loss and action distance loss
easily outperforms PEBBLE baseline and reaches a reasonable success rate and return values.
We also compare our results with SURF, and refer thereaders to the reported results in (Park et al. 2022) for the
locomotion and the robotic manipulation task. In the reported results, for Sweep-Into, SURF performs closely to
our baseline PEBBLE and achieves ∼ 75% success ratewith the same number of feedbacks compared to almost perfect 100% for our work. Similarly, SURF’s doesn’t show
any gains on Quadruped over PEBBLE in their reported results and, as discussed, the learnt policy achieves 50% of
the expected return by SAC, whereas as shown in fig. 2, we
not only show significant improvement over the baseline but
also achieve similar performance as SAC.

Discussion
In this work we presented two key observations (and corresponding loss functions) regarding the utilization of unlabeled trajectories for a PbRL agent. We first proposed a
triplet loss under the optimistic assumption that an unlabeledtrajectory would be preferred by the human, and secondly,
our action distance loss function that attempts to structure
the embedding space of the reward model being learnt toreflect action distances between state pairs. We show that although these individual losses perform much better than thebaseline PbRL and RL (SAC) in terms of reward recovery
and human feedback sample efficiency, the synergic combination of these yield a more powerful PbRL agent with low
demands of human sample feedback and high performance.
Future work includes a more thorough investigation of
the effects of proposed method across diverse locomotion,
robotic manipulation as well as explicit knowledge discrete
domains. We also intend to perform an extensive evaluationof the effects of several key hyperparameters like the query
trajectory length and maximum number of feedbacks to further bolster our claims.

Acknowledgements
Kambhampati’s research is supported by the J.P. Morgan
Faculty Research Award, ONR grants N00014-16-1-2892,
N00014-18-1-2442, N00014-18-1-2840, N00014-9-1-2119,
AFOSR grant FA9550-18-1-0067 and DARPA SAIL-ON
grant W911NF19-2-0006.
4References

Park, J.; Seo, Y.; Shin, J.; Lee, H.; Abbeel, P.; and Lee, K.
2022. SURF: Semi-supervised Reward Learning with Data
Augmentation for Feedback-efficient Preference-based Reinforcement Learning. arXiv preprint arXiv:2203.10050.
Soni, U.; Sreedharan, S.; Verma, M.; Guan, L.; Marquez,M.; and Kambhampati, S. 2022. Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansion. arXiv preprint
arXiv:2210.15096.
Sreedharan, S.; Soni, U.; Verma, M.; Srivastava, S.; andKambhampati, S. 2020. Bridging the Gap: Providing
Post-Hoc Symbolic Explanations for Sequential DecisionMaking Problems with Inscrutable Representations. arXiv
preprint arXiv:2002.01080.
Vamplew, P.; Dazeley, R.; Foale, C.; Firmin, S.; and Mummery, J. 2018. Human-aligned artificial intelligence is amultiobjective problem. Ethics and Information Technology,
20(1): 27–40.
Venkattaramanujam, S.; Crawford, E.; Doan, T.; and Precup, D. 2019. Self-supervised learning of distance functions
for goal-conditioned reinforcement learning. arXiv preprint
arXiv:1907.02998.
Verma, M.; Bhambri, S.; Gupta, S.; and Buduru, A. B.
2019. Making Smart Homes Smarter: Optimizing Energy
Consumption with Human in the Loop. arXiv preprint
arXiv:1912.03298.
Verma, M.; Kharkwal, A.; and Kambhampati, S. 2022. Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop.
arXiv preprint
arXiv:2210.03455.
Verma, M.; and Metcalf, K. 2022. Symbol Guided HindsightPriors for Reward Learning from Human Preferences. arXiv
preprint arXiv:2210.09151.
Verma, M.; Shah, N.; Nayyar, R. K.; and Hanni, A. 2021.
Perfect Observability is a Myth: Restraining Bolts in theReal World.
Wilson, A.; Fern, A.; and Tadepalli, P. 2012. A bayesian
approach for policy learning from trajectory preference
queries. Advances in neural information processing systems,
25.
Zahedi, Z.; Sreedharan, S.; Verma, M.; and Kambhampati,
S. 2022. Modeling the Interplay between Human Trust andMonitoring. In 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 1119–1123. IEEE.
Zahedi, Z.; Verma, M.; Sreedharan, S.; and Kambhampati,
S. 2021. Trust-aware planning: Modeling trust evolution
in longitudinal human-robot interaction. arXiv preprint
arXiv:2105.01220.
Zha, Y.; Bhambri, S.; and Guan, L. 2021. Contrastively
Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
7835–7842. IEEE.

Arulkumaran, K.; Deisenroth, M. P.; Brundage, M.; andBharath, A. A. 2017. Deep reinforcement learning: A brief
survey. IEEE Signal Processing Magazine, 34(6): 26–38.
Bradley, R. A.; and Terry, M. E. 1952. Rank analysis ofincomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4): 324–345.
Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.;
and Amodei, D. 2017. Deep reinforcement learning fromhuman preferences. Advances in neural information processing systems, 30.
Eysenbach, B.; Salakhutdinov, R. R.; and Levine, S. 2019.
Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 32.
Gopalakrishnan, S.; Verma, M.; and Kambhampati, S.
2021a. Computing Policies That Account For The Effects
Of Human Agent Uncertainty During Execution In Markov
Decision Processes. arXiv preprint arXiv:2109.07436.
Gopalakrishnan, S.; Verma, M.; and Kambhampati, S.
2021b. Synthesizing Policies That Account For Human Execution Errors Caused By State Aliasing In Markov Decision Processes. In ICAPS 2021 Workshop on Explainable AI
Planning URL https://openreview. net/pdf.
Guan, L.; Verma, M.; Guo, S. S.; Zhang, R.; and Kambhampati, S. 2021. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data
augmentation. Advances in Neural Information Processing
Systems, 34: 21885–21897.
Guan, L.; Verma, M.; and Kambhampati, S. 2020. Explanation augmented feedback in human-in-the-loop reinforcement learning. arXiv preprint arXiv:2006.14804.
Haarnoja, T.; Zhou, A.; Hartikainen, K.; Tucker, G.; Ha, S.;
Tan, J.; Kumar, V.; Zhu, H.; Gupta, A.; Abbeel, P.; et al.
2018. Soft actor-critic algorithms and applications. arXiv
preprint arXiv:1812.05905.
Kambhampati, S.; Sreedharan, S.; Verma, M.; Zha, Y.; andGuan, L. 2022. Symbols as a lingua franca for bridging
human-ai chasm for explainable and advisable ai systems.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 12262–12267.
Krakovna, V.; Uesato, J.; Mikulik, V.; et al. 2020. Specification gaming: The flip side of AI ingenuity— DeepMind.
Lee, K.; Smith, L.; and Abbeel, P. 2021. Pebble: Feedbackefficient interactive reinforcement learning via relabelingexperience and unsupervised pre-training. arXiv preprint
arXiv:2106.05091.
Lee, K.; Smith, L.; Dragan, A.; and Abbeel, P. 2021. Bpref: Benchmarking preference-based reinforcement learning. arXiv preprint arXiv:2111.03026.
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control
through deep reinforcement learning. nature, 518(7540):
529–533.
5