State Augmentation Based Approach toReinforcement Learning from Human Preferences
Mudit Verma, Subbarao Kambhampati
School of Computing & AI, Arizona State University{muditverma, rao} @ asu.edu

R2HCAI, AAAI 2023

Objectives
While Preference Based Reinforcement Learning has progressed in several
continous control tasks, progress with image based state inputs is stilllimited.
Data augmentation has provided a zero-cost to human in the loop method
in several areas of Machine Learning. We intend to use State based data
augmentation technique for improving the reward learning objective.
Augmented trajectories modify the irrelevant parts of the image
representation. We hypothesize image regions that do no update upon atransition as potentiall irrelevant.

Figure 1: Overview of our proposed approach

Empirically verify claims on popular PbRL benchmarks.
PbRL Preliminaries• Approximating the Human Reward Model> Objective: Compute probability for the human preferring trajectory !!
over !" :> Objective: The human reward model can then by learnt by minimizing
the cross-entropy between the predictions made by the supervised learnerand the ground truth human labels as follows:

Method : State Augmentation
We augment the trajectory data in the feedback buffer and propose theinvariance loss to ensure that the reward predictions are consistent acrossthe perturbed trajectories.
• State Augmentation> Insight : We hypothesize that for tacit tasks such as locomotion or
robotic manipulation, an important feature to consider is how the stateschange upon transition.

Experiments & Results• Investigate via Empirical Evaluation> Question : Does the proposed state augmentation approach improveagent performance in PbRL setup?
> We use the backbone PbRL
algorithm as PEBBLE.
> We find that the data
augmentation technique ishelpful for OpenAI Gym taskof Mountain Car, Locomotion
task of Quadruped-Walk andMetaworld robotic
manipulation task of SweepInto.

(a) Mountain Car (Continuous)> While image based PbRL is
still in infancy, data
augmentation based methods
that adds zero cost to thehuman in the loop becomes amust-have.
(b) Metaworld Sweep-Into> We create a mask for state-differences as follows :> Perturb the image representations to get new state as :> Finally, generate perturbed trajectories as :• Invariance Consistency across Augmentations> Following prior work in data augmentation, we force predicted reward
vector for perturbed trajectories to be same as the original trajectory.

(c) DM Control Quadruped-Walk
Figure 2: Evaluation curves on various continous control tasks
compairing PEBBLE with Our method.

TLDR
We propose a state augmentation technique tailored for image-based
Preference-based Reinforcement Learning.
Insight : Regions of the image observation that update upon a transition are
at least a subset of all the ``content" available in the image observation.
Reward model should be invariant to changes to regions except content.

Kambhampati's research is supported by the J.P. Morgan Faculty Research Award, ONR grants
N00014-16-1-2892, N00014-18-1-2442, N00014-18-1-2840, N00014-9-1-2119, AFOSR grant FA955018-1-0067 and DARPA SAIL-ON grant W911NF19-2-0006.

We evaluate on OpenAI gym's Mountain Car, DM Control Quadruped-Walk,
and Meta World's Sweep-Into.