Adversarial Imitation Learning with Preferences
Authors: Aleksandar Taranovic
Andras Gabor Kupcsik
Niklas Freymuth
Gerhard Neumann
Venue: ICLR 2023 poster (2023)
Keywords: preference learning
learning from demonstration
adversarial imitation learning

Abstract: Designing an accurate and explainable reward function for many Reinforcement Learning tasks is a cumbersome and tedious process. 
Instead, learning policies directly from the feedback of human teachers naturally integrates human domain knowledge into the policy optimization process. 
However, different feedback modalities, such as demonstrations and preferences, provide distinct benefits and disadvantages. For example, demonstrations convey a lot of information about the task but are often hard or costly to obtain from real experts while preferences typically contain less information but are in most cases cheap to generate. 
However, existing methods centered around human feedback mostly focus on a single teaching modality, causing them to miss out on important training data while making them less intuitive to use.
In this paper we propose a novel method for policy learning that incorporates two different feedback types, namely \emph{demonstrations} and \emph{preferences}. 
To this end, we make use of the connection between discriminator training and density ratio estimation to incorporate preferences into the popular Adversarial Imitation Learning paradigm. 
This insight allows us to express loss functions over both demonstrations and preferences in a unified framework.
Besides expert demonstrations, we are also able to learn from imperfect ones and combine them with preferences to achieve improved task performance.
We experimentally validate the effectiveness of combining both preferences and demonstrations on common benchmarks and also show that our method can efficiently learn challenging robot manipulation tasks.

Published as a conference paper at ICLR 2023
ADVERSARIAL IMITATION LEARNING WITH PREFER- ENCES
Aleksandar Taranovic Andras Kupesik?, Niklas Freymuth' Gerhard Neumann'
1 Autonomous Learning Robots Lab, Karlsruhe Institute of Technology, Karlsruhe, Germany 2 Bosch Center for Artificial Intelligence, Renningen, Germany
ABSTRACT
Designing an accurate and explainable reward function for many Reinforcement Learning tasks is a cumbersome and tedious process. Instead, learning policies directly from the feedback of human teachers naturally integrates human domain knowledge into the policy optimization process. Different feedback modalities, such as demonstrations and preferences, provide distinct benefits and disadvan- tages. For example, demonstrations convey a lot of information about the task but are often hard or costly to obtain from real experts while preferences typically contain less information but are in most cases cheap to generate. However, exist- ing methods centered around human feedback mostly focus single teaching modality, causing them to miss out on important training data while making them less intuitive use. In this paper we propose novel method for policy learning that incorporates two different feedback types, namely demonstrations and prefer- ences. To this end, we make use of the connection between discriminator training and density ratio estimation to incorporate preferences into the popular Adversar- ial Imitation Learning paradigm. This insight allows us to express loss functions over both demonstrations and preferences in a unified framework. Besides ex- pert demonstrations, we are also able to learn from imperfect ones and combine them with preferences to achieve improved task performance. We experimentally validate the effectiveness of combining both preferences and demonstrations on common benchmarks and also show that our method can efficiently learn chal- lenging robot manipulation tasks.
INTRODUCTION
This paper aims to progress research towards enabling humans without expert knowledge of machine learning or robotics teach robots to perform tasks using diverse feedback modalities. Enabling human teachers to use various feedback types allows for more natural human-robot training in- teraction. In particular, this paper focuses on human demonstrations of the desired behaviour, and preferences, which are pairwise comparisons of two possible robot behaviors. Both types of feed- back have distinct benefits. Demonstrations exploit the domain knowledge of human teachers for given task. Yet, they are often cumbersome to generate, and present considerable cognitive load on the human teacher, up to the point that the teacher might not be capable to demonstrate optimal behavior on their own. In contrast, preferences easier to evaluate, albeit less informative they only indicate relative quality, i.e., which option is better among two provided ones.
Both feedback types have been extensively researched in isolation. For instance, plethora of ap- proaches focus on learning from demonstrations, also referred to as imitation learning (Osa et al.) 2018). In recent years, preference learning has been an active research topic (Wirth al. 2017). but integrative work on learning from multiple feedback modalities has been much sparser and is often limited e.g., theoretical analysis (Jeon al. 2020).
In this paper, we introduce Adversarial Imitation Learning with Preferences (AILP), novel method for learning from a combination of demonstrations and preferences that builds upon the well-known Adversarial Imitation Learning (AIL) framework. AlL. uses a discriminator that indicates the change
"Corresoonding author; audkaandartaranoiiqqaartnettete

---

Published as a conference paper at ICLR 2023
of the policy, which can be seen differential reward. In preference learning, typically we directly encode the reward in preference and require a static reward. We show that combining a differential and static reward directly in the form of adversarial comparisons and preferences is incompatible and leads to poor performance. To alleviate this issue, we present a preference loss that is compatible with AlL. and therefore enables AlL approaches to benefit from additional preference feedback that is available alongside demonstrations. We provide an overview of our method in Figure
Preferemoes
Toacher
Preference Par
Demonetrations
Iryest
Preference Query
New Demonstration
Policy
Samples
Discriminator
Libes
Environment
m(als)
(
Pretrain
Relabel Train
R(s,
H((s)
Figure Schematic overview of Adversarial Imitation Learning with Preferences (AILP). Given policy (blue) that may be pre-trained self-supervised maximum entropy objective, we can optionally query a teacher for new demonstrations (yellow) and preferences between trajectories from the buffer B (green). Next, discriminator o(s, a) (purple) is trained to discriminate between samples from and teacher demonstrations (Ldem), while at the same time preferring better samples over worse ones (Lpret). For an off-policy Reinforcement Learning (RL) algorithm, this training makes use of the environment (grey) and sample buffer that needs to be relabeled according to the reward that depends on d(s, a)). This process is iterated until convergence, at which point the policy produces samples that are indistinguishable from the expert demonstrations.
The contribution of the paper are as follows: (i) we present novel preference-leameee approach that is compatible with Adversarial Imitation Learning and achieves results comparable the state of the art, (ii) we extend Adversarial Imitation Learning to include learning from both preferences and demonstrations, (iii) we present extensive evaluations of the proposed method, outperforming various baseline methods well-known benchmarks.
RELATED WORK
Teaching robots by human teachers has been an active research topic in the past years (Chernova & Thomaz, 2014). There has been research unifying multiple feedback types into single frame- work, e.g.. by (Jeon et al., (2020), but the focus has been mostly on unimodal feedback. Two com- monly researched human feedback modes are demonstrations and preferences. Besides these, there are numerous others, such as assigning numerical values to different options by human teachers, as in the work by Wilde al. (202T).
Learning from Demonstrations. Recently, learning from demonstrations and Imitation Learning (IL) (Osa al. 2018 Argall et al.) 2009) have been active research fields in robot learning which the robot is presented with instances of desired expert behavior. IL algorithms can generally be classified as either Behavioral Cloning (BC) (Torab et al. Florence et al. (202T). where a pol- icy is directly regressed from demonstrations in a supervised fashion, or as Inverse Reinforcement Learning (IRL) (Abbeel & Ng 2017 Ziebart et al. 2008 Zakka et al., 2021), which recovers and subsequently optimizes reward function from demonstrations. In recent years, there has been rise in adversarial methods inspired by Generative Adversarial Networks (Goodfellow et al.) 2014). Starting with Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon (0166). these meth- ods utilize a discriminator based distribution matching objective for both BC (Ho Ermon, 2016 Torabi et al.) and IRI. (Fu et al. 2018 Xiao al. 2019). Building this, another body work
2

---

Published as a conference paper at ICLR 2023
utilizes the inherent connection between discriminator training and (log) density ratio estimation for AIL with Gaussian Mixture Model (GMM) policies (Becker et al. 2020 Freymuth et al. 2021 2022)
Learning from Preferences. Preference evaluation indicates the preferred behavior out of two pos- sible options. Compared to demonstrations, preferences generally easier to provide for human teachers, but contain significantly less information. Here, early work made use of Gaussian Process (GP) for encoding preferences for classification tasks (Chu & Ghahramani 2005 Houlsby et al.) 2012). Biyik et al. (2020) have also used GPs to learn a reward function with a predefined set of behaviors. Moreover, these models can be used with trajectory features lower the dimensionality of the problem (Sadigh et al. [017]] Myers et al., [222]] Blyık & Sadigh 2018). Christiano et al. ) recover the full policy by using deep RI. approach. An extension this work, named Peb- ble (Leee et al. (02Ta), combines unsupervised entropy-based pre-training with a Soft Actor-Critic (SAC) (Haarnoja et al.) 2018) policy to optimize the learned reward function. [ee et al.) (202Tb) evaluate Pebble using different teacher profiles. In this paper compare our method with the results presented in Lee et al. (2021a), leaving the evaluation with different teachers for future work.
Learning from Demonstrations and Preferences. peon et al. (2020) define formalism that en- ables integrating various feedback types into a unified framework. However, this work is a theo- retical analysis and requires further empirical evaluation. Practical methods that combine demon- strations and preferences have so far mostly been extensions of existing work for learning from one of those feedback types. In BByyk et al. (202T), the authors propose Bayesian method which infers an initial belief about the unknown reward function from demonstrations, and then uses pref- erences (similar BLyuk & Sadigh (2018) to refine this function better match the true reward. Yet, this method requires trajectory-wist features, while AILP directly acts on state-action pairs. Moreover, Ibarz et al.) (2018) propose bootstrapping the initial policy using demonstrations and then apply preterence learning approach (Christiano et al. (017). In our method we integrate both demonstrations and preferences into the learning process, without any assumptions on when and what type of feedback is required for learning. [Brown et al. (2019) extended work from Brown et al.) and introduced a method that uses suboptimal demonstrations to train model train using behavior cloning. By using this model with different levels of noise they can self generate preferences which they learn using the same loss function as in Lee et al. (2021a).
PRELIMINARIES
In this section, we present overview of used notations in the paper, well as preliminaries on generative adversarial imitation learning.
Demonstrations and preferences. We define T consecutive pairs of states and actions trajectory& { = {s1, .i.... ST.. We also interchangeably use the term rollout to refer to a trajectory. also consider trajectory segments, which are shorter sections of the full
Besides full trajectories, trajectory with length Teeg < T. A trajectory that is generated by a teacher also referred to as demonstration. Unless stated otherwise, we assume that the teacher provides optimal examples of a given behavior. Preferences are labeled pairs of trajectories E2), such that &1 is preferred over &2. In contrast to demonstrations, where the teacher responsible for generating the trajectory, trajectories in a preference pair are generated by the agent. We assume that the preferred trajectory has higher likelihood under the expert policy π (Wirth al. 2017). i.e.,
E > & + p (E1(TT) > (E2)=T)
which also implies that the cumulative reward of trajectory & is greater than the reward of Ez, that is,
T
R(G1) > R(G2), R(g) Σ r(s az),
1=0
where r(s, a) is reward function that is unknown to the agent.
In the literature, such as by Lee et al. (2021a), a common approach for defining the loss function is based on Brarley-Terry model (Bradley & Terry 1952) where they model
exp Σr t= r rss] a.)
PBT(Ei > (2)
Σ
Ei=1 exp Σt 755)
3

---

Published as a conference paper at ICLR 2023
Based of the previous preference probability and assuming that the preferred trajectory is always labeled as E1, the following loss function is used
L LBT
E [ogPET(S E2)
(51-52)~P
Preference query and labels. Preferences are obtained by sampling trajectories from the policy, or alternatively, they can also be sampled from buffer of existing trajectories, [ee et al. (2021b) ana- lyze different preference querying methods in order to obtain more informative pairs. The approach that we consider in this paper is based on cross-entropy. We select a pair of trajectories from the buffer that has the largest cross-entropy value. After the agent generates the query, it is presented to the teacher for evaluation. The teacher then labels one trajectory as better according to its evaluation criteria. In simulated tasks, the teacher is often referred as oracle and it has access the true underlying reward function of the task.
Trajectory likelihood. Assuming Markovian dynamics, the likelihood of the trajectory under the expert policy is
T-1 p(E|T))= = ((0)) II p(s st)=" (agls)).
(1)
Adversarial Imitation Learning. The main training loop of AlL revolves around iteratively train- ing a discriminator D on samples, and training generator, which is usually given as a policy R(]ss) on some reward induced by this discriminator. The discriminator and the policy updated sequen- tially in a loop after minimizing their respective losses as described below.
At the start of iteration k, we generate set of rollouts M₁ { following policy (als).. Using M2 and expert demonstrations = {Eat that are provided by an expert teacher x (als), update the discriminator ((s a.) A S R such that it distinguishes between expert demonstrations (s, a) E and samples (s.a) E .MM- After training the discriminator, it can be used recover an intermediate reward function R2 (s, which subsequently used obtain new policy 1 (a) s) using a regular Reinforcement Learning algorithm, such as Soft Actor-Critic (SAC) (Haarnoja et al. 2018). The exact form of [s, (s. a) varies depending on the concrete AlL algorithm. Popular choices include RGAIL (a, s) = - loog D(a, s)) from GAIL (Ho & Ermon 2016 and RAIRL sa
log(1 D(a, s)) from AIRL (Fu et al. 2018). For more details we refer to Orsini et al.) (2021). The former reward optimizes the Lensen-Shannor diver- [a]s and π (a|s) (). while the latter optimizes the Kullback-Leible divergence between the same distributions. After updating the policy, the above process is repeated. The algo- rithm converges when the policy creates samples that are indistinguishable from that of the teacher. At this point, the policy successfully imitates the behavior of the expert teacher.
(ag
gence between the ( )
4 ADVERSARIAL IMITATION LEARNING WITH PREFERENCES
Our approach has two main steps. In the first step, instead of a discriminator, we update a log density ratio estimator (LDRE) using the demonstration and preference loss functions that we introduce in Section 4.3 In the second step, we update our policy using the IDRRE based reward and SAC. Every K iterations, we add M new preferences to our preference buffer. We summarize the algorithm in Section 4.5
4.1 REFORMULAATIGG AIL USING DENSITY RATIO ESTIMATION
AIL can be derived under divergence minimization perspective such as the gence (Ho & Ermon 2016 Orsini et al. 202I), KL -i-divergence (Fu et al.) 2018 Becker et al.) 2020)
Diver-
or more general f-divergences (Ghasemipour et al. 2020). We will follow a Kl-minnimization per-
spective as it allows a clear interpretation of the learned discriminator (Arenz & Neumann 2020).
Similar to Fu et al. (2018); Becker et al. (2020), we start by minimizing the KL-divergence between the generator's is and the demonstrator's is trajectory distributions, i.e.
T
π² = -argmin (p" with (²(()) = p(so) II
p(3 186, (erls)))
(2)
0=0
4

---

Published as a conference paper at ICLR 2023
The distribution (E) is defined similarly with unknown demonstrator policy g(als). As demon- strator and generator trajectories use the same system dynamics, the KL minimization simplifies to
(I)(2)())l))EE)) = (T) Σ log ((())()) -
T
de
(3)
g(ac(s))
Σ |l (0s))))((alos))) log
π. ((rl)t)) ds,dor:
(4)
g(a)))))
where p" (s;) is the state distribution of policy at time stept. t. We can see that this objective depends x(ax/s:) of the generator and demonstrator action distributions. This log on the log density ratio log q(a.(=)) density ratio can be estimated by discriminator which is typically trained by a BCE classification loss [Becker al. (2020); /Arenz & Neumann (2020). Yet, in order break the dependency of the
discriminator on the generation policy π and following Arenz Neumann (2020), we introduce the following trick
T Σ (si) ((aa)ss) log
Th (acls)))) g(ar|se)
+ log T(eess)) (arl())))
(L)
ds,das,
(5)
where Ta the old generator used for sampling the data. In the following, we will replace the log
density ratio log TA (az|$,) log g(ue(s:) by an estimator o(s, a) which is trained using the BCE objective. We further rewrite our minimization problem into an maximization problem and arrive at
Σ Ep (as)mtaaa)))) (ss, az) + log ml + (=(-ss)))
(6)
where (-)11s) is the conditional entropy of policy æ. Note that J now needs to be maximized w.r.t.
4.2 REWARDS IN AIL
While in standard AIL we typically use the discriminator as reward signal for updating the generator, it is important to note that the discriminator does not constitute reward signal in the classical sense. is non-stationary and heavily depends on the current generator policy тę- It provides information which trajectories should be reinforced given the current generator, i.e., policy T& but can not be used to evaluate the quality of trajectories independently of the currently used generator. To illustrate this issue, consider the case that our model policy is perfectly matching our expert, and that the discriminator has been trained accordingly. In this case, the discriminator outputs 0.5 for every possible input, thus being incapable of providing a metric compare different trajectories.
Yet, a different reward formulation can be extracted from our discussion on density ratio estimation in AIL. Eq. can be interpreted as average reward reinforcement learning problem with rewards
(5 ax) log T (a (se) o( sr, (())
and additional max-entropy objective for the policy. It is casy to that, for perfect density ratio estimator, i.c. o(s,, a.) = log T& (ayls,) - log q(a,) s:). the reward function would reduce to log q(a,)a st),, breaking the dependency of the reward function definition in AlL from the generator Ta (ar]sz). Hence, we can now directly use our reward definition to evaluate trajectories which will be important in Section 4.3 for introducing an additional preference loss.
4.3 COMBINING PREFERENCE AND DEMONSTRATION Loss
Following discussion from above, the demonstration loss for training the discriminator is speci- fied by the BCE loss which results in reliable density ratio estimators, i.e.,
Ledem (b) D,Mo) = (a.x)>
[log a (á(a, s))]
E (a.s)aM
[log (1-0 s)))]
(8)
5

---

Published as a conference paper at ICLR 2023
where M contains samples generated by the current model policy ft.
Furthermore, we need to include a loss function for the available preferences to obtain a more ac- curate discriminator. To this end, will use reward function defined in Eq. to estimate the trajectory return
R(E) - E r(st
Σ log Th (aelss) - dfst, as),
of the two trajectories & and E2 that constitute the preference pair. We now further restrict our discriminator also comply with the given preferences using hinge loss per preference pair,
R Lepres P) (E1-E))~P
[hinge (Ri ((=))),
i.e. there is an additional penalty if the estimated return of the preferred trajectory R(Ei) is smaller then the return of R(E2). In our experiments, we also present an ablation study testing different loss functions for the preferences, including without addition of the log-density of the old generator in our reward definition as well as the typically used sigmoid loss for the preference pairs. We observed that the hinge loss resulted in the most stable results. Our interpretation is that this is due to the property of the hinge loss that it is only active if the preference is violated while the sigmoid loss is forcing the returns of two preference pairs be far apart, causing instabilities.
Preference and demonstration feedback may impact the learnt policy to different degrees depend- ing on the teacher and the task. We can compensate for this effect by increasing the influence of one feedback type by scaling the corresponding loss function. For example, we can multiply the preference loss Lpref with hyperparameter preferences, our method represents a novel preference-learnig method.
do so. In the case when we are only considering
4.4 OPTIMIZING THE GENERATOR
Although there are algorithms for solving average reward RL problems (Zhang Ross 202T), we will use standard discounted RL formulations such as SAC (Haarnoja al., 2018) for policy update as such off-policy methods are known to be more data efficient. While our derivations only hold for the average reward case, we do not expect this make major difference for larger discount factors. In this paper we concentrate on the addition of preferences in the discriminator loss and leave the usage of average reward RL algorithms for AIL for future work. We SAC to optimize the reward, which is an off-policy method that uses a replay buffer B. However, because our reward definition changes with o(a, 8) we need to also relabel the data in the buffer accordingly.
As we use Gaussian policy class, the term log TA (arls)) used in the reward definition was also too restrictive for SAC as policy optimizer as this term quickly tends dominate, making the value function approximation for the first part of the reward unreliable. Hence, for policy optimization, we directly used - a) as reward signal. Note that the resulting average reward RL problem from Eq. can also be reformulated as
JR
Ep (o))(aa))))))) (685, at)] + Bep (ox) [KL T (+)s))))
(10)
ie., we get an additional KL regularization term that punishes the derivation to the old policy. While there are several on-policy RL algorithms that directly use such KL regularization terms for their policy update (Schulman et al., 2015 2017), we resort to off-policy actor critic methods such as SAC. In SAC, we still have the maximum entropy formulation, but we ignored the KL term. As the KL-term is a 'moving' objective that prevents the policy from making too large updates, this assumption should in theory only affect the learning speed, not the quality of the found policy. The resulting optimization corresponds to the optimization performed by AIRL (Fu et al. 2018). as presented in Orsini et al. 2022).
4.5 RESULTING ALGORITHM
The input to the algorithm is a set of demonstrations D and set of preferences P. which may be empty sets as well. In order to have more informative trajectories for preference evaluations already at the beginning of the learning, pretrain the policy स maximize state entropy H(s)

---

Published as a conference paper at ICLR 2023
-E(log p(s)), in Lee et al. (2021a). Without this step the initial trajectories would be completely
random, and comparison among them is less insightful.
In each iteration of the main learning loop, we first collect additional demonstrations and request preference queries, and update the feedback sets D and P. Note that this is an optional step, as update the generator policy using only the existing sets 'D and P. To generate significantly different trajectories for preference evaluation, we first sample a batch of trajectory pairs ((E,.) X S following the policy म. From these we select the pair that has the largest entropy H (P >
G,a)), where P(C > {ソ,2) = softmax ((()) 1), 2)(), as in Lee et al. 202 The R(g) an estimated reward function, described in more detail below. Wee present the trajectory pair to an expert to obtain the preference feedback. The expert uses its criteria to compare and label the two options. In simulations we use an oracle that labels preferences according to the defned environmental reward of the trajectories. The reward values of these trajectories are never directly provided to AILP. In the next stage we update the LDRE Φb. We then collect the samples from the replay buffer B and afterwards, we compute the demonstration loss Ldem and the preference loss Ly and optimize them w.r.t. 6. We use the optimized LDRE to define an expert reward RAIL and optimize the generator स using SAC. An overview of this procedure is shown in Figure [ and we additionally provide pseudocode in Appendix
5 EXPERIMENTAL EVALUATION
With our experiments would like to answer the following questions Q1: How does AILP com- pare with the current state of the art in preference learning? Q2: What is the influence of including expert demonstrations on improving policy learning in addition to preferences? Q3: What is the influence of using imperfect demonstrations on policy learning in addition to preferences?
We consider 6 different manipulation tasks from the metaworld benchmark (Yu et al., 2019), namely button press, door open, door close, window open, window close, drawer close. For comparison with other work we consider episodes of 250 rather than 500 steps, which is still sufficient to finish the task successfully. For evaluating the performance we consider normalized trajectory returns as well as task success rate. The success rate is 1 when a task-specific success criteria is achieved, and O otherwise. We report the average success rate over 10 rollouts. For computing the normalized trajectory reward, we sum all the environmental rewards of individual steps and then normalize by dividing the resulting sum with the average trajectory reward of the expert. For this metric we also consider the same 10 rollouts and we calculate their mean and standard deviation. In all experiments use the same set of 10 random seeds. For each rollout from task the position of the object in the scene is randomly set, c.g., location of the button in the button press task. For more details about different environments refer to the Appendix A
Peoble ALLP(ddcm)
AIRL
BC+Pabbie
#C+reitheeas)
ALPieol
ALP0mp!
Stymold only LDRE prof.
AIL
ALL.)ddmm)
Drox-
100
100
300
Success Rate 1N
60
80
60
80
80
20
28
0.01
2.5
4,0
0.01
5.0
Bio
0.01
2.5
4,0
60
Enw. stepstte5)
Env. steps1le5)
Env. stw.dtppsle]
(a) Comparison of AILP and vari- (b) Comparisons when using expert ous basclines.
(c) Comparisons of AILP with dif- ferent preference loss functions.
and imperfect demonstrations.
Figure 2: Interquartile mean for aggregated success rates for 6 metaworld tasks. Compared is our proposed method AILP with relative baselines and performance of AILP with modified preference loss function.
7

---

Published as a conference paper at ICLR 2023
To answer Q1, we have compared our method to the Pebble algorithm (Lee al. 2021a). We use the official implementation which is contained in the same code repository as for LLee et al. b For all evaluation we have used the parameters listed by (Lee et al.] 202 Ib). In all our evaluations and for all methods we use the same network architecture for the actor and the critic of SAC, details in the Appendix C However, we use different networks for the reward function in Pebble and for the LDRE in AILP. Namely, for Pebble we use the network size from the the paper that consists of 3 layers with 256 nodes each. For AILP we use a smaller network with layers with 64 nodes which is the best performing network architecture for AlL (Orsini et al. (021). Pebble uses an ensemble of 3 reward networks which is motivated by increasing the stability of reward learning. However, we found no benefit from having an ensemble of LDRE networks in our method, and due to the increased computation time, we decided to train only a single network.
In order to answer Q2 we add one expert demonstration to the start of learning process. Afterwards, in our experiments we do not add any demonstrations. Additionally, we also included the case of using our algorithm without any preference, in which AILP is practically combination of AIRL and SAC Moreover, Orsini et al. (2021) show that the optimally performing AIL algorithm is also based on the combination of those two methods, and therefore it is the best option for AIL baseline in our paper. We can see in Figure jtthat AlLP is successful in learning every task with the combined feedback. Using demonstrations helps significantly in the beginning, thus increase the speed of the whole learning process. Moreover, we also tend to learn much faster in comparison to other baselines. Without preferences (AIRL+SAC) learning saturates early and success rates remain low. Additionally, we compared our method with two extensions of Pebble that consider demonstrations. In one case we used demonstrations initialize the model policy using behavior cloning and start with preference learning using Pebble from that point. This approach is inspired by the work of Ibarz jtt al. (2018) but we also rely on the improvements provided in Pebble such as using SAC to optimize the reward. Note that in this case we do not have the entropy-based pretraining. Moreover, we also consider a combination of DREX and Pebble, in which we use the approach from DREX to generate the initial set of preferences, and then we apply Pebble from that point onward. We refer to this method as DREX+ In addition to the success rate presented in Figure 2 we report the values of normalized trajectories returns in Appendix D.I.
Additionally, in Appendix D.4 we present similar evaluations on a more difficult metaworld task, lever pull. Furthermore, we also evaluate the performance in Mujoco task, HalfCheetah (Todorov ct al. 2012). Details and results are in the Appendix [.5]
After the initial pretraining of मे we start adding 20 preferences each 5000 environmental steps, which equates to 20 full trajectories. In the case of Pebble the reward is updated only when we add new preferences. However, in the case of AILP because the LDRE depends on the current policy we need to update it more often, and we do that every 20 environmental steps. The updates are computationally expensive, especially since we increase the amount of preferences during training. Therefore, we limit the amount of previous preferences to 300 during training. We assume that the remaining, more recent preferences provide relevant information to guide the learning. Con- sidering that the early preferences are sampled from older policies their contribution to learning is questionable if any. Moreover, given the frequent updates of the reward function training on fewer preferences would benefit us from the point of compute time. Entropy-based pretraining is per- formed for all baselines that are not initialized using BC and it is done once for each random seed and then shared for all experiments. Note that the performance of the algorithm from this initial part is not shown nor is counted in the steps shown in Figure Moreover, in our comparisons we utilize interquartile means (Agarwal et al. 2021) over the 6 aggregated metaworld tasks more robust statistical measure.
For answering Q3 we use partially trained policies to generate imperfect demonstrations. None of the policies was on average capable of successfully finishing the task. Using those demonstrations we ran the same set of experiments as before. The results are shown in Figure 5bb AILP learns the task successfully even in the presence of imperfect demonstrations and still benefits from the demonstration in comparison to the case with demonstrations.
https::///// /i thub. com/ wwtp//ow./ppp..ooe pok.xpoka/aaakk..o /B_rrf
Here, we used standard discriminator instead of the policy dependent discriminator architecture from the paper Fu et (2018). We were not successful using the original discriminator architecture together with SAC, yiclding poor performance. Note that the ATRL. original paper, TRPO was used to train the policy.
8

---

Published as a conference paper at ICLR 2023
Ablation of the Preference Loss. Finally, present ablation study regarding the used prefer- ence loss. Here, we ablate the sigmoid preference loss function used by Pebble (with using the log policy densities
additional rewards) as well as algorithm using the hinge loss without the log policy density as reward. The results are shown in Figure [c] The sigmoid loss enforces that the rewards of two trajectories in preference pair distinct, which causes instabilities in particular if the quality of the trajectories is quite similar. Here, the hinge loss shows much more robust behavior and the agent's performance converges gracefully. Not using the log policy densities as additional rewards and hence treating the LDRE on its own as reward function for preferences also leads to a poor performance the LDRE not stationary reward function.
8
58.3
96.7
01.7
22.3
28.3
8-
58.3
102
05.7
92.2
78.3
103
95.7
92.3
95
78.3
98
98.3
15
96.7
35.3
10
71.7
50.3
91.7
100
25.3
26.7
71.7
96.7
05.7
98.3
120
100
91.7
91.7
00.3
73.3
01.7
91.3
913
8.23
70.7
88.3
363
6.23
58.3
93
21.7
0
demor
cemor destion
(a) Using expert demonstrations
(b) Using imperfect demonstrations
Figure 3: Success rates for AILP with different numbers of demonstrations and preferences for the window open task with expert and imperfect demonstrations. Adding additional demonstrations and preferences leads to orthogonal improvements in performance.
Evaluating the Trade-off between Number of Preferences and Demonstrations. We evaluate the performance of AILP for different amounts of preference and demonstrations the window open task in Figure We find that additional demonstrations generally improve performance, particularly for the case with low number of preference. In the aforementioned figure we indicated the the number of preference that are added after 500 environmental steps. In the same figure we can compare the cases when we have expert and imperfect demonstrations. we use more demonstrations. the number of preferences can be reduced in order to achieve good performance. The contribution of additional demonstrations to the algorithm performance is most evident when we have a smaller number of preferences. Additional results over all 6 tasks with different number of demonstrations are provided in Appendix D.2 D.2
can be observed that
6 CONCLUSION AND FUTURE WORK
We introduce Adversarial Imitation Learning with Preference, novel method for learning from demonstrations and preferences. We modify and extend well-known Adversarial Imitation Learning methods to also be able to learn from preferences. This is achieved by using a log density ratio estimator, which is trained on separate losses for demonstrations and preferences. After training, the resulting network is used to express reward function, and then used to optimize the generator policy. This procedure is iterated over until convergence and allows for novel teacher feedback in each loop. We show suite of simulated robotic manipulation tasks that this approach to learning from preferences benefits from multiple types feedback, outperforming the state of the art.
A limitation of our proposed method is the number of environment steps needed when updating the generator policy which is challenging on a real robot. future research, we will investigate the use of multiple trajectories and ask for rankings from the teacher (Myers et al. (021). Additionally, will expand AILP to deal with preference queries where two options are too similar for teachers to compare and will also evaluate approach with human-generated data.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
GN was supported by the Carl Zeiss Foundation under the project JuBot (Jung Bleiben mit Robotern). NF was supported by the BMBF project Davis (Datengetriebene Vernetzung für die ingenieurtechnische Simulation).
REFERENCES
Pieter Abbeel and Andrew Ng. Inverse reinforcement learning. Encyclopedia of Machine Learn- ing and Dala Mining, pp. 678-682 2017.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. volume 34, pp. 29304-29320, 2021.
Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to ad- versarial methods. CoRR, 2020.
Brenna D. Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57:469-483, 5 2009.
Philipp Becker, Oleg Arenz, and Gerhard Neumann. Expected information maximization: Using the i-projection for mixture density estimation. International Conference on Learning Represen- tations, 2020.
Erdem Biyik, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active meference-based gaussian process regression for reward learning. Proceedings of Robotics: Science and Systems (RSS), 2020.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. International Con- ference Machine Learning, 4.
Daniel S. Brown, Wonjoon Goo, and Scott Niekum. Better-than tatter-han-ddmmosttattte imitation learning via automatically-rankedd demonstrations. Conference Robot Learning (CoRL), 2019.
Erdem Bıyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. Con- ference on Robot Learning, 2018
Erdem BIyık, Dylan P. Losey, Malayandi Palan, Nicholas C. Landolfi, Gleb Shevehuk, and Dorsa Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrat- ing demonstrations and preferences. The International Journal of Robotics Research, 6 2021.
Sonia Chernova and Andrea L Thomaz. Robot learning from human teachers. Synthesis Lectures on Artificial Intelligence and Machine Learning, 8:1-121, 2014.
Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 2017.
Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. Conference on Machine Learning, 2005.
International
Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. Confer- ence on Robot Learning (CoRI.) 2021, 2021.
Niklas Freymuth, Philipp Becker, and Gerhard Neumann. Versatile inverse reinforcement learning via cumulative rewards. 4th Robot Learning Workshop at Neural Information Processing Systems (NeurIPS), 2021.
10

---

Published as a conference paper at ICLR 2023
Niklas Freymuth, Nicolas Schreiber, Philipp Becker, Aleksander Taranovic, and Gerhard Neumann. Inferring versatile behavior from demonstrations by matching geometric descriptors. Conference on Robot Learning (CoRL), 2022.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. Sixth International Conference on Learning Representations ICLR, 2018.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective imitation learning methods. In Conference Robot Learning. 2020.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Advances in Neural Information Processing Systems 27, 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in Neural Information Processing Systems 30, 2016.
Neil Houlsby, Jose Miguel Hemmández-Lobtto Ferene Huszár, and Zoubin Ghahramani. Collab-
orative gaussian processes for preference learning. Advances in Neural Information Processing Systems, 3:2096-2104, 2012.
Borja Ibarz, Geoffrey Irving. Jan Leike. Shane Legg, Tobias Pohlen, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in Neural Information Processing Systems, 201 8- 8-Deeem:80111 18-223 2018.
Hong Jun Jeon, Smitha Milli, and Anca Dragan.
( (implicit ) choice: unifying
formalism for reward learning. Advances in Neural Information Processing Systems 33, 2020.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. International Conference on Machine Learning, 2021a.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. NeurlPS Datasets and Benchmarks Track, 2021b.
Vivek Myers, Erdem BIyIk, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. Conference on Robot Learning (CoRI.,, 2021.
Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? Advances in Neural Information Processing Systems, 6 2021.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7:1-179, 2018.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn- ing of reward functions. Robotics: Science and Systems, 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1107.003447 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026-5033, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. Proceedings of the ICML Workshop Imitation, Intent, and Interaction.

---

Published as a conference paper at ICLR 2023
Faraz Torabi, Garrett Wareell, and Peter Stone. Behavioral cloning from observation. International Joint Conference on Artificial Intelligence (IJCAI 2018), 5 2018.
Nils Wilde, Erdem Brysk, Dorsa Sadigh, and Stephen L Smith. Learning reward functions from scale feedback. Conference Robot Learning (CoRL), 2021.
Christian Wirth, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- rein- forcement learning methods. Journal of Machine Learning Research, 18:1-46, 2017.
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning. 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and eval- uation for multi-task and meta reinforcement learning. Conference on Robot Learning (CoRL), 10 2019.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. Conference on Robot Learning (CoRL), 2021.
Yiming Zhang and Keith W Ross. On-policy deep reinforcement learning for the average-reward criterion. In International Conference on Machine Learning, pp. 12535-12545. PMLR, 2021.
Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. AAAI, 2008.
12

---

Published as a conference paper at ICLR 2023
A METAWORLD ENVIRONMENTS
All tasks in the metaworld benchmark (Yu et al. 2019) share the same observation and action space. The observation space is 39 dimensional and consists of the of the current and previous state and desired position, with the last element related to the position of the goal object, such as the button in the button press scenario. Each state has 18 dimensions and consists of the end-effector position (3 dimensions), distance between the two fingers of the gripper (1 dimension), Euclidean position (3 dimensions) and orientation expressed using 4 dimensional quaternions of object 1 and the same for an object 2. Amongst considered tasks we only have environments with a single object, and the values of the position and orientation are set to zero. This object is defined by the task, e.g., in the button press is the button. The action space is defined by controls of the position of the end- effector well the distance between gripper fingers. The orientation of the end-effector is fixed such that it always points towards the table. Using environmental rewards and SAC we have trained expert robot policies and the mean and standard deviation of trajectory returns over 100 rollouts are presented in Table
Task
Expert
Imperfect
Button press
1773.0
614.7
Door open
2015.9
769.9
Window open
1982.9
681.3
Door close
2024.0
1166.4
Drawer close
2032.2
1473
Window close
2032.2
395.5
Table 1: Mean of trajectory returns 100 rollouts from expert and imperfect policy.
Button press
(b) Door open
(c) Window open
(d) Drawer close
(e) Door close
Window close
Figure 4: Manipulation tasks from the metaworld benchmarking task set (Yu et al. 2019). (a) In the button press task, the robot needs reach and press the button. (b) In the door open task, the robot needs to grab the handle of the door and move it to the green point, thus opening the door. (c) In window open task, the robot needs to move the handle to the left edge of window. The button, the door, the drawer, and the window are randomly placed in front of the robot for each rollout. (d) In the drawer close task, the robot needs to close the drawer. (e) In the door close task, the robot needs to move the door to the green point, thus closing the door. (f) In window close task, the robot needs to close window. The button, the door, the drawer, and the window are randomly placed in front of the robot for each rollout.
13

---

Published as a conference paper at ICLR 2023
B FULL ALGORITHM
The AILP algorithm with all individual steps in shown in Alg
below.
Data: demonstrations D {Si) preferences P 1 &,2 bootstrap initial policy (s) to maximize state entropy (()
i 0
while max iterations do
Record demonstrations and preferences, update and
Generate samples from current policy #z (-(ss) and add replay buffer B
Step 1: Update LDRE o;
Get model samples B; from the replay buffer B Optimize demonstration loss Leem (0), D, Optimize preference loss Lpres (Ri) P) Step 2: Update policy
Relabel buffer using (((,aa) a)
Using SAC with - ( a) as reward, update #;+1
i+i+1
end
Algorithm 1: Adversarial Imitation Learning with Preferences. In each iteration we generate samples from the current policy (a(ss,) then we optimize the demonstration and preference losses update our LDRE Φ(8,a), Optionally, depending on the iteration we generate new preference samples or add demonstrations. After we have updated LDRE e(s, a) we use it to define a reward function R(s, a) that then optimized using SAC. Using the updated policy we generate new samples and reiterate over the whole process.
C SAC HYPERPARAMETERS
In all evaluated experiments in Section we use the same parameters for SAC and those are listed in Table 2
Hyperparameter
Value
Learning rate
0.0003
Optimizer
Adam
Critic target update frequency
2
Discount factor
0.99
Batch size
512
Initial temperature
0.1
(81,82)
(0.9,0.999)
Actor network (nodes per hidden layer)
[256,256,256)
Critic network (nodes per hidden layer)
[256,256,
Critic EMA T
0,005
Table 2: Hyperparameters for SAC that are shared for all experiments
D ADDITIONAL EXPERIMENTS AND ABLATIONS
D.1 NORMALIZED TRAJECTORY RETURNS
we have also analyzed the normalized trajectory returns for the 6 metaworld tasks. Those results presented in Figure 5 We observe same trends and relative performance between our algorithm and the baselines as we see in success rate results.
In addition to the success rate that is presented in Figure 2
14

---

Published as a conference paper at ICLR 2023
AlLP(Osem) AlLP(tdem)
BC+Pebole Proxi
SAA
AtPlosel
ALMinat
Sigmold
AIL
LDRE pref.
2.0
torp Rettur
Tralecory
0.0
2.81
0.5
0.6
0.4
2.4
0.2
0.2
0.0
0.0
0.0
0.51
28
4.0
5.0
0.51
2.0
4.0
5.0
5.0
0.01
2.0
4,0
5.0
50
2555[1e55
stops/Le5)
stops(1c5)
(a) Comparison of AILP and vari- ous baselines.
(b) Comparisons when using expert and imperfect demonstrations.
(c) Comparisons of AILP with dif- ferent preference loss functions.
Figure 5: Interquartile mean for aggregated normalized trajectory returns for metaworld tasks. Compared is our proposed method AILP with relative baselines and performance of AILP with modified preference loss function.
.dam
100
Normalized Trajectory Return
1.0
Success Rate (%)
80
0.8
60
0.6
40
0.4
20
0
0.01
2.0
4.0
6.0
8.0
0.01
2.0
4.0
6.0
8.0
steps[1e5]
steps[le5]
(a) Success rate
(b) Normalized trajectory return
Figure 6: Direct comparison of AILP with more demonstrations
D.2 DIRECT COMPARISON OF USING MORE DEMONSTRATIONS
For the default setup from Section we also an ablation with additional demonstrations over all tasks. Results are shown the Figure
D.3 INFLUENCE OF WEIGHTED PREFERENCE LOSS
We have evaluated our method when trained with the preference loss function L (ó, P) that scaled by a hyperparameter a. In Figure 7 show the performance of algorithm for values of a E 10.1, 1, 10, 100). Additionally, we include two extreme cases, AlLP with no demonstrations and AIRL+SAC that theoretically corresponds to the case of a 0. We observe that for 0.1, our method performs similar to the case without preferences (AIRL+SAC), Likewise, for 100, AIL. performs similar to the case when we have only preferences.
In Figure 8 we present the performance of our method with imperfect demonstrations. Similar on the performance of our method in the case
conclusion can be drawn about the influence with expert demonstrations, As we increase the value of a, we rely in the learning process more on preference data.
D.4 LEVER- PULL METAWORLD TASK
Besides the metaworld tasks Appendix A we additionally evaluate lever pull task. This task significantly harder, and expert policy which with obtained by training agent using SAC
15

---

Published as a conference paper at ICLR 2023
0.1
100.0
0.1
100.0
1.0
AILP(pref only)
1.0
AILP(pref only)
10.0
AIRL+SAC
10.0
AIRL+SAC
100
Normalized Trajectory return
1.0
Success Rate (%)
80
0.8
60
0.6
40
0.4
20
0
0.0
0.01
2.0
4.0
6.0
8.0
0.01
2.0
6.0
8.0
Env. steps[1e5]
Env. steps[1es]
(a) Success rate
(b) Normalized trajectory return
Figure 7: Comparison of different values of when optimizing a& pref with expert demonstrations.
61
35.0
WALVGAA
316-3
100
1.0
Normalized Trajectory return
Success Rate [%6]
80
0.8
60
40
0.4
20
0.2
0
0.0
0.01
0.01
8.0
Env. steps[1e5]
Env. 115p
(a) Success rate
(b) Normalized trajectory return
Figure 8: Comparison of different values of a when optimizing a. pref with imperfect demonstra- tions.
a.Pres
16

---

Published as a conference paper at ICLR 2023
AiPipefeident
Smo-hhokk
70
1.2
Success Rate (%)
1.0
50
Normalized Trajectory
40
30
20
0.4
0.2
0
0.0
0.01
2.0
4.0
6.0
8.0
0.01
2.0
6.0
8.0
Env. steps[]e5]
Env. s1eps(1e5]
(a) Success rale
(b) Normalized trajectory return
Figure 9; Evaluation of the Lever-pull task
with the environmental reward, was only capable successfully finishing the task 22% of the evaluated rollouts. In Figure 2 are presented the success rate and normalized trajectory returns for our method as well as for the same baseline we evaluated in Section We notice that all algorithms struggle with this task, but after some time, our method starts to learn and noticeably improve its performance, Still the trained policy on average has not reached the 50% success rate, and the variance is noticeable, but on average is performing better than the expert we used generate the demonstrations.
D.5 HALFCHEETAH TASK
HalfCheetah a common reinforcement leaning benchmark locomotion task (Todorov et al. 2012) In comparison to metaworld tasks
claim that it is a simpler task, because for each metaworld task, the initial position of a target object, such as the button in the button press task, is moved, but for the HalfCheetah
don't 'tn't have such substantial changes. Additionally, it's nonepisodic. In Figure 10 the normalized trajectory returns of our method and the set of baselines we evaluated in Section 5 are shown for this task. We notice that initially Pebble and BC+Pebble performing better, but our method achieves similar performance at the end of the training. Moreover, it is worth noting that in AIL papers, tasks are usually made more difficult by sub-sampling demonstrations so only every 20th step is considered. Under these circumstances, AIL methods perform better. How- ever, for completely fair comparison in this paper we have sub-sampled expert demonstrations. To normalize the trajectory returns we use the average return of our expert policy.
Reable
AIRL=SAC
All Edem) AlUP 2dem)
RC+
Pubblc
0.8
Trajectory return
0.6
0.4
0.2
0.0
0.01
2.0
4.0
6.0
8.0
Env. steps[1e5]
Figure 10: Normalized trajectory return various algorithm for the HalfCheetah task
17


Reviews:
[{"review_id": "nYhSfSSTqG", "reviewer": "Reviewer_qpWS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the setting of the combination of learning from demonstration (LfD) and learning from preference (LfP) is studied. Under the adversarial imitation learning (IL) framework, a new algorithm is proposed, whose main idea is to include a new ranking loss term for LfP to combine with the adversarial LfD loss. The proposed method is tested under the meta-world benchmark tasks, showing performance improvement over existing baselines.\n\n", "summary_of_the_review": "In summary, I think this is a solid paper proposing an empirically well-performed algorithm. However, I think the technical contribution is relatively weak, which is the major concern to me.", "strength_and_weaknesses": "Strengths:\n\n1. The experiments show significant performance improvement over existing baselines under the LfD + LfP setting.\n\n2. In my view, the paper is technically sound. The proposed approach is reasonable to work in practice.\n\nWeaknesses:\n\n1. I think the technical contribution is somehow limited. Comparing to existing adversarial IL approaches, the essential part of the proposed method is in Equation (9), which is a ranking loss to make use of the preference information. However, the hinge loss is one of the common choices for ranking tasks. \n\n2. From the paper, I am not quite clear why the proposed method can achieve such good performance over the baselines, since the algorithm itself does not show significant novel improvements. I also suggest including more discussions in the paper.\n\n3. I think it would be useful if the code could released when the paper gets accepted.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y0J0smp5E7p", "reviewer": "Reviewer_nvkH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel algorithm, named adversarial imitation learning with preferences (AILP), for learning from demonstrations and preferences. AILP builds upon the well-known adversarial imitation learning (AIL) framework which uses discriminator to construct reward function. To train the discriminator, AILP uses a combination of a BCE classification loss and a hinge loss per preference pair. Then, AILP uses SAC to update policy where the reward term is composed of the trained discriminator. Finally, the authors provide empirical comparison of AILP and baselines on the 6 different manipulation tasks from the meta-world benchmark.", "summary_of_the_review": "This paper proposes an interesting policy learning algorithm which uses both demonstrations and preferences. However, I have some questions about this algorithm.", "strength_and_weaknesses": "**Strength**\n\nAILP successfully handles feedback from demonstrations and preferences effectively.\n\n**Weakness**\n\n- AILP uses two losses, $L_\\text{dem}(\\phi_i,\\mathcal{D},\\mathcal{M}_i)$ and $L_\\text{pref}(R_i(\\phi_i \\mathcal{P})$ to update LDRE $\\phi_i$. However, I am wondering if a stable update of $\\phi_i$ is possible using both losses. \n- In Eq. (10), The authors remove the KL term, but the entropy term is added due to the usage of SAC. Consequently, $\\sum_{t=1}^T\\mathbb{E}_{p^\\pi(s_t)}[-\\log\\pi_k(\\cdot|s_t)]$ is removed. Does it make sense to remove this term without a theoretical analysis?\n\n**Questions**\n\n- I'm just wondering what's the problem with using a loss like $L_\\text{dem}(\\phi_i,\\mathcal{D},\\mathcal{M}_i)+ \\alpha L_\\text{pref}(R_i(\\phi_i), \\mathcal{P})$? Here, $\\alpha\\in[0,1]$ is a hyperparameter.\n- Why is there no AIL algorithm as a baseline for experimental evaluation? In addition, do baselines also use the pre-trained policy $\\tilde\\pi$ to maximize state entropy?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L9OIgJ2AQNp", "reviewer": "Reviewer_suaS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes policy training that combines two commonly used types of human feedback — demonstrations and preferences, each of which has its own strengths and weaknesses. Specifically, the authors propose a method for incorporating preferences into adversarial imitation learning as a unified framework, by reformulating adversarial imitation learning with density ratio estimation, then using the same learned density ratio estimator to learn from preferences. The authors compare their method against other preference based methods and adversarial IL methods, as well as naive combinations of imitation learning and preference based learning.", "summary_of_the_review": "I recommend a 6. The overall methodology seems sound and experimental results empirically support the author’s claim that AILP manages to combine learning from both demonstrations and preferences. The paper could be improved with more direct comparisons to PEBBLE (more diverse set of tasks, seeing if the method still works when learning from human preferences for stylistic behaviours, etc.).", "strength_and_weaknesses": "Strengths\n- The motivation is compelling and the paper is well written, easy to follow. In particular, the use of a density ratio estimator for both AIL and for judging preferences seems reasonable.\n- The empirical results seem compelling, and compared against relevant baselines (of AIL methods, preference-based methods, and naive combinations of the two). The experimental results also present with ablations on their reward formulation choice and compare the effect of providing varying numbers of demonstrations and preferences. In particular, the combination of preferences and imperfect demonstrations is an interesting setting — although in this work it does not seem to show a big improvement over just using the preferences.\n\nWeaknesses\n- The main paper of comparison (PEBBLE) shows experiments across a wider set of environments (including locomotion tasks like Quadruped, Walker, etc.) and using real human preferences. It would be an interesting and stronger comparison if those same experiments were replicated here as well, with AILP.\n- It also seems like in these domains, a single demonstration is sufficient to saturate performance. It would be interesting to find a domain where more demonstrations are actually helpful / necessary.\n- Formatting could be improved. Plots are currently a little hard to read, should increase legend size or move them off the figures.\n\nQuestions\n- Is relabelling the data in the buffer with the updated reward function time consuming and/or difficult to do as the buffer grows? How frequently is this relabelling necessary?  \n- This method assumes preferences are aligned with expert demonstrations. I would be curious to see what happens if preferences weren’t exactly aligned -- e.g. if the preferences were trying to guide for more stylistic behaviours.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster