Residual Q-Learning: Offline and Online Policy Customization without Value
Authors: Chenran Li
Chen Tang
Haruki Nishimura
Jean Mercat
Masayoshi Tomizuka
Wei Zhan
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
imitation learning

Abstract: Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. To this end, we formulate a new problem setting called policy customization. It defines the learning task as training a policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by a target downstream task. We propose a novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q-learning, which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy. We derive a family of residual Q-learning algorithms that can realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: https://sites.google.com/view/residualq-learning.

Residual Q-Learning: Offline and Online Policy Customization without Value
Chenran Li'; Chen Tang': Haruki Nishimura', Jean Mercat Masayoshi Tomizuka', Wei Zhan'
1 University of California Berkeley, 2 Tooota Research Institute, USA
{chenran_ _ln, chwnnteettttee tang, tomizuka, wzbaa)) (haruki. nishimura, jean. mem.mmtrrttrrrr global
Abstract
Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. this end, we formulate a new problem setting called policy customization. It defines the learning task as training policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by target downstream task. We propose novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q- Q-learning (RQL), which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy. We derive a family of residual Q- learning algorithms that realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: httpps://sites http:///ccceeeeeeeeiiaaaaalaaaee
Markov
Introduction
Imitation learning (IL) a widely used framework for learning imitative behavior from demonstra- tions. It is especially appealing for solving complicated real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic expert behavior. For instance, IL has been widely studied for synthesizing human-like agents from large-scale human demonstrations
challenging
real-world tasks, such as autonomous driving 4 7 and robot manipulation (59] 37 39). However, when applying the imitative policy in open real-world scenarios, we may encounter different circum- stances that impose diverse requirements beyond merely following the behavior in the demonstration, such reaching particular goals 45 enforcing safety (36), or complying with certain behavior preferences [61] In this case, we need to customize the imitative policy to meet those task-specific objectives. Meanwhile, in many applications, we may still want the customized policy possess some inherent characteristics of the expert demonstration (e.g., human-like).
Equul Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To this end, we formulate new problem setting we refer to as policy customization. In the policy customization problem, we are given a pre-trained policy, and the task to train a new policy that 1) inherits the properties of the prior policy; 2) satisfies some additional requirements imposed by a given downstream task. first glance, this newly formulated problem may seem to be seamlessly solved by existing reinforcement learning (RL) fine-tuning algorithms [44 40,
[1] S1], where RL is applied
to fine-tune a pre-trained policy obtained from behavior cloning (BC) or offline RL [1 29 34].
In particular, some RL fine-tuning methods combine different forms of IL objectives with RL to accelerate policy training, e.g., maximizing the policy likelihood on demonstration -.], regularizing the KL-divergence between the tin--tune and prior policies 440 61]. Different from the policy customization problem we consider, the IL objectives are introduced heuristically as regularization in RL fine-tuning. Solving policy customization instead requires a principled way to design the learning objective to jointly optimize the policy's ' performance on the imitative and downstream tasks, which has not been well-studied in the RL fine-tuning literature.
In this work, we propose such a principled approach to interpret and design the policy customization objective with solid theoretical basis. Formally, we formulate policy customization as Markov decision process (MDP) whose reward function is a linear combination of 1) the inherent reward of the demonstration; and 2) add-on reward specified by the downstream task. The reward weight directly determines the trade-off off between the imitative and downstream tasks for policy optimization. However, we cannot solve this MDP with existing RL algorithms, since can only access the imitative policy without knowing its underlying reward.
To tackle this challenge, we propose novel Residual Q-Learning (RQL. framework that can solve the formulated MDP without knowing the inherent reward or value function of the prior policy. Specifically, we define residual Q-function that, together with the log-likelihood of the prior policy, can conveniently construct the maximum-entropy policy solving the target MDP. We show that the residual Q-function can be iteratively learned with update rule derived from soft Bellman equation 77.. Hence, the proposed RQL method can be applied on top of any value-based maximum-entropy RL algorithms. In particular, we derive residual soft Q-learning and residual soft actor-critic- two model-free policy customization algorithms through RL fine- mone-tunig---- which are adapted from soft Q-learning [17] and soft actor-critic [18] respectively. In addition, introduce a moodel-bases policy customization algorithm adapted from maximum-entrog
Monte Carlo Tree Search (MCTS) (55), which enables zero-shot online policy customization when dynamics model is available. In our experiments, we show that the proposed RQL algorithms can effectively customize the policies toward the additional task objective while maintaining their performance the basic tasks for which the prior policies are trained.
2 Policy Customization with Residual Q-Learning
2.1 Problem Formulation: Policy Customization
We consider a MDP defined by the tuple M = AS, A.p.r. where S is the state space, A is the action space, and S A + IR is the reward function. Without loss of generality, we introduce the formulation under continuous state and action spaces. The state transition probability function p 8 x A S [0, oo) represents the probability density of the next state 8,-1 E S given the current state 8, E S and action a E A. While the following formulation applies whether the reward function and, instead, expert demonstrations are available to indicate desirable behaviors. We assume that we have access prior policy π. methods. In particular, we follow the common practice and model π as maximum-entropy policy to account for sub-optimal demonstration 660, which is especially important for human demonstration. Specifically, the policy 7 maximizes the entropy-augmented cumulative reward that is inherent in the demonstrations. The resulting optimal policy follows a Boltzmann distribution [17] as follows:
given or not, we are primarily interested in the case when r' remains unknown
A + [0, oo) pre-trained with imitation learning [T3] 22
1 Z.
exp L (
π (als)
(1)
where (s a) is the soft Q-function as defined in 17 and it satisfies the soft Bellman equation:
L ( ,a') da!
Q' a) = r(s, a) + E. s.a)
a log exp
(2)
2

---

and Z, is the normalization factor defined as /a exp (1 (s. a)) da. The temperature coefficient determines the relative importance of entropy and reward in the maximum-entrop policy [7. is straightforward to derive from Eqn. ① that:
Q' ("(s. - a log ग (als) + a log Z..
(3)
Instead of directly deploying the policy и,, we aim customize it to satisfy some additional re- quirements for a given downstream task, specified by an add-on reward function rg S x A - R. Meanwhile, we want the customized policy to inherit the characteristics of the original policy. For- mally, we aim to leverage the pre- pre-trained prior policy π to train new policy for the MDP defined by .M = (S, A, wr + rn: p). The new reward function is defined as a weighted sum of the basic reward and the add-on reward r'R so that the customized policy meets both requirements. Under this setting, the main challenge lies in how to find the optimal policy for the new task without knowing the prior reward r but only the imitative policy т.
2.2 Residual Q-Learning
Since the reward r is unknown, we cannot directly train the new policy to maximize the total reward through RI. One plausible solution is to learn the inherent reward via inverse RL [12] and then combine the inferred reward with the add-on reward for RL training. However, practical inverse RL algorithms require jointly optimizing the policy and reward function, which could be challenging to yield good performance in certain environments when compared to imitation learning [16]. Instead, aim to explore solution without the need explicitly infer the underlying reward to derive more flexible policy customization framework. It turns out that we can leverage the imitative policy π construct the new policy without knowing π 'ss underlying reward or value function. The proposed method can be applied on top of any value-based maximum- -nntrpy RL algorithms which rely on the soft Bellman update operator defined as:
[77 55].
Qu+1 a.) = rR(ss a) wr(s, a) -B à log / exp (1 da' dad);
(4)
where Qi is the estimated soft Q-function for M at the tth iteration. Instead of directly applying the soft Bellman update operator to iterate Q during policy learning, we define residual Q-function as QR, Q. wQ" and attempt to iterate this residual Q-function during policy learning. We then derive an update rule for Q R. that does not depend on the reward or value function of .:
QR,1+I(S,a)
=rn(s, a) + wr(ss a) tEx |log / exp ( (())() da' - wQ"
(5a)
=rn(s, a) wQ" (x) - ByB aoog / exp (220(6)))) da'
/ exp 1 (1(()))a))) da! a))
+yE,, à log A
(5b)
= =TR(s, a) worlEa log Zs
+ +7B. log / exp (QRa(s', a') war log x (a'ls') + wax log Zoi) da!
(5c)
A
= =ra(s, a) warks log Zo + warBs log Zo
+ 7E. log / exp ( (QRe(s', + wa lo (a'ls') da')
(5d)
(
(2 a') + log (a'ls')
= =rn(s, a) + yEa
à log
exp
da'
(5e)
where Eqn. 5bp results from expressing the reward r with the soft Q-function Q" based on Eqn. 2
Eqn. (5c) results from the definition of the residual Q-function and Eqn. (3). Please refer Ap- pendix for a step-by-step breakdown of the derivation process. In each iteration, we can then define the policy corresponding to the current estimated Q. without computing Q::
कe (a(s) x exp ( (QR,(s, a) + wa log T (alss))
(6)
3

---

One remaining issue is that we do not have access to the underlying temperature coefficient a of the prior policy и, Note that a is multiplied upon the reward weight w in Eqn. (5) and Eqn. (6). In practice, the reward weight is hyperparameter to tune. Hence, we can define w = wa and directly treat w' as the hyperparameter without the need know the true a.
Practical Algorithms
In this section, present three practical algorithms for policy customization under different settings based on the update rule residual Q-function in Eqn. (5) and the maximum-entroe policy defined in Eqn. (6). In these algorithms, the goal is to find policy that solves the MDP .M given a prior policy and the add-on reward function ra- We first introduce
model-free algorithms to learn the residual Q-function and customize the policy through additional RL steps, which are adapted from soft Q-learning [17] and soft actor-critic [18 respectively. Then introduce model-based policy customization algorithm adapted from maximum-entropy MCTS
55 which enables zero-shot online policy customization when dynamics model is available.
Residual Soft Q-Learning. We adapt the soft Q-learning [17 algorithm to a residual soft Q-learning algorithm for offline policy customization (i.e., policy customization via additional RL training steps) when the action space is discrete. The residual algorithm is essentially the same as the soft Q-learning. The only difference is that use function approximator parameterized by to model the residual Q-function instead of the soft Q-function and denote it as QR, a). At each iteration, the parameters are updated by taking gradient steps minimize the cemporal-difference (TD) error:
JQ (0) = Ten
( Qaagett (8t, az) QR, a.)
where D the replay buffer and the target residual Q-value Q Qtarget (s,, a,) is computed as: R,0
( (QR,6 a') + wt log ( ((a1)(++1))
Q Qtargot (8t, az) TR. + Vā log exp
Σ a'EAA
(8)
with being the target parameters. The above objective function can be derived straightforwardly from the original objective function of soft Q-learning. We provide the detailed derivation in Appendix B for completeness. With discrete action space, we can compute the policy distribution exactly given the residual Q-function and the prior policy as in Eqn.
For tasks with continuous action spaces, we adapt the soft actor- critic 188 algorithm to residual soft actor-critic algorithm. We aim to train parameterized residual (a)s (8) by alternating between policy evaluation and improvement. In the policy evaluation step, we update the residual Q-function for evaluating the current policy. The loss function is essentially the one in Eqn. 7 but with the target residual Q-value computed with respect the current policy नेद
Residual Soft Actor-Critte
Q-function Q2,0(s, (8. a) and a policy network a
1/ar (Br, at) = TR. + Ylaa'tt (R a') + log (a' ((4)))+)))---) - à log ã (a'|s:+1)] (9)
The formula for the residual Q-value is derived from the modified Bellman update operator of soft actor-critic [18] following similar procedure in Eqn. ( The detailed derivation is attached to Appendix for completeness. In the policy improvement step, we improve the policy म. leveraging the current estimated residual Q-value QR, KL-divergence between it and the desired maximum-entropy policy given the current QR,((8, a). which is equivalent to minimizing the loss function below:
(s,a) a) and the prior policy by minimizing the expected
J+ (Ø) Ea [aaniy (6 log (al)s)) QR,0(8t, a) w/ log π (a]s:))]]
(10)
In our experiments, we implement the algorithm based on the practical soft actor-critic algorithm presented in [19], which incorporates auto entropy adjustment and double residual Q-functions (14)
Residual Meximum-Entrope MCTS. The last algorithm we introduce is a model-based policy customization algorithm adapted from the maximum-entropy MCTS (55) algorithm, which we refer to as residual maximum-entropy MCTS. Our primary interest is in applying this algorithm in the online setting with an offline learned dynamics model. MCTS essentially solves an online planning

---

problem leveraging the prior policy and add-on reward function, which enables zero-shot online policy customization. Such a paradigm is particularly useful for problem domains like autonomous driving, where large-scale human driving datasets [II 58 5
are available for learning good imitative policies [ 46] as well as accurate traffic predictive models for online planning [T). [3]. Note that MCTS- methods inherently assume a discrete action space. In practice, we may extend it to tasks with continuous action space by discretizing the action space [47 57).
To solve the online planning problem, maximum-entropy MCTS constructs a look-ahead tree T incrementally via Monte Carlo simulation. In the tree T, each node n(s) T corresponds to a state 8. For each action E A, the node n(s) stores an estimated soft Q-value Q(s,, a) and a visit count N (ss, a). At each iteration, simulations are conducted starting from the root node of the search tree, which consists of two stages: 1) querying a tree policy to select actions and traversing the tree until a leaf node is reached; 2) querying an evaluation function (e.g., Monte Carlo simulations leveraging roll-out policy) at the leaf node to estimate the return-to-go. After simulation, the estimated returns propagated backward to update all the nodes along the simulated paths. Afterward, the tree is grown by expanding the leaf nodes reached during the simulations. Different from a typical MCTS algorithm [8) maximum-entropy MCTS exploits maximum-entropy policy optimization. Hence, the tree policy used in maximum-entropy MCTS is defined as:
exp( 2 (s)
A(als) = (1- En'ed exp( ((p
A
(11)
A
where A₂ =<Al/oogEE N(a.a)11 with as hyperparameter. Given simulated trajectory from the root node to a leaf node, i... {50, ", ST. a the Q-values of all the visited nodes are updated with softmax backup:
FR(St, az) + wr(s; a.) + rh
t - TR(St, a:) + wr(s; a.) + yā log En'EA exp ()))))))))) <<T-1, (12)
Q(5,, a.) =
where A is the return-to-gi to-go estimated by querying an evaluation function on the leaf node n (ST))
To adapt the algorithm to residual maximum-entropy MCTS, we instead have each node store an estimated residual Q-value Qn(8, a)). The tree policy is revised as:
exp( (QR(s, a) + w' log=(als))
1
#(als) = (1 - Xa)
+ X
(13)
En'ed exp( (Qne(s, a) + wt log (als)))
|A]
During backward propagation, the residual Qavalue is updated as follows:
(TR(Sr, ar) + + (nn(se, at) + >Vn(8+11)
t= T 1,
QA(s,, oz)
(14)
1 <<T-1 - 1,
with (8( = â log En'e. E.A exp (1 (QR(8 a') + w' log (aa (841)))))))))))) In particular, we use Monte Carlo simulations to estimate the return-to-go R. The roll-out policy is set as the prior policy т, and R is computed as the cumulative return of the add-on reward I'n over the roll-out trajectories. While the estimated R is biased, it serves as heuristic to guide the tree search to select nodes with future trajectories collecting high add-on rewards while staying close to the prior policy. The effect of this biased estimation will gradually be mitigated as the search tree expands to the terminal time step.
4 Related Work
Our residual Q-lcarning framework can be considered a principled approach to combine the objectives of IL and RL. Prior works have proposed different approaches for this purpose. In the IL literature, reward augmentation [9] has been introduced as a general framework to provide additional incentives from prior knowledge into the imitation learning process. Under the Generative Adversarial Imitation Learning (GAIL) framework, a surrogate reward can be conveniently incorporated to augment the leamned discriminator for policy optimization [35] B. The augmented imitative reward is essentially the same as the total reward in our framework. The difference is that ours aims to further customize a trained imitative policy for downstream tasks, instead of regularizing IL with prior knowledge.
There are also a variety of works that directly design the objective function as a weighted mixture of IL and RL objectives. In some methods, like DQID (21). supervised classification loss is combined
5

---

with TD loss to train the Q-function. It regularizes the learned Q-function to assign higher values to demonstrated actions. Other methods, such as DAPG 44 and BC-SAC [36. use a weighted mixture of behavior cloning loss and RL objectives in their loss function. SHAIL 25
formulates the imitation problem as a constrained RI. problem with a constraint on the lower bound of policy likelihood on demonstrations. These approaches have shown that II can help accelerate RL. learning, especially under sparse rewards [21] [4], and vice versa [25]. As discussed in Sec. I similar objectives are also adopted in some RL fine-tuning algorithms [44 40 51]. The pre-trained policy is not only used to initialize RL but also leveraged consistently over the fine-tuning stage as regularization. The introduction of RL has also been shown to effectively robustify behavior cloning [6] for challenging and safety-critical autonomous driving tasks. Conversely, IL regularization is used in offline RL approaches. such as TD3+BC [ and CQL BII to prevent overestimating the values out-of- distribution actions and improve the robustness of offline RI policies. Different from these works, where the additional objective is treated regularization, our framework formulates the learning task as a new MDP and provides a principled way to interpret and design the combined objective.
TS2C [56] is similar to ours from the perspective of having imperfect expert-the prior policy in RQL can be considered as an imperfect expert for the overall task-and fusing knowledge extracted from this imperfect expert and RL exploration. The difference is that TS2C aims to train a policy towards a known reward; Thus, a value function can be trained to assert if the expert is reliable. In case, since we rely on the prior policy to encode the basic task reward implicitly, we cannot use such a value function to govern the fusion between imitation and RL. One work in the literature that aligns with motivation behind MCTS online customization is Deep Imitative Model 45 which aims to direct an imitative policy to arbitrary goals during online usage. It is formulated as a planning problem solved online to find the trajectory that maximizes its posterior likelihood conditioned on the goal. In contrast, we formulate the additional requirements as rewards instead of goals, which are more flexible to specify, Also, RQL is unified framework for both offline and online customization.
In curriculum RL, the concept of learning the residuals of Q-functions has been adopted in boosted curriculum RL (BCRL) 28
Under the BCRL framework, task curriculum is constructed by decomposing a complex task into a sequence of subtasks with increasing difficulty. For each task of the curriculum, its Q-function is modeled and learned the of residuals, with each residual corresponding to a previous task of the curriculum, which was shown to outperform fitting a single function approximator [50 28 Similar to RL fine-tuning, BCRL. focuses expediting RL learning on a target task with the Q-functions learned on the previous tasks. The residual is learned to construct the Q-function for the target task with known reward. Conversely, RQL aims at learning the residual to construct the Q-function for a new task whose reward function combines the unknown underlying reward of the expert and the known add-on reward.
5 Experiments
We evaluate the proposed algorithms on four environments selected from different domains: CartPole and Continuous Mountain Car environments from the OpenAl gym classic control suite [3] and Highway and Parking from the highway-env environments [32] In our experiments, we implemented our algorithms upon Stable- Basel-BBssl (43 and its imitation library [16]. In Sec. S.I. we provide the configurations of our experiments, including the settings of policy customization tasks in different environments, baselines, and evaluation metrics. In Sec. 5.2 we present and analyze the experimental
results of RL offline policy customization. In Sec. 5.3 we demonstrate an example of zero-shot online customization leveraging the residual maximum investigate two representative RL fine-tuning methods under the context of policy customization and compare them with the proposed algorithms. Please refer to the appendices for detailed experiment configurations, implementation details, ablation studies, and additional experiments in MuloCo [49]
mexximum-entrrpp entropy MCTS algorithm. In Sec. [.4] we
5.1 Experiment Setup
The detailed configurations of the environments can be found in Appendix D We mainly describe the policy customization task settings here. In each environment, we design the basic and add-on rewards to illustrate practical application scenario, where the basic reward corresponds to some basic task requirements and the add-on reward reflects customized specifications such as behavior preference or additional task objectives.
Environments.

---

CartPole balancing; In the cartpole environment, the basic task is to balance the pole as long as possible. During policy customization, we demand an additional task that requires the cart to stay at the center of the rack. To validate if the additional objective is satisfied, we monitor the average absolute position error of the cart which is defined as e, = Σ It:l/T, where 2, is the coordinate of the cart with respect to the center of the rack.
Continuous mountain car: In the mountain car environment, the basic task is to climb the mountain with the least energy consumption. During policy customization, we enforce an additional preference to avoid negative actions whenever possible. This corresponds to use cases where certain actions are less favorable. To monitor the change in action distribution, count the number of negative actions executed during each episode, denoted Naeg-
Highway navigation: In the highway environment, the basic task is to navigate the vehicle safely and efficiently through the traffic. The vehicle can switch to arbitrary lanes to overtake the other vehicles. During policy customization, we enforce additional preference to stay on the rightmost lane whenever possible. This reproduces driving habits that human drivers may have in real life. For example, drivers may prefer to drive on the rightmost lane to avoid missing the exits or when they want to drive the car slowly. We compute the average normalized lane index over an episode validate if the desired behavior emerges, which is defined as hame = Σi= Jinn..//TT, where /ann, E {0, 1, 2} is the lane index with the leftmost lane the lane and the rightmost lane as the lane 2.
Parking: In the parking environment, the basic task is to park the car in the target parking slot. During policy customization, we add an additional requirement to avoid touching the boundaries of the parking slots during parking, which is considered good practice to avoid scratching other cars and to leave sufficient space for neighboring parking slots. We compute the non-violation rate to monitor if the requirement is met, which is defined as the percentage of episodes where the constraint is not violated, denoted as Tno violation
Baselines. In our experiments, we mainly compare the performance of four categories of policies:
RL prior policy: We train an RL policy optimizing the basic reward. It serves as the prior policy for policy customization. For environments with discrete action space (i.e., CartPole and Hlghhwy),, we train the RL prior policy using soft Q-learning. For environments with continuous action space (i.e., Continuous Mountain Car and Parking), soft actor-critic is used. test time, we also evaluate its performance on the overall task without customization, which serves as baseline to show the effectiveness of the proposed algorithms.
IL prior policy: We train another IL prior policy that imitates the RL prior with GAIL 22 As discussed in Sec. the proposed RQL algorithms are primarily motivated to customize imitative policy without knowledge of its inherent reward. The IL prior policy serves as a baseline for comparison with the residual-Q policy customized from the II. prior policy. In GAIL,, we use soft Q-learning as the policy learning algorithm when the action space is discrete, and use soft actor-critic when the action space is continuous.
Residual-Q customized policies (Ours): In each environment, we train two residual-Q customized policies leveraging the RL and IL prior policies respectively, They are mainly compared against the corresponding prior policies to validate the effectiveness of policy customization. We apply residual soft Q-learning for environments with discrete action space and apply the residual soft actor-critic for environments with continuous action space. Besides, we evaluate the zero-shot customization ability of the proposed residual maximum-entropp MCTS algorithm in the Highway environment.
We also train an RL policy with the total reward (i.c., + WTR). Note that we do not intend to treat it as a baseline for fair comparison under the policy customization setting. We aim compare the residual-Q policies with it to validate that the customized policies indeed solve the overall task M.
RL with total reward:
Metric. We are mainly interested in validating the performance of the proposed policy customization methods from two perspectives: 1) whether the policy is customized toward the add-on reward; 2) whether the customized policy still possesses the characteristics of the prior policy, in other words, maintains the same level of performance on the basic task. Hence, we use the average basic reward and the success rate of completing the basic task as metrics to evaluate the policies' performance on
7

---

the basic task. Meanwhile, we use the average add-on reward and task-specific metrics introduced above as the metries for evaluating the policies' performance with respect to the customized objective.
5.2 RL Offline Customization
The experimental results of RL offline customization are summarized in Table 1 In all the envi- romments and with either RL or IL prior policies, the customized policies significantly outperform the prior policies on the add-on task objectives. Meanwhile, the customized policies maintain the same level of performance as the prior policies on the basic tasks. It validates that the proposed residual-Q algorithms can tailor the policies for customized requirements while maintaining their original characteristics. One interesting observation is that the policy customization improves the performance of the IL. policy on the basic task in the Parking environment. One factor is that GAIL does not perform well in the first place. The Parking environment is indeed more challenging than the other environments given the continuous action space and, more importantly, the nature of the task-navigation in tight space under the non-holonen con-holonomic constraint. Another aspect we think can explain this phenomenon is that the add-on reward implicitly guides the policy to improve its basic task performance. As shown in Appendix G it is difficult for the imitative policy to stop the vehicle accurately at the target parking slot. Since GAIL merely learns match the trajectory distributions under the expert and learned policies without the knowledge of the actual task objective, it is chal- lenging for the imitative policy to precisely localize such a narrow parking space. In contrast, the add-on boundary violation constraint confines the vehicle trajectory within the parking space once the vehicle approaches the target parking slot. Hence, the policy is able to reach the goal point faster and receives higher returns even on the basic parking task.
In Table M we also compare the customized policies against the RL policies trained with the total reward function, named RL Full Policy in the table. In environments with discrete action spaces, the residual soft-Q Q learning policies behave very similarly the RL full policies. Meanwhile, the residual soft actor-critic agents have different reward patterns compared to the RL full policies. We think the behavior difference is mainly due to the approximation errors in value and policy networks. The proposed residual-Q Q framework relies on the prior policy to acquire knowledge of the basic task. Thus, a suboptimal prior will affect the performance of the customized policy, especially on the basic task. It also explains the performance difference between the policies customized from RL and IL priors. Inspired by the observation mentioned above on IL policy customization, we think a practical solution to mitigate this issue is partially incorporating some basic task objectives into the add-on reward if available. is reasonable for many practical use cases even with an imitative prior. While is difficult to precisely describe the underlying reward function of human experts, can comfortably assume that they follow some obvious commonsense objectives (c.g., collision avoidance). In this way, our method can be considered more principled framework for the general synergy between IL and RL beyond policy customization where the reward functions are completely decomposed.
5.3 MCTS Online Customization
We choose the Highway environment to demonstrate the zero-shot online customization ability of residual maximum -nnuuppyy MCTS. We implement residual maximum maximum-entropy MCTS based on the MCTS planner provided by the highway-env environment. The planner leverages the ground-truth dynamics model of the environment for roll-out simulation. The results are summarized in Table [ MCTS online customization is able to achieve similar performance as the RL offline counterpart.
5.4 Ablation Study: Comparison with RL Fine-tuning
In RQL framework, we formulate policy customization as an MDP whose reward function balances the utilitics of the imitative and add-on tasks. It provides the theoretical foundation to determine the synergy between the IL and RL objectives in principled manner. As discussed in Sec. T and Sec. 4 different approaches have been proposed in the RL fine-tuning literature to combine IL. and RL objectives. We investigate
representative methods under the context of policy customization. We compare them against RQL and summarize the experimental results in this subsection. Please refer to Appendix FF for the complete ablation study.
Greedy Reward Decomposition. One common approach is regularizing the divergence between the trained RL and prior policies during policy updates. It has been widely used in offline RL algorithms
8

---

to address the distributional shift issue 30 54] and Advantage Weighted Actor-Critic (AWAC) 40 demonstrates the benefits of such policy regularization in stabilizing and accelerating RL fine-tuning. As shown in AWAC [40], when using KI. KI.-divergence to measure the distance between policies, the regularized optimal policy becomes a maximum ennrropp policy with the advantage weighted by the prior policy. If we directly adapt it to solve the policy customization problem, the customized policy is essentially defined as the solution to the following optimization problem at each policy update step:
iT2 arg max (.)+) (RR(S,,) â log [(aas) XDRL (7(-)s #Ell
(15)
where QR. is the soft Q-function of the MDP with only the add-on reward rs(8, a). which is defined iteratively with the following update rule:
QR,(+I(S, a) = =rR(s, a) + + -Bx==(('aaa) [Ba'na [QRa(S', ('))ss'a)= - à log [r(aa'ss]]
(16)
The solution to the optimization problem given as:
* (a)ss) x exp ( 1 à + A (Qn.(s,a) + Alog :
(17)
We can reduce our RQL policy to the same form by simply replacing the residual Q-function in Eqn. (6) with the soft Q-function QR,I. The resulting policy solves the optimization problem in Eqn. (15) but with an entropy weight of á - in the objective function. If we consider it for the policy customization problem, is essentially equivalent to heuristically estimating the optimal Q-function as the sum of the optimal Q-functions corresponding to the basic and add-on rewards. Hence, the synthesized policy is not the optimal solution to the target MDP but greedy approximation 52 26 We compare RQL with this greedily customized policy on the Continuous Mountain Car and Parking environments. The results can be found in Appendix F.T We found that this greedy customized policy performed much worse than RQL in the most challenging Parking environment. Compared to the prior policy, greedy customization compromises the policy's 's success rate the basic parking task to customize them towards satisfying the add-on objective. However, the greedy policy still has a significantly lower non-violation rate compared the RQL policy, It validates that our framework is a more principled method to jointly optimize the IL and RL objectives for policy customization than divergence-baset regularization.
Augmenting Reward with Policy KL-DDvergence. Alternatively, some works [4 61 have explored adding the policy divergence to the reward function. It has been shown to outperform policy regularization in offline RL setting [54]. To adopt this line of approaches for policy customization, we formulate the reward function of the target MDP as the sum of the add-on reward and policy divergence regularization term. In particular, we follow [6]] to augment the add-on reward with a penalty whose expectation regularizes the KL-divergence between the customized and prior policies. Formally, the reward function is defined as:
#.. (a|s) π (a|s)
i. (s, a) rn(s, a) log
(18)
where it denotes the customized policy after the th iteration of policy update, and |3 is hyperparam- eter to balance the add-on reward and the policy regularization term. We can then apply standard RL algorithms to customize the policy toward this reward function. However, we found that it made the learning task much more difficult in our policy customization problem-as shown in Appendix F.2 soft actor-critic, for example, failed to find customized policy that at least performed well on the basic parking task. We believe that one crucial factor contributing to the failure of divergence-basee regularization is that the basic and add-on rewards are mostly orthogonal in our setting. The add-on reward only encodes additional task requirements; thus, we heavily rely on the prior policy to embed the desirable behavior on the basic task. Merely regularizing the KL-divergence is therefore insuffi- cient boant-biigg close to the prior policy in terms of probabilistic distance does not necessarily imply behavior that accomplishes the basic task objective. Besides, incorporating the policy likelihood into the reward function makes it more challenging to update the Q-function in practice. For example, in Appendix
[F33 we show that our RQL algorithm is theoretically equivalent to directly solving an MDP whose reward function is weighted sum of the add-on reward and the log- likeliiod of the prior policy. However, similar to the case of iivergence-augmeen reward, soft actor-critic failed to find a solution in Parking environment.

---

Table 1: Experimental Results of Residual-Q Policy Customization
Basic Task
Add-on Task
Env.
Policy
Suce. Rate
Basic Reward
6z
Add-on Reward
RL Prior Policy
100%
427.21 + 2.25
0.15 + 0.11
-31.11 22.82 -1..18 + 10.12
RL Customized
100%
425.73 2.74
0.05 0.05
CartPole
IL Prior Policy 100%
422.17 + 4.19
0.19 + 0.12
-38.57 + 24.26
IL Customized
100%
423.37 = 4.28
0.07 0.06
144.12 12.09
RL Full Policy
100%
425.27 3.10
0.05 0.04
-10.72 7.43
Env.
Policy
Succ. Rate
Basic Reward
Aneg
Add-on Reward
RL Prior Policy
100%
95.78 = 0.64
42.26 + 1.29
-4.23 + 0.13
RL Customized
100%
95.61 0.43
37.90 0.75
-3.79 0.07
Cont.
Mt. Car
Prior Policy
100%
92.84 2.35
64.40 21.03
-6.44 2.10
IL Customized
100%
94.41 0.06
41.08 = 1.01
-4.11 0.10
RL Full Policy
100%
95.14 0.21
27.86 + 3.87
-2.79 + 0.39
Env.
Policy
Suce. Rate
Basic Reward
Iane
Add-on Reward
RL Prior Policy
97.22%
44.67 + 2.43
0.46 + 0.28
9.17 +: 5.54 5.54 19.42 = 0.99 19.19 = 1.38
RL Customized
97.19%
42.23 1.55
0.97
RL ME-MCTS
98.00%
42.71 = 1.65
0.96
Highway
II. Prior Policy
96.04%
40.03 = 0.02
0.50 + 0.40
10.06 + 8.07
IL Customized
93.00%
40.36 0.34
+
19.88 0.24
IL ME-MCTS
97.00%
40.68 0.46
0.99 0.01
19.85 - 0.30
RL Full Policy
96.35%
41.98 1.53
0.97 + 0.01
19.49 + 1.24
Env
Policy
Succ. Rate
Basic Reward
noo-violatin
Add-on Reward
RL Prior Policy
99.09%
-7.08 + 2.65
57.61%
-1.35 + 3.71 -0.03 + 0.20
RL Customized
98.73%
-7.60 = 3.07
96.09%
Parking
IL Prior Policy
70.91% %
-12.37 + 9.05
11.27%
-4.94 5.52
II. Customized
83.59%
-7.93 + 4.84
66.07%
-0.32 + 0.73
RI. Full Policy
99.34%
66.81 + 2.39
74.24%
0..8 + 0.72
The evaluation results computed over 4000 episodes for model-free policies and 100 episodes for maximum- entropy MCTS (MF-MCTSS). The statistics with + in the format of mean + std.
6 Limitations
As discussed in Sec. 5.2, one limitation of our current algorithms is that the performance of policy customization is bottlenecked by the imitative prior policy. Policy customization not only requires it perform well on the trajectory distribution under the imitative prior policy itself but, more importantly, we need the imitative policy to accurately indicate desired behavior with respect to the basic task objective careful design, our algorithms would suffer from the distributional shift issue. While, in practice, this issue can be mitigated by partially incorporating some basic task objectives into the add-on reward if available, a more principled and fundamental solution is to obtain a diverse imitative prior that is best suited for customization. Generative models, such as normalizing flows (88 and, more recently, diffusion models (23] 6 20 have been shown promising in constructing such diverse behavior prior. Besides, incorporating human experts the loop has also been shown be effective in improving the generalizability of imitation learning (27 38 41 33]. We will investigate their applications in the policy customization problem in future work. Also, we only demonstrate MCTS online customization with an ideal perfect dynamics model in our current experiments. In future work, we will combine residual maximum-entropy MCTS with offline learned predictive models achieve zero-shot online customization in more practical scenarios.
the trajectory distribution under the cuslomized policy. Hence, without
10

---

References
[1] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In Proceedings of Robotics: Science and Systems, FreiburgimBreis- gau, Germany, June 2019.
[2] R. Bhattacharyya, B. Wulfe, D. J. Phillips, A. Kuefler, J. Morton, R. Senanayake, and M. J. Kochenderfer. Modeling human driving behavior through generative adversarial imitation learning. IEEE Transactions on Intelligent Transportation Systems, 2022.
[3] R. P. Bhattacharyya, D. J. Phillips, C. Liu, J. K. Gupta, K. Trigg--Campbeld and M. J. Kochen- derfer. Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 789-795. IEEE, 2019.
[4] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 2016.
[5] G. Brockman, V. Cheung, I.. Pettersson, J. Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint 1606.
2016.
[6] C. Chi, S. Feng, Y. Du Z. Xu, E. Cousineau, Burchfiel, and Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04152 2023.
[7] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via con- ditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4693-4700. IEEE, 2018.
[8] R. Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Computers and Games: 5th International Conference, CG 2006, Turin, Italy, May 29-31, 2006. Revised Papers 5, pages 72-83. Springer, 2007.
[9] Englert, N. A. Vien, and M. Toussaint. Inverse kkt: Learning cost functions of manipulation tasks from demonstrations. The International Journal of Robotics Research, 36(13-14):1474- 1488, 2017.
[10] L. V. Espinoza. A. Liniger, W. Schwarting, D. Rus, and Van Gool. Deep interactive motion prediction and planning: Playing games with motion prediction models. In Learning for Dynamics and Control Conference, pages 1006-1019, PMLR, 2022.
[11] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp. C. Qi, Y. Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[12] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprini arXiv: 11248, 2017.
[13] S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 0220012-221455 2021.
[14] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[15] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems, volume 34, pages 4028-4039, 2021.
[16] A. Gleave, M. Taufeeque, J. Rocamonde, E. Jenner, S. H. Wang, S. Toyer, M. Ernestus, N. Belrose, S. Emmons, and S. Russell. imitation: Clean imitation learning implementations. arXiv preprini arXiv:2211.1 1972, 2022.
[17] Haarojja, H. Tang, Abbeel, and Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pages 1352-1361. PMLR, 2017.
[18] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.

---

[19] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 2018.
[20] P. Mlansen-Estruche I. Kostrikov. M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[21] T. Hester, M. Vecerik, O. Pictquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan. Quan, A. Sendonaris, I. Osband, et al. Deep q-lcarning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence volume 32, 2018.
[22] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, volume 29, 2016.
[23] M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 099911, 2022.
[24] S. Jauhri, J. Peters, and G. Chalvatzaki. Robot learning of mobile manipulation with reachability behavior priors. IEEE Robotics and Automation Letters, 7(3):8399-8406, 2022.
[25] K. Judah, Fern, P. Tadepalli, and R. Goetschalckx. Imitation learning with demonstrations and shaping rewards. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, 2014.
[26] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Deshi-Velez. Explainable reinforcement learn- ing via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence, 2019.
[27] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Roboties and Automation (ICRA), pages 8077-8083. IEEE, 2019.
[28] P. Klink, C. D' D'Eramo, J. Peters, and J. Pajarinen. Boosted curriculum reinforcement learning. In International Conference on Learning Representations, 2021.
[29] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.061699 2021.
[30] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[31] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learingg for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[32] E. Leurent. An environment for autonomous driving decision-makinn GitHub repository, 2018.
[33] C. Li, T. Trinh, L. Wang. C. Liu, M. Tomizuka, and W. Zhan. Efficient game-theoretic planning with prediction heuristic for socially-compliant autonomous driving. IEEE Robotics and Automation Letters, 1(4):1024--102)
2022.
[34] J. Li, C. Tang, M. Tomizuka, and W. Zhan. Dealing with the unknown: Pessimistic offline reinforcement learning. In Conference Robot Learning, pages 1455-1464. PMLR, 2022.
[35] Y. Li, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. Advances in Neural Information Processing Systems, 30, 2017.
[36] Y. Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, B. Roelofs, B. Sapp. B. White, A. Faust, S. Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. arXiv preprint 2xxv:2212.114199 2022.
[37] A. Mandlekar, D. Xu, R. Martin-Martin, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint 270iv22003.06085, 2020.
[38] A. Mandlekar, D. Xu, R. Martín-Martin, Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in-the-loop imitation learning using remote teleoperation. arXiv preprint arXivv20/2.06733, 2020.
[39] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martnn. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, pages 1678-1690. PMLR, 2022.
[40] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2020.
12

---

[41] Z. Peng, Q. Li, C. Liu, and B. Zhou. Safe driving via expert guided policy optimization. In Conference on Robot Learning, pages 1554-1563. PMLR, 2022.
[42] A. Raffin. RI baselines3 zoo. CitHub repository, 2020.
[43] A. Raffnn, A. Hill, A. Gleave, A. Kanervisto, M. Emestus, and N. Dormann. bable--aseelines: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[44] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: iet.i1709.000077 2017.
[45] N. Rhinehart, R. McAllister, and S. Levine. Deep imitative models for flexible inference, planning. and control. In International Conference on Learning Representations, 2020.
[46] O. Scheel, L. Bergamini, M. Wolczyk, B. Osiński, and P. Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In Conference on Robot Learning. pages 718-728. PMLR, 2022.
[47] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839)6004-69) 2020.
[48] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10024, 2020.
[49] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-basee control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033 2012.
[50] S. Tosatto, M. Pirotta, C. d' Eramo, and M. Restelli. Boosted fitted q-iteration. In International Conference on Machine Learning. pages 3434-3443. PMLR, 2017.
[51] I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, Simon, M. Bennice, C. K. Fu, C. Ma, J. Jiao, S. Levine, and K. Hausman. Jump-start reinforcement learning. In NeurIPS 2021 Robot Learning Workshop, RSS 2022 Scaling Robot Learning Workshop. 2022.
[52] H. Van Seijen, M. Fatemi, Romoff, R. Laroche, T. Barnes, and J. Tsang. Hybrid reward architecture for reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
[53] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. K. Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting, arXiv preprint arXhv:2301.00499 2023.
[54] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911. 11361, 2019.
[55] C. Xiao, R. Huang, J. Mei, D. Schuurmans, and M. Müller. Maximum entropy monte-carlo planning. Advances in Neural Information Processing Systems, 32, 2019.
[56] Z. Xue, Z. Peng, Q. Li, Z. Liu, and Zhou. Guarded policy optimization with imperfect online demonstrations. In The Eleventh International Conference on Learning Representations, 2022.
[57] W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25488, 2021.
[58] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse M. Naumann, J. Kummerle, H. Konigshof, C. Stiller, A. de La Fortelle, et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv:1910.084 03088 2019.
[59] Zhang, Z. McCarthy, O. Jow. D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Roboties and Automation (ICRA), pages 5628-5635. IEEE, 2018
[60] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. In Aaai, volume 8, pages 1433-1438. Chicago, II., USA, 2008
[61] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv: 19990099933
2019.
13


Reviews:
[{"review_id": "t3zCba0cJp", "reviewer": "Reviewer_6WQ5", "summary": "This paper introduces a novel concept policy customization, which trains a new policy based on a pre-trained policy so as to solve the downstream tasks. Compared to RL fine-tuning algorithms, policy customization directly optimize the imitative objectives and downstream RL objectives jointly. The authors propose a new approach, Residual Q-learning, for policy customization. The proposed algorithm has shown effectiveness across 4 different environments in this work, suggesting their potential for efficient policy customization.", "questions": "The optimization is based on the assumption that the optimal policy pre-trained with IL follows a Boltzmann distribution. The imitation policies in experiments are from stable-baselines' library. But for some more practical or complex envs, such as CARLA and MuJoCo, this assumption is hard to satisfy since the expert demonstrations have more and larger bias. I'm wondering the Eq (6) cannot be satisfied, how to optimize the Residual Q-Learning problems?", "limitations": "1. The statement about the definition and significance of policy customization is poor. \n2. The experiment environments are simple, lacking convincing.\n3. The assumption in Eq 1 is strong.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper tackles an important problem in RL, learning new tasks based on given policy priors.\n2. The formulation and theorem are sound.\n3. The approach is based on a simple concept \"residual optimization\", yet appears to generate some benefits, especially in harder tasks.", "weaknesses": "1. The author should clarify the significance of policy customization and make more arguments about the difference between policy customization and RL fine-tuning methods. The main difference between them seems that policy customization jointly trains the reward from the inherent rewards and add-on rewards, while RL fine-tuning methods learn from inherent rewards more intrinsically. Why is policy customization deserved to study, or why jointly combining the reward significant? Furthermore, I haven't noticed a detailed argument about the necessity of the proposed novel concept.\n2. The experimental environments are simple. More complex tasks are better. Moreover, the authors should compare with some RL fine-tuning methods mentioned in related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and valuable suggestions. We are glad you find the policy customization problem novel and our theoretical formulation sound. Please find below our responses to address your concerns regarding our work: \n\n> The author should clarify the significance of policy customization and make more arguments about the difference between policy customization and RL fine-tuning methods. The main difference between them seems that policy customization jointly trains the reward from the inherent rewards and add-on rewards, while RL fine-tuning methods learn from inherent rewards more intrinsically. Why is policy customization deserved to study, or why jointly combining the reward significant? Furthermore, I haven't noticed a detailed argument about the necessity of the proposed novel concept.\n> \nThank you for your suggestions on further elaborating on the significance of policy customization and its difference from RL fine-tuning. Please find our clarification on these questions and proposed revisions in the author rebuttal thread.\n\n> The experimental environments are simple. More complex tasks are better. \n\nWe understand that the current experiments do not fully reflect the wide scope of applications our proposed theoretical framework could have. To address this issue, we have conducted additional experiments in MuJoCo environments. Please refer to the author rebuttal thread for a summary of the additional experimental results. We plan to add these additional experiments in the revised version of the paper.\n\n> Moreover, the authors should compare with some RL fine-tuning methods mentioned in related works.\n\nThank you for proposing the RL fine-tuning baselines. They are indeed very important baselines for comparison. In the submitted manuscript, we have actually investigated two representative RL fine-tuning methods under the context of policy customization and discussed our findings in Section 6 (lines 329-361, pages 8-9) and Appendix D.1-2. Please refer to the \"Comparison with RL Fine-tuning\" paragraph in the author rebuttal thread for a brief summary. \n \n> The optimization is based on the assumption that the optimal policy pre-trained with IL follows a Boltzmann distribution. The imitation policies in experiments are from stable-baselines' library. But for some more practical or complex envs, such as CARLA and MuJoCo, this assumption is hard to satisfy since the expert demonstrations have more and larger bias. I'm wondering the Eq (6) cannot be satisfied, how to optimize the Residual Q-Learning problems?\n\nWe understand the Boltzmann assumption is an important underlying assumption of our framework that need further justification. We consider the Boltzmann (i.e. maximum-entropy) assumption a reasonable one in imitation learning, since it has been commonly adopted to account for sub-optimal demonstration [1], which is particularly useful when the demonstration is collected from human experts. In the revised manuscript, we will add this reference to further justify the maximum-entropy assumption of demonstration when introducing Eqn. (1). \n\nMeanwhile, the demonstration's quality indeed affects residual Q-learning's performance. If the expert demonstration is limited and biased, then the IL prior cannot accurately represent the expert behavior. Since we rely on the IL prior to encode the inherent characteristics of the expert, the customized policy is not the optimal solution to the MDP with joint rewards due to the biased prior. However, this issue is not specific to policy customization and residual Q-learning but a fundamental question in learning from demonstration. Therefore, we believe it does not hinder the value of the newly proposed policy customization setting and the residual Q-learning framework. In the submitted manuscript, we discussed two potential solutions to mitigate this issue: 1) as discussed in Section 5.2 (lines 294-321, page 8), we can partially incorporate some basic task objectives into the add-on reward if available. It is reasonable for many practical use cases, even with an imitative prior. While it is difficult to describe the underlying reward function of human experts precisely, we can comfortably assume that they follow some obvious commonsense objectives (e.g., collision avoidance); 2) as mentioned in Section 6 (lines 369-372, page 9), we can adapt the recent advances in policy pretraining (e.g., [2]) to obtain a diverse imitative prior that is best suited for customization. \n\n[1] B. D. Ziebart, et al. \"Maximum Entropy Inverse Reinforcement Learning.\" AAAI 2008.\\\n[2] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. \"Parrot: Data-driven behavioral priors for reinforcement learning.\" ICLR 2021. \n\nWe hope our clarifications and additional experimental results can address your concerns and raise your impression of our work. Please let us know if you have any further questions."}, {"review_id": "tv73GUoZci", "reviewer": "Reviewer_2HY5", "summary": "This paper proposes a new problem setting called policy customization, where the goal is to train a new policy that inherits the characteristics of a prior policy while satisfying additional requirements imposed by a downstream task. The authors propose a novel Residual Q-Learning framework to solve this problem, which leverages the prior policy without knowing its inherent reward or value function. The authors derive a family of residual Q-learning algorithms for offline and online policy customization and show their effectiveness in various environments. The empirical results on four tested environments seem promising.\n", "questions": "- What is the advantage of Residual Q-Learning when compared with inverse-RL? One can learn the inherent reward function from interactions and combine it with the additional task-specific reward.\n- What does it mean by “principled way”? Why is Residual Q-Learning better than related works?\n- How is the prior policy $\\pi$ used? Are you assuming you have an “online” policy to imitate? Then you may also need to compare with those papers focusing on RL with shared control [1,2,6].\n", "limitations": "References\n\n[1] Peng, Zhenghao, et al. \"Safe driving via expert guided policy optimization.\" Conference on Robot Learning. PMLR, 2022.\n\n[2] Xue, Zhenghai, et al. \"Guarded Policy Optimization with Imperfect Online Demonstrations.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[3] Li, Quanyi, et al. \"Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning.\" IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3461-3475.\n\n[4] Kelly, Michael, et al. \"Hg-dagger: Interactive imitation learning with human experts.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n\n[5] Mandlekar, Ajay, et al. \"Human-in-the-loop imitation learning using remote teleoperation.\" arXiv preprint arXiv:2012.06733 (2020).\n\n[6] Li, Quanyi, Zhenghao Peng, and Bolei Zhou. \"Efficient learning of safe driving policy via human-ai copilot optimization.\" arXiv preprint arXiv:2202.10341 (2022).", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a new problem setting called policy customization, which addresses the need to customize a pre-trained policy to meet additional requirements imposed by a downstream task.\n- The Residual Q-Learning framework proposed in this paper provides a principled approach to interpret and design the policy customization objective, enabling the joint optimization of imitative and downstream task performance.\n", "weaknesses": "- The evaluation for the proposed method is limited. It would be beneficial to see more extensive comparisons and discussions with more baselines in the field, such as EGPO [1] and TS2C [2]. Also it would be beneficial to include some larger-scale RL environments, such as MuJoCo or MetaDrive [3].\n- A more comprehensive discussion with related works should be presented. There is a whole line of research that assumes the availability of an online expert policy to imitate [1,2,4,5,6]. With these methods, it will be natural to perform policy customization by adding task-specific rewards to the learning process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and valuable suggestions. We are glad you find the policy customization problem and residual Q-learning framework novel. Please find below our responses to address your concerns regarding our work: \n\n> The evaluation for the proposed method is limited. It would be beneficial to include some larger-scale RL environments, such as MuJoCo or MetaDrive.\n\nThank you for suggesting additional experiments in MuJoCo environments. We understand that the current experiments do not fully reflect the wide scope of applications our proposed theoretical framework could have. To address this issue, we followed your suggestions and conducted additional experiments in MuJoCo environments. Please refer to the author rebuttal thread for a summary of the additional experimental results. We plan to add these additional experiments in the revised version of the paper.\n\n> What is the advantage of Residual Q-Learning when compared with inverse-RL?\n\nThank you for raising inverse RL for comparison. It is indeed a very important point. Please refer to the author rebuttal thread for our clarification on this question. We plan to add the discussion of inverse RL in Section 2.2 of the revised version, in order to motivate our choice of customizing the policy without inferring its underlying reward. \n\n> What does it mean by “principled way”? Why is Residual Q-Learning better than related works?\n\nPolicy customization aims to train a policy that inherits the characteristics of an imitative prior while satisfying additional downstream task requirements. Residual-Q is a principled solution to policy customization because it provides a theoretically sound way to interpret and determine the trade-off between the imitative objective and additional task requirements, especially compared to the widely studied RL fine-tuning methods. Many RL fine-tuning methods also combine IL and RL objectives. However, the IL objective is typically chosen as either divergence in policy distribution or distance in action space. While it is not necessarily an issue in RL fine-tuning problem -- the IL objective is added as a regularization to accelerate RL training -- it becomes an issue when we consider these methods in the context of policy customization. It is challenging to determine a good trade-off between two very different objectives which evaluate the policy in distinct dimensions. In contrast, in residual-Q, the trade-off between these two objectives is equivalent to the trade-off between the basic and add-on rewards; Thus, by optimizing the residual-Q objective, we are directly optimizing the policy's performance on the basic and add-on tasks given a desired weight between these two tasks. It is supported by our experiments with RL fine-tuning methods for policy customization. For a brief summary, please refer to the \"Comparison with RL Fine-tuning\" paragraph in the author rebuttal thread.\n\n> A more comprehensive discussion of related works should be presented. There is a whole line of research that assumes the availability of an online expert policy to imitate. With these methods, it will be natural to perform policy customization by adding task-specific rewards to the learning process.\n\nThank you for bringing these related and interesting papers to our attention. First, we would like to clarify that residual-Q does not assume access to an online expert (e.g., human or trained policy) for the overall task (i.e., MDP with the weighted sum of the basic and add-on rewards as its reward). We assume access to a prior policy that accomplishes the basic task without the add-on reward. The goal is to customize the prior policy to complete both the basic and add-on tasks. In this sense, our work is different from the referred literature on learning from human intervention [1, 3, 4, 5] whose objective is to train a good imitative policy from human feedback without environment reward. However, we think those works could be potentially useful for training a good prior policy with better generalizability, when a human expert is available during online training. Such a generalizable prior could tackle the distributional shift issue we discussed in the paper. In the revised manuscript, we will include [1, 3, 4, 5] when discussing potential solutions to the distributional shift issue in the \"Limitations\" subsection (Line 367-372). \n\nRegarding [2], it is similar to ours from the perspective of having an imperfect expert -- the prior policy in residual-Q can be considered as an imperfect expert for the overall task -- and fusing knowledge extracted from this imperfect expert and RL exploration. The difference is [2] aims to train a policy towards a known reward; Thus, a value function can be trained to assert if the expert is reliable. In our case, since we rely on the prior policy to encode the basic task reward implicitly, we cannot use such a value function to govern the fusion between imitation and RL. In the revised manuscript, we will discuss [2] in Section 4, since it provides a different perspective on fusing imitation and RL.\n\nWe hope our response can address your concerns and raise your impression of our work. Please let us know if you have any further questions.\n\n[1] Zhenghao Peng, Quanyi Li, Chunxiao Liu, and Bolei Zhou. \"Safe driving via expert guided policy optimization.\" CoRL 2022.\\\n[2] Zhenghai Xue, Zhenghao Peng, Quanyi Li, Zhihan Liu, and Bolei Zhou. \"Guarded Policy Optimization with Imperfect Online Demonstrations.\" ICLR 2023.\\\n[3] Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J. Kochenderfer. \"Hg-dagger: Interactive imitation learning with human experts.\" ICRA 2019.\\\n[4] Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, Silvio Savarese. \"Human-in-the-loop imitation learning using remote teleoperation.\" 2020.\\\n[5] Quanyi Li, Zhenghao Peng, and Bolei Zhou. \"Efficient learning of safe driving policy via human-ai copilot optimization.\" 2022."}, {"review_id": "g3SUOZKtdQ", "reviewer": "Reviewer_QjWD", "summary": "<< I have read the authors' rebuttal and have raised my score based on the discussion >>\n\nThis paper introduces a novel problem setting, named 'policy customization,' which seeks to train a new policy that inherits the properties of an existing pre-trained policy and meets additional requirements from a given downstream task. The policy customization is formulated as a Markov Decision Process (MDP), with a reward function that combines the inherent reward of the demonstration and an additional reward defined by the downstream task. To address the challenge of solving the MDP without knowledge of the underlying reward of the imitative policy, the paper proposes a novel Residual Q-Learning framework. This includes defining a residual Q-function that, when combined with the log-likelihood of the prior policy, can construct the maximum-entropy policy for the target MDP. The paper presents two model-free policy customization algorithms, namely, 'residual soft Q-learning' and 'residual soft actor-critic' and empirically demonstrates that the proposed residual Q-learning algorithms effectively customize the policies toward the additional task objective, while preserving their performance on the original tasks for which the prior policies were trained.", "questions": "Q1. This work proves beneficial when the downstream task's objective is orthogonal to the prior policy's objective. However, what happens when the objectives conflict? For instance, suppose a prior policy for robot navigation is conservatively trained to minimize speed and avoid collision. If the downstream task necessitates rapid completion, wouldn't the prior policy potentially impede the optimization of the downstream task's objective?\n\nQ2. If possible, could you share the results of experiments where the Residual Q-learning framework is compared with RL fine-tuning combined with the IL objective of the prior policy used as heuristics? Such comparisons would provide a more comprehensive evaluation of the proposed approach.\n\nQ3. Based on the results, \"RL with total reward\" consistently outperforms the Residual Q-learning framework. Could you elucidate potential real-world scenarios where the Residual Q-learning framework would be more advantageous or preferred over the \"RL with total reward\" approach?", "limitations": "The authors discussed the limitations of their work. No discussion needed regarding potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. Clarity of Presentation: The paper is commendably articulated, offering clear motivations for the research and a thorough explanation of the proposed solution. The technical details are presented in a digestible manner, which significantly enhances the reading experience.\n\nS2. Technical Soundness: The paper exhibits robust technical soundness. The equations and underlying logic are laid out explicitly, providing a comprehensive understanding of the objectives. The problem formulation adheres to rigorous technical standards.\n\nS3. Novelty: The paper contributes original research to a specific problem domain - policy customization - and I have not seen papers targeting this problem. As such, the solution proposed is novel.\n\nOverall, the paper provides a compelling read and I found the research intriguing. However, some concerns about the experimental setup were noted (refer to the 'Weaknesses' section for details).", "weaknesses": "Despite the innovative approach presented in this paper, certain aspects of the experimental setup present potential limitations:\n\nW1. The selection of relatively simple problems for testing constrains the perceived impact of the work. With more complex environments like Atari and Procgen games now serving as standard test suites, the study's relevance might be diminished due to the primitive nature of the chosen experimental domains.\n\nW2. The paper lacks a comparison with a directly relevant baseline — RL fine-tuning combined with the IL objective of the prior policy used as heuristics. Given that RL or IL prior policies are trained on objectives distinct from the downstream task, their performance on the latter is likely to be sub-optimal. A head-to-head comparison with this method, which is a natural consideration for the policy customization problem, would better establish the strengths and limitations of the proposed Residual Q-learning framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and suggestions. We are very excited that you find our paper commendably articulated, novel, and clearly motivated. We are also glad to have your recognition of the technical soundness of the problem formation and derivation. Please find below our responses to address your questions regarding our work:\n\n>The selection of relatively simple problems for testing constrains the perceived impact of the work.\n\nWe understand that the current experiments do not fully reflect the wide scope of applications our proposed theoretical framework could have. To address this issue, we have conducted additional experiments in MuJoCo environments. Please refer to the author rebuttal thread for a summary of the additional experimental results. We plan to add these additional experiments in the revised version.\n\n>The paper lacks a comparison with a directly relevant baseline — RL fine-tuning combined with the IL objective of the prior policy used as heuristics. \n\nThank you for proposing the RL fine-tuning baselines. They are indeed very important baselines for comparison. In the submitted manuscript, we have actually investigated two representative RL fine-tuning methods under the context of policy customization and discussed our findings in Section 6 (lines 329-361, pages 8-9) and Appendix D.1-2. Please refer to the \"Comparison with RL Fine-tuning\" paragraph in the author rebuttal thread for a brief summary. \n\n>This work proves beneficial when the downstream task's objective is orthogonal to the prior policy's objective. However, what happens when the objectives conflict? For instance, suppose a prior policy for robot navigation is conservatively trained to minimize speed and avoid collision. If the downstream task necessitates rapid completion, wouldn't the prior policy potentially impede the optimization of the downstream task's objective?\n\nThank you for your insightful question. When the objectives of the prior policy and the downstream task conflict, our residual Q-learning framework still provides a principled way to interpret and determine the trade-off between these conflicting objectives. We would not consider the prior policy to impede the performance of residual Q-learning in this circumstance. Our goal is not to find the optimal policy that merely maximizes the add-on reward but achieves the desired trade-off between the basic (imitative) and downstream task objectives. \n\n>Could you elucidate potential real-world scenarios where the Residual Q-learning framework would be more advantageous or preferred over the \"RL with total reward\" approach?\n\nAs noted in Section 5.1 (line 278, page 7), we do not intend to treat \"RL with total reward\" as a baseline for fair comparison under the policy customization setting, since it requires ground-truth knowledge of the basic reward. In policy customization, we are mainly interested in application domains where handcrafting the basic reward function is challenging, but large-scale demonstrations are available to synthesize imitative prior policies. We included the \"RL with total reward\" baseline for comparison with residual-Q to validate that the customized policies indeed solve the overall task $\\hat{\\mathcal{M}}$.\n\nWe hope our clarifications and additional experimental results can address your concerns and raise your impression of our work. Please let us know if you have any further questions."}, {"review_id": "eAMKAXAdgW", "reviewer": "Reviewer_ePZC", "summary": "This paper proposes a new problem setting called pulicy customization, where the goal is to train a policy that inherits the characteristics of a prior policy while satisfying some additional requirements imposed by a downstream task. To solve this problem, this paper uses a novel and principled framework by formulating it as a MDP with a reward function that combines the inherent reward of the demonstration and the add-on reward of the downstream task. The framework is consistent with many prior RL algorithms, and leads to residual Q-learning, residual SAC, and residual ME MCTS. A comprehensive evaluation of the proposed residual Q-learning framework is conducted on various types of environments, and algorithms with residual Q outperform the prior RL or IL policies in all of them.", "questions": "There is no direct questions but I am curious about the following points.\n- Does the performance of residual-Q highly rely on the performance of the prior policy? For example, if the prior policy is learned from a mediocre dataset in an offline manner, will the residual Q-learning solve the following downstream task?\n- What is the relationship between the policy customization problem and the continuous learning problem? Is the policy customization a special case of continuous RL or transfer RL?", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- It introduces a novel and general problem setting of policy customization that can be applied to various domains and tasks.\n- It proposes a principled and flexible framework of residual Q-learning that can customize a policy without knowing the reward or value function of the prior policy, which can be easily added to existing algorithms.\n- Following the previous point, it provides a family of algorithms that can learn the residual Q-function and customize the policy.\n- For experiments, it demonstrates the effectiveness and robustness of the residual Q-learning framework on both discrete and continuous tasks, showing that the customized policies can inherit the characteristics of the prior policy while satisfying the downstream task requirements.", "weaknesses": "While this paper has some vague weaknesses and limitations, they are fully discussed in Section 6 (Discussion and Limitations). So I would like to leave blank here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We are very excited that you find policy customization a novel problem setting and find the proposed residual Q-learning framework a principled and flexible solution to the policy customization problem. \n\n> Does the performance of residual-Q highly rely on the performance of the prior policy? For example, if the prior policy is learned from a mediocre dataset in an offline manner, will the residual Q-learning solve the following downstream task?\n\nThis is a crucial point in the application of our approach, thank you for raising this question. Yes, the performance of residual Q-learning indeed relies on the performance of the prior policy. It is decided by the nature of the policy customization problem --- we rely on the prior policy to encode the inherent characteristics of the demonstration. In the submitted manuscript, we discussed this issue in Section 5.2 (lines 294-321, page 8). When we investigated residual Q-learning in the Parking environment, we actually started with an imperfect IL prior policy, since the environment is quite challenging given the continuous action space and, more importantly, the nature of the task—navigation in tight space under the non-holonomic constraint. We observed that policy customization improves the performance of the IL policy on the basic task. One aspect that we think can explain this phenomenon is that the add-on reward *implicitly* guides the policy to improve its basic task performance. Inspired by this observation, we think a practical solution to mitigate the issue of a mediocore prior is partially incorporating some basic task objectives into the add-on reward if available. It is reasonable for many practical use cases even with an imitative prior. While it is difficult to precisely describe the underlying reward function of human experts, we can comfortably assume that they follow some obvious commonsense objectives (e.g., collision avoidance). In this way, our method can be considered a more principled framework for the general synergy between IL and RL beyond policy customization where the reward functions are completely decomposed.\n\n> What is the relationship between the policy customization problem and the continuous learning problem? Is the policy customization a special case of continuous RL or transfer RL? \n\nThank you for raising the connection between policy customization and continual learning. It is an excellent point and worthy of in-depth study in future work. The policy customization problem can be considered a special case of continual learning [1] with two stages. The similarities lie in 1) the agent learns to incrementally conquer the basic task and then the task with additional objectives; 2) the customized policy should retain its ability to fulfilling the basic task objective, i.e., avoiding catastrophic forgetting. In this sense, policy customization differs from transfer RL, since transfer RL primarily concerns the policy's performance on the transferred target task. In the revised manuscript, we plan to mention this connection at the end of Section 6 and list the in-depth study of policy customization under continual learning as a future work. In particular, an interesting direction is to study sequential policy customization (i.e., customizing the policy with multiple add-on rewards added sequentially), which could benefit from drawing insights from continual learning literature. \n\n[1] K. Khetarpal, et al. \"Towards Continual Reinforcement Learning.\" *Journal of Artificial Intelligence Research (JAIR)*, 2022. "}, {"review_id": "nKAJ36Irui", "reviewer": "Reviewer_sftZ", "summary": "This paper proposes a problem setting called policy customization, which involves training a policy that inherits the characteristics of a prior policy while satisfying additional requirements imposed by a target downstream task. They present a novel approach to interpreting and determining the two task objectives' trade-offs. Specifically, they formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines the inherent reward of the demonstration and the additional reward specified by the downstream task. The authors introduce a new framework called Residual Q-learning that leverages the prior policy to solve the formulated MDP without knowledge of the prior policy's inherent reward or value function. They derive a family of residual Q-learning algorithms capable of offline and online policy customization, demonstrating their effectiveness in accomplishing policy customization tasks in various environments.", "questions": "Please refer to the Weakness.", "limitations": "Please refer to the Weakness.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel problem setting, policy customization, which addresses an important issue in imitation learning. This innovative approach better meets the personalized requirements of imitation policies in practical applications.\n\n2. The paper provides experimental validation by conducting experiments in various environments, demonstrating the effectiveness of the proposed algorithms in achieving policy customization for real-world tasks. These experiments verify the practicality and adaptability of the methodology, providing strong support for further research and applications.\n\n3. The proposed methodology in the paper is shown to be feasible and scalable. It does not rely on specific assumptions or constraints, making it applicable to different scenarios and tasks. The scalability of the approach allows for its extension and adaptation to more complex and diverse environments.", "weaknesses": "1. The derivation process in Equation 5 lacks clarity, particularly the transition from eq. 5b to eq. 5c and eq. 5d. It is necessary to provide a more detailed and step-by-step derivation to enhance the understanding of the readers. A comprehensive explanation of the intermediate steps and the mathematical transformations involved would greatly improve the transparency and rigor of the derivations.\n\n2. While the proposed setting in the paper is intriguing, there is a need for further clarification on the rationale behind its adoption. It is unclear why the chosen measure is employed, and it would be beneficial to provide a clear justification for this decision. Additionally, if a prior policy is available, elaborating on how policy expansion can be utilized to address the problem posed in the paper would help readers grasp the underlying reasoning.\n\n3. The experimental evaluation in this paper is limited in its scope. To substantiate the claims made, it is recommended to conduct more comprehensive experiments and provide extensive results. Expanding the experimental analysis with diverse scenarios, benchmark datasets, and evaluation metrics would contribute to a more thorough evaluation and provide stronger evidence to support the proposed approach.\n\n4. The derived forms of Residual SQL and SAC in Eq. 7-10 appear to be similar to the original formulations. It is necessary to provide a clear explanation of how to interpret and understand the derived results. Highlighting the differences, if any, and elucidating the implications of the derived forms compared to the original ones would enhance the clarity and significance of the proposed modifications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The derivation process in Equation 5 lacks clarity, particularly the transition from eq. 5b to eq. 5c and eq. 5d. It is necessary to provide a more detailed and step-by-step derivation to enhance the understanding of the readers. \n\nThank you for your suggestion on including a step-by-step derivation of Eqn. 5. In the revised manuscript, we plan to incorporate the following step-by-step breakdown of the derivation process in the appendix: \n\nFirst, we can write $r(s, a)$ as a function of the soft Q-function $Q^*(s, a)$ according to Eqn. (2):\n\n\\begin{equation*}\n    r(s,a) = \\gamma \\mathbb{E}\\_{s'\\sim p(\\cdot|s,a)}\\left[\\alpha \\log \\int_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\alpha}Q^* (s',a')\\right)da'\\right]-Q^*(s,a).\n\\end{equation*}\n\nEqn. (5b) results from substituting $r(s, a)$ with the right-hand side of the above equation. To derive Eqn. (5c), we first write $\\int_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\alpha}Q^* (s',a')\\right)da$ as the normalization factor $Z_{s'}$ based on its definition. Then note that, according to the definition of the residual Q-function $Q_{R,t}$, we can substitute $\\hat{Q}\\_t(s', a')$ in Eqn. (5b) with:\n\\begin{equation*}\n\\begin{aligned}\n\\hat{Q}\\_t(s',a') &= Q\\_{R,t}(s', a')+\\omega Q^*(s', a') \\\\\\\\\n& = Q\\_{R,t}(s', a') + \\omega \\alpha \\log \\pi\\left(a' \\vert s'\\right) + \\omega\\alpha \\log Z\\_{s'},\n\\end{aligned}\n\\end{equation*}\n\nwhere the second step comes from Eqn. (3). To obtain Eqn. (5d), note that $Z\\_{s'}$ is not a function of $a'$, so we can write the last term of Eqn. (5c) as:\n\\begin{equation*}\n\\begin{aligned}\n&\\mathbb{E}\\_{s'}\\left[\\hat{\\alpha}\\log \\int\\_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\hat{\\alpha}}\\left({Q}\\_{R,t} (s',a')+ \\omega\\alpha \\log \\pi\\left(a' \\vert s'\\right) + \\omega\\alpha \\log Z\\_{s'}\\right)\\right)da'\\right]\\\\\\\\\n= & \\mathbb{E}\\_{s'}\\left[\\hat{\\alpha}\\log Z\\_{s'}^{\\omega\\alpha/\\hat{\\alpha}}\\int\\_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\hat{\\alpha}}\\left({Q}\\_{R,t} (s',a')+\\omega\\alpha\\log \\pi\\left(a' \\vert s'\\right)\\right)\\right)da'\\right]\\\\\\\\\n= & \\mathbb{E}\\_{s'}\\left[\\hat{\\alpha}\\log \\int\\_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\hat{\\alpha}}\\left({Q}\\_{R,t} (s',a')+\\omega\\alpha\\log \\pi\\left(a' \\vert s'\\right)\\right)\\right)da'\\right] + \\omega \\alpha \\gamma \\mathbb{E}\\_{s'}\\log Z_{s'}.\n\\end{aligned}\n\\end{equation*}\n\n> While the proposed setting in the paper is intriguing, there is a need for further clarification on the rationale behind its adoption. It is unclear why the chosen measure is employed, and it would be beneficial to provide a clear justification for this decision. Additionally, if a prior policy is available, elaborating on how policy expansion can be utilized to address the problem posed in the paper would help readers grasp the underlying reasoning.\n\nCould you clarify which parts of the paper you find unclear and need further clarification? In the author rebuttal thread, we further elaborated on the motivation of policy customization and its difference from RL fine-tuning and inverse RL. We hope it clarifies the rationale behind our proposed problem setting and solution to the readers.\n\n> The experimental evaluation in this paper is limited in its scope.\n\nWe understand that the current experiments do not fully reflect the wide scope of applications our proposed theoretical framework could have. To address this issue, we have conducted additional experiments in MuJoCo environments. Please refer to the author rebuttal thread for a summary of the results. We plan to add these additional results in the revised version of the paper.\n\n> The derived forms of Residual SQL and SAC in Eq. 7-10 appear to be similar to the original formulations. It is necessary to provide a clear explanation of how to interpret and understand the derived results.  \n\nThank you for your suggestions. To make the derivations of the residual SQL and residual SAC clear to the readers, we plan to include a more detailed explanation in the appendix of the revised manuscript. Due to the space limit, we cannot post the complete derivation process here. We provide a sketch of the residual SQL's derivation for your reference. The residual SAC algorithm can be derived in a similar manner.\n\nThe original objective of soft Q-learning is to minimize the TD error of the soft Q-function:\n$J\\_{Q}(\\theta) = \\mathbb{E}\\_{(s\\_t,a_t)}\\left[\\frac{1}{2}\\left(\\hat{Q}^{\\mathrm{target}}\\_{\\bar{\\theta}}(s\\_t,a_t) - \\hat{Q}\\_{\\theta}(s\\_t,a\\_t)\\right)^2\\right]$,\nwhere the target Q-value is defined as $\\hat{Q}\\_{\\bar{\\theta}}^{\\mathrm{target}}(s\\_t,a\\_t) = r\\_{R}(s\\_t,a\\_t) + \\omega r(s\\_t,a\\_t) + \\gamma \\mathbb{E}\\_{s'\\sim p(\\cdot|s\\_t,a\\_t)}\\left[\\hat{\\alpha}\\log \\int_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\hat{\\alpha}}\\hat{Q}_{\\theta}(s',a')\\right)da'\\right]$.\n\nGiven the defined residual Q-function, we can write the soft Q-function as \n$\\hat{Q}\\_{\\theta}(s,a)=Q\\_{R,\\theta}(s, a) + \\omega Q^*(s, a)$. Following steps similar to Eqn. (5), we can express the target Q-value with the residual Q-function as \n$\\hat{Q}\\_{\\bar{\\theta}}^{\\mathrm{target}}(s\\_t,a\\_t) = r\\_{R}(s\\_t,a\\_t) + \\omega Q^*(s\\_t,a\\_t) + \\gamma \\mathbb{E}\\_{s'\\sim p(\\cdot|s\\_t,a\\_t)}\\left[\\hat{\\alpha}\\log \\int_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\hat{\\alpha}}\\left(Q\\_{R,\\theta}(s',a') + \\omega'\\log\\pi(a'\\vert s')\\right)\\right)da'\\right].$\n\nThus, if we define the target residual Q-value as $\\hat{Q}\\_{R,\\bar{\\theta}}^{\\mathrm{target}}(s\\_t,a\\_t)=r\\_{R}(s\\_t,a\\_t) + \\gamma \\mathbb{E}\\_{s'\\sim p(\\cdot|s\\_t,a\\_t)}\\left[\\hat{\\alpha}\\log \\int_{\\mathcal{A}}\\exp\\left(\\frac{1}{\\hat{\\alpha}}\\left(Q\\_{R,\\theta}(s', a') + \\omega'\\log\\pi(a'\\vert s')\\right)\\right)da'\\right]$, it is straightforward to see that the TD error of the soft Q-function equals to the TD error of the residual Q-function. It leads to the Eqn. (7-8) in the paper when we use the data in the replay buffer to estimate the target residual Q-value and the TD error.\n\nWe hope our response can address your concerns and raise your impression of our work. Please let us know if you have any further questions. "}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback and suggestions. We are delighted that the reviewers found policy customization novel (R1, R2, R3, R4, and R5) and essential (R1, R5). We are glad that they found residual-Q novel (R1, R2, R3, R4, and R5), principled (R2, R4), and flexible (R4). We also appreciate the reviewers for acknowledging the technical soundness of our derivations (R1, R3), and clarity of the technical details (R3). In this thread, we summarize our responses to the common concerns shared by the reviews. Please refer to the rebuttal attached to each reviewer's comments for a reviewer-specific response. 
 
### Motivation 
Several reviewers raised questions on the motivation for policy customization. Since it is a novel problem setting, we want to clarify these questions to help the reviewers assess the value of the proposed problem setting and solution. Policy customization aims to train a policy that inherits the characteristics of an imitative prior while satisfying downstream task requirements. It is useful for applications where: 1) handcrafting reward function is difficult, but demos are available to synthesize imitative policies; and 2) the downstream tasks impose diverse requirements beyond imitation. For example, autonomous vehicles should behave like human drivers to coordinate with other human-driven cars, which can be achieved through IL. Meanwhile, driving tasks impose additional objectives, such as reaching particular goals, enforcing safety, and complying with users' preferences. Residual-Q provides 1) a theoretical ground to interpret and determine the trade-off between the imitative and add-on objectives; and 2) flexible policy customization algorithms in offline and online manners.

**Comparison with RL Fine-tuning.** Multiple reviewers asked for clarification on the difference between residual-Q and RL fine-tuning. For clarification, we discussed this point in Sec. 4, Sec. 6, and Appx. D.1-2 of the submitted manuscript. Please find a summary of our arguments below. In the revised version, we will add the subtitle "Comparison with RL Fine-tuning" at the beginning of Sec. 6 to make this part more accessible to the readers. 

In Sec. 4 (lines 194-201), we discussed the RL fine-tuning literature. In RL fine-tuning, the IL objective is added as a heuristic to accelerate RL training. Conversely, we aim to jointly optimize the IL and RL objectives in policy customization. In Sec. 6 (lines 329-361) and Appx. D.1-2, we investigated two representative RL fine-tuning methods under the context of policy customization, which are: 1) directly regularizing the KL-divergence between the trained RL and prior policies [1] during policy updates; and 2) adding the policy KL-divergence to the reward function [2, 3]. We found that both methods performed worse than residual-Q in the most challenging Parking environment --- RL fine-tuning failed to find a good trade-off between the imitative and add-on rewards. In contrast, residual-Q is able to find the policy that jointly optimizes the basic and add-on reward. 

**Comparison with Inverse RL.** R2 asked for a comparison between inverse RL (IRL) and residual-Q. Indeed, learning the inherent reward via adversarial IRL (AIRL) [4] and then combining the inferred reward with task-specific rewards is a feasible solution to policy customization. However, AIRL could perform significantly worse than GAIL in certain environments [5]. Our residual-Q framework does not require inferring the inherent reward from demos; Thus, it allows flexible adoption of any maximum-entropy imitation learning algorithms depending on the environments (e.g., AIRL, GAIL, and IQ-learn [6]).

[1] A. Nair, et al. "Awac: Accelerating online reinforcement learning with offline datasets." 2020.\
[2] Y. Wu, et al. "Behavior regularized offline reinforcement learning." 2019.\
[3] M. Ziegler, et al. "Fine-tuning language models from human preferences." 2019.\
[4] J. Fu, et al. "Learning robust rewards with adversarial inverse reinforcement learning." ICLR 2018.\
[5] A. Gleave, et al. "Imitation: Clean imitation learning implementations." 2022.\
[6] D. Garg, et al. "Iq-learn: Inverse soft-q learning for imitation." NeurIPS 2021.

### Additional Experiments
We understand that the current experiments do not fully reflect the wide scope of applications our proposed theoretical framework could have. To address this issue, we conducted experiments in MuJoCo environments. The selected environments are Ant-v3, Humanoid-v3, and Hopper-v3. The basic task is to control the robot to move along the positive $x$ direction. During policy customization, we added an add-on reward to encourage the robot to also move along the positive $y$ direction for Ant and Humanoid. For Hopper, we added an add-on reward to encourage the robot to jump higher. The results are summarized in the attached file. In all the environments, the residual-Q customized policies achieved 1) higher add-on rewards than the prior policies; 2) a trade-off between the basic and add-on tasks similar to the RL full policies. In contrast, the RL fine-tuning baseline (i.e., the one described in Appx. D.1) tends to achieve higher add-on rewards but lower basic rewards compared to residual-Q and RL full policies. The total rewards of the RL fine-tuning baseline are also always lower than residual-Q. The results further validate that residual-Q outperforms RL fine-tuning in policy customization problems. The only exception is Humanoid with IL prior. Note that this IL prior was trained by BC since we did not find hyperparameters to let GAIL succeed. The BC prior is not an ideal prior suitable for residual-Q, as it does not follow the maximum-entropy policy distribution. It is reasonable that residual-Q performs worse than RL fine-tuning given the less diverse BC prior, since residual-Q relies on the prior policy to encode the basic task reward, whereas RL fine-tuning only uses the prior policy as a regularization. 

Meta-review: This paper addresses the problem of imitations learning in settings where an initial policy learned from demonstrations is intended to be adapted to a variety of downstream tasks. The authors formulate the problem as policy customization with several inferred reward components, and an approach called residual Q-learning designed to address the problem. The introduced approach is supported with theoretical insights and empirical validation.

Initial reviews saw a large difference in reviewer opinions. Noted strengths included the soundness of the approach as well as its generality, novelty and motivation of the problem formulation, demonstrated feasibility and scalability of the approach as shown in empirical validation, as well as a generally clear presentation. At the same time, concerns remained regarding the clarity and correctness of the presented theory, conceptual and empirical comparison with related work, such as fine-tuning for RL, as well as more general limitations of the empirical validation (baselines, tasks). 

Reviewers noted that many concerns were resolved during the rebuttal period and assessments have become more positive overall. One remaining negative review initially noted an error in the theoretical part of the manuscript. The authors have resolved this issue in the rebuttal, and the relevant reviewer has indicated that their assessment has improved.

Overall, the assessment of the paper is now positive, with several reviewers strongly supporting acceptance. It is the ACs opinion that the clarifications, corrections and improvements discussed in the rebuttal period can be incorporated before the camera ready deadline. The recommendation is to accept, and the authors are strongly encouraged to take all feedback into account as they prepare the final version.
Decision: Accept (poster)