Title: Residual Q-Learning: Offline and Online Policy Customization without Value
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

imitation learning
Abstract: Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. To this end, we formulate a new problem setting called policy customization. It defines the learning task as training a policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by a target downstream task. We propose a novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q-learning, which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy. We derive a family of residual Q-learning algorithms that can realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: https://sites.google.com/view/residualq-learning.

Residual Q-Learning: Offline and Online Policy Customization without Value

Abstract

Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. To this end, we formulate a new problem setting called policy customization . It defines the learning task as training a policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by a target downstream task. We propose a novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q-learning (RQL) , which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy . We derive a family of residual Qlearning algorithms that can realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: https://sites.google.com/view/residualq-learning .

1 Introduction

Imitation learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complicated real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic expert behavior. For instance, IL has been widely studied for synthesizing human-like agents from large-scale human demonstrations on challenging real-world tasks, such as autonomous driving [4, 7, 2] and robot manipulation [59, 37, 39]. However, when applying the imitative policy in open real-world scenarios, we may encounter different circumstances that impose diverse requirements beyond merely following the behavior in the demonstration, such as reaching particular goals [45], enforcing safety [36], or complying with certain behavior preferences [61]. In this case, we need to customize the imitative policy to meet those task-specific objectives. Meanwhile, in many applications, we may still want the customized policy to possess some inherent characteristics of the expert demonstration (e.g., human-like).

To this end, we formulate a new problem setting we refer to as policy customization . In the policy customization problem, we are given a pre-trained policy, and the task is to train a new policy that 1) inherits the properties of the prior policy; 2) satisfies some additional requirements imposed by a given downstream task. At first glance, this newly formulated problem may seem to be seamlessly solved by existing reinforcement learning (RL) fine-tuning algorithms [44, 40, 61, 51], where RL is applied to fine-tune a pre-trained policy obtained from behavior cloning (BC) or offline RL [31, 29, 34]. In particular, some RL fine-tuning methods combine different forms of IL objectives with RL to accelerate policy training, e.g., maximizing the policy likelihood on demonstration [44], regularizing the KL-divergence between the fine-tuned and prior policies [40, 61]. Different from the policy customization problem we consider, the IL objectives are introduced heuristically as regularization in RL fine-tuning. Solving policy customization instead requires a principled way to design the learning objective to jointly optimize the policy's performance on the imitative and downstream tasks, which has not been well-studied in the RL fine-tuning literature.

In this work, we propose such a principled approach to interpret and design the policy customization objective with a solid theoretical basis. Formally, we formulate policy customization as a Markov decision process (MDP) whose reward function is a linear combination of 1) the inherent reward of the demonstration; and 2) an add-on reward specified by the downstream task. The reward weight directly determines the trade-off between the imitative and downstream tasks for policy optimization. However, we cannot solve this MDP with existing RL algorithms, since we can only access the imitative policy without knowing its underlying reward.

To tackle this challenge, we propose a novel Residual Q-Learning (RQL) framework that can solve the formulated MDP without knowing the inherent reward or value function of the prior policy . Specifically, we define a residual Q-function that, together with the log-likelihood of the prior policy, can conveniently construct the maximum-entropy policy solving the target MDP. We show that the residual Q-function can be iteratively learned with an update rule derived from soft Bellman equation [17]. Hence, the proposed RQL method can be applied on top of any value-based maximum-entropy RL algorithms. In particular, we derive residual soft Q-learning and residual soft actor-critic-two model-free policy customization algorithms through RL fine-tuning-which are adapted from soft Q-learning [17] and soft actor-critic [18] respectively. In addition, we introduce a model-based policy customization algorithm adapted from maximum-entropy Monte Carlo Tree Search (MCTS) [55], which enables zero-shot online policy customization when a dynamics model is available. In our experiments, we show that the proposed RQL algorithms can effectively customize the policies toward the additional task objective while maintaining their performance on the basic tasks for which the prior policies are trained.

2 Policy Customization with Residual Q-Learning

2.1 Problem Formulation: Policy Customization

We consider a MDP defined by the tuple M = ( S , A , r, p ) , where S is the state space, A is the action space, and r : S × A → R is the reward function. Without loss of generality, we introduce the formulation under continuous state and action spaces. The state transition probability function p : S × A × S → [0 , ∞ ) represents the probability density of the next state s t +1 ∈ S given the current state s t ∈ S and action a t ∈ A . While the following formulation applies whether the reward function r is given or not, we are primarily interested in the case when r remains unknown and, instead, expert demonstrations are available to indicate desirable behaviors. We assume that we have access to a prior policy π : S × A → [0 , ∞ ) pre-trained with imitation learning [15, 22] methods. In particular, we follow the common practice and model π as a maximum-entropy policy to account for sub-optimal demonstration [60], which is especially important for human demonstration. Specifically, the policy π maximizes the entropy-augmented cumulative reward that is inherent in the demonstrations. The resulting optimal policy follows a Boltzmann distribution [17] as follows:

where Q ∗ ( s , a ) is the soft Q-function as defined in [17] and it satisfies the soft Bellman equation:

and Z s is the normalization factor defined as ∫ A exp ( 1 α Q ∗ ( s , a ) ) d a . The temperature coefficient α determines the relative importance of entropy and reward in the maximum-entropy policy [17]. It is straightforward to derive from Eqn. (1) that:

Instead of directly deploying the policy π , we aim to customize it to satisfy some additional requirements for a given downstream task, specified by an add-on reward function r R : S × A → R . Meanwhile, we want the customized policy to inherit the characteristics of the original policy. Formally, we aim to leverage the pre-trained prior policy π to train a new policy for the MDP defined by ˆ M = ( S , A , ωr + r R , p ) . The new reward function is defined as a weighted sum of the basic reward r and the add-on reward r R so that the customized policy meets both requirements. Under this setting, the main challenge lies in how to find the optimal policy for the new task without knowing the prior reward r but only the imitative policy π .

2.2 Residual Q-Learning

Since the reward r is unknown, we cannot directly train the new policy to maximize the total reward through RL. One plausible solution is to learn the inherent reward via inverse RL [12] and then combine the inferred reward with the add-on reward for RL training. However, practical inverse RL algorithms require jointly optimizing the policy and reward function, which could be challenging to yield good performance in certain environments when compared to imitation learning [16]. Instead, we aim to explore a solution without the need to explicitly infer the underlying reward to derive a more flexible policy customization framework. It turns out that we can leverage the imitative policy π to construct the new policy without knowing π 's underlying reward or value function . The proposed method can be applied on top of any value-based maximum-entropy RL algorithms [17, 18, 55], which rely on the soft Bellman update operator defined as:

where ˆ Q t is the estimated soft Q-function for ˆ M at the t th iteration. Instead of directly applying the soft Bellman update operator to iterate ˆ Q t during policy learning, we define a residual Q-function as Q R,t := ˆ Q t -ωQ ∗ and attempt to iterate this residual Q-function during policy learning. We then derive an update rule for Q R,t that does not depend on the reward or value function of π :

where Eqn. (5b) results from expressing the reward r with the soft Q-function Q ∗ based on Eqn. (2). Eqn. (5c) results from the definition of the residual Q-function and Eqn. (3). Please refer to Appendix A for a step-by-step breakdown of the derivation process. In each iteration, we can then define the policy corresponding to the current estimated ˆ Q t without computing ˆ Q t :

One remaining issue is that we do not have access to the underlying temperature coefficient α of the prior policy π . Note that α is multiplied upon the reward weight ω in Eqn. (5) and Eqn. (6). In practice, the reward weight is a hyperparameter to tune. Hence, we can define ω ′ = ωα and directly treat ω ′ as the hyperparameter without the need to know the true α .

3 Practical Algorithms

In this section, we present three practical algorithms for policy customization under different settings based on the update rule of residual Q-function in Eqn. (5) and the maximum-entropy policy defined in Eqn. (6). In these algorithms, the goal is to find a policy that solves the MDP ˆ M given a prior policy π and the add-on reward function r R . We first introduce two model-free algorithms to learn the residual Q-function and customize the policy through additional RL steps, which are adapted from soft Q-learning [17] and soft actor-critic [18] respectively. Then we introduce a model-based policy customization algorithm adapted from maximum-entropy MCTS [55], which enables zero-shot online policy customization when a dynamics model is available.

Residual Soft Q-Learning. We adapt the soft Q-learning [17] algorithm to a residual soft Q-learning algorithm for offline policy customization (i.e., policy customization via additional RL training steps) when the action space is discrete. The residual algorithm is essentially the same as the soft Q-learning. The only difference is that we use a function approximator parameterized by θ to model the residual Q-function instead of the soft Q-function and denote it as Q R,θ ( s , a ) . At each iteration, the parameters are updated by taking gradient steps to minimize the temporal-difference (TD) error:

where D is the replay buffer and the target residual Q-value Q target R, ¯ θ ( s t , a t ) is computed as:

with ¯ θ being the target parameters. The above objective function can be derived straightforwardly from the original objective function of soft Q-learning. We provide the detailed derivation in Appendix B for completeness. With a discrete action space, we can compute the policy distribution exactly given the residual Q-function and the prior policy as in Eqn. (6).

Residual Soft Actor-Critic. For tasks with continuous action spaces, we adapt the soft actorcritic [18] algorithm to a residual soft actor-critic algorithm. We aim to train a parameterized residual Q-function Q R,θ ( s , a ) and a policy network ˆ π ϕ ( a | s ) by alternating between policy evaluation and improvement. In the policy evaluation step, we update the residual Q-function for evaluating the current policy. The loss function is essentially the one in Eqn. (7) but with the target residual Q-value computed with respect to the current policy ˆ π ϕ :

The formula for the residual Q-value is derived from the modified Bellman update operator of soft actor-critic [18] following a similar procedure as in Eqn. (5). The detailed derivation is attached to Appendix C for completeness. In the policy improvement step, we improve the policy ˆ π ϕ leveraging the current estimated residual Q-value Q R,θ ( s , a ) and the prior policy π by minimizing the expected KL-divergence between ˆ π ϕ and the desired maximum-entropy policy given the current Q R,θ ( s , a ) , which is equivalent to minimizing the loss function below:

In our experiments, we implement the algorithm based on the practical soft actor-critic algorithm presented in [19], which incorporates auto entropy adjustment and double residual Q-functions [14].

Residual Maximum-Entropy MCTS. The last algorithm we introduce is a model-based policy customization algorithm adapted from the maximum-entropy MCTS [55] algorithm, which we refer to as residual maximum-entropy MCTS. Our primary interest is in applying this algorithm in the online setting with an offline learned dynamics model. MCTS essentially solves an online planning

problem leveraging the prior policy and add-on reward function, which enables zero-shot online policy customization. Such a paradigm is particularly useful for problem domains like autonomous driving, where large-scale human driving datasets [11, 58, 53] are available for learning good imitative policies [1, 46] as well as accurate traffic predictive models for online planning [10, 33]. Note that MCTS-based methods inherently assume a discrete action space. In practice, we may extend it to tasks with continuous action space by discretizing the action space [47, 57].

To solve the online planning problem, maximum-entropy MCTS constructs a look-ahead tree T incrementally via Monte Carlo simulation. In the tree T , each node n ( s ) ∈ T corresponds to a state s . For each action a ∈ A , the node n ( s ) stores an estimated soft Q-value ˆ Q ( s , a ) and a visit count N ( s , a ) . At each iteration, simulations are conducted starting from the root node of the search tree, which consists of two stages: 1) querying a tree policy to select actions and traversing the tree until a leaf node is reached; 2) querying an evaluation function (e.g., Monte Carlo simulations leveraging a roll-out policy) at the leaf node to estimate the return-to-go. After simulation, the estimated returns are propagated backward to update all the nodes along the simulated paths. Afterward, the tree is grown by expanding the leaf nodes reached during the simulations. Different from a typical MCTS algorithm [8], maximum-entropy MCTS exploits maximum-entropy policy optimization. Hence, the tree policy used in maximum-entropy MCTS is defined as:

where λ s = ϵ |A| / log( ∑ a N ( s , a )+1) with ϵ as a hyperparameter. Given a simulated trajectory from the root node to a leaf node, i.e., { s 0 , a 0 , · · · , s T , a T } , the Q-values of all the visited nodes are updated with a softmax backup :

where ˆ R is the return-to-go estimated by querying an evaluation function on the leaf node n ( s T ) . To adapt the algorithm to residual maximum-entropy MCTS, we instead have each node store an estimated residual Q-value Q R ( s , a ) . The tree policy is revised as:

During backward propagation, the residual Q-value is updated as follows:

with V R ( s t +1 ) = ˆ α log ∑ a ′ ∈A exp ( 1 ˆ α ( Q R ( s t +1 , a ′ ) + ω ′ log π ( a ′ | s t +1 )) ) . In particular, we use Monte Carlo simulations to estimate the return-to-go R . The roll-out policy is set as the prior policy π , and R is computed as the cumulative return of the add-on reward r R over the roll-out trajectories. While the estimated R is biased, it serves as a heuristic to guide the tree search to select nodes with future trajectories collecting high add-on rewards while staying close to the prior policy. The effect of this biased estimation will gradually be mitigated as the search tree expands to the terminal time step.

4 Related Work

Our residual Q-learning framework can be considered a principled approach to combine the objectives of IL and RL. Prior works have proposed different approaches for this purpose. In the IL literature, reward augmentation [9] has been introduced as a general framework to provide additional incentives from prior knowledge into the imitation learning process. Under the Generative Adversarial Imitation Learning (GAIL) framework, a surrogate reward can be conveniently incorporated to augment the learned discriminator for policy optimization [35, 3]. The augmented imitative reward is essentially the same as the total reward in our framework. The difference is that ours aims to further customize a trained imitative policy for downstream tasks, instead of regularizing IL with prior knowledge.

There are also a variety of works that directly design the objective function as a weighted mixture of IL and RL objectives. In some methods, like DQfD [21], a supervised classification loss is combined

with a TD loss to train the Q-function. It regularizes the learned Q-function to assign higher values to demonstrated actions. Other methods, such as DAPG [44] and BC-SAC [36], use a weighted mixture of behavior cloning loss and RL objectives in their loss function. SHAIL [25] formulates the imitation problem as a constrained RL problem with a constraint on the lower bound of policy likelihood on demonstrations. These approaches have shown that IL can help accelerate RL learning, especially under sparse rewards [21, 44], and vice versa [25]. As discussed in Sec. 1, similar objectives are also adopted in some RL fine-tuning algorithms [44, 40, 61, 51]. The pre-trained policy is not only used to initialize RL but also leveraged consistently over the fine-tuning stage as regularization. The introduction of RL has also been shown to effectively robustify behavior cloning [36] for challenging and safety-critical autonomous driving tasks. Conversely, IL regularization is used in offline RL approaches, such as TD3+BC [13] and CQL [31], to prevent overestimating the values on out-ofdistribution actions and improve the robustness of offline RL policies. Different from these works, where the additional objective is treated as regularization, our framework formulates the learning task as a new MDP and provides a principled way to interpret and design the combined objective.

TS2C [56] is similar to ours from the perspective of having an imperfect expert-the prior policy in RQL can be considered as an imperfect expert for the overall task-and fusing knowledge extracted from this imperfect expert and RL exploration. The difference is that TS2C aims to train a policy towards a known reward; Thus, a value function can be trained to assert if the expert is reliable. In our case, since we rely on the prior policy to encode the basic task reward implicitly, we cannot use such a value function to govern the fusion between imitation and RL. One work in the literature that aligns with our motivation behind MCTS online customization is Deep Imitative Model [45], which aims to direct an imitative policy to arbitrary goals during online usage. It is formulated as a planning problem solved online to find the trajectory that maximizes its posterior likelihood conditioned on the goal. In contrast, we formulate the additional requirements as rewards instead of goals, which are more flexible to specify. Also, RQL is a unified framework for both offline and online customization.

In curriculum RL, the concept of learning the residuals of Q-functions has been adopted in boosted curriculum RL (BCRL) [28, 24]. Under the BCRL framework, a task curriculum is constructed by decomposing a complex task into a sequence of subtasks with increasing difficulty. For each task of the curriculum, its Q-function is modeled and learned as the sum of residuals, with each residual corresponding to a previous task of the curriculum, which was shown to outperform fitting a single function approximator [50, 28]. Similar to RL fine-tuning, BCRL focuses on expediting RL learning on a target task with the Q-functions learned on the previous tasks. The residual is learned to construct the Q-function for the target task with a known reward. Conversely, RQL aims at learning the residual to construct the Q-function for a new task whose reward function combines the unknown underlying reward of the expert and the known add-on reward.

5 Experiments

We evaluate the proposed algorithms on four environments selected from different domains: CartPole and Continuous Mountain Car environments from the OpenAI gym classic control suite [5], and Highway and Parking from the highway-env environments [32]. In our experiments, we implemented our algorithms upon Stable-Baselines3 [43] and its imitation library [16]. In Sec. 5.1, we provide the configurations of our experiments, including the settings of policy customization tasks in different environments, baselines, and evaluation metrics. In Sec. 5.2, we present and analyze the experimental results of RL offline policy customization. In Sec. 5.3, we demonstrate an example of zero-shot online customization leveraging the residual maximum-entropy MCTS algorithm. In Sec. 5.4, we investigate two representative RL fine-tuning methods under the context of policy customization and compare them with the proposed algorithms. Please refer to the appendices for detailed experiment configurations, implementation details, ablation studies, and additional experiments in MuJoCo [49].

5.1 Experiment Setup

Environments. The detailed configurations of the environments can be found in Appendix D. We mainly describe the policy customization task settings here. In each environment, we design the basic and add-on rewards to illustrate a practical application scenario, where the basic reward corresponds to some basic task requirements and the add-on reward reflects customized specifications such as behavior preference or additional task objectives.

CartPole balancing: In the cartpole environment, the basic task is to balance the pole as long as possible. During policy customization, we demand an additional task that requires the cart to stay at the center of the rack. To validate if the additional objective is satisfied, we monitor the average absolute position error of the cart which is defined as ¯ e x = ∑ T t =1 | x t | / T , where x t is the coordinate of the cart with respect to the center of the rack.

Continuous mountain car: In the mountain car environment, the basic task is to climb the mountain with the least energy consumption. During policy customization, we enforce an additional preference to avoid negative actions whenever possible. This corresponds to use cases where certain actions are less favorable. To monitor the change in action distribution, we count the number of negative actions executed during each episode, denoted as n neg .

Highway navigation: In the highway environment, the basic task is to navigate the vehicle safely and efficiently through the traffic. The vehicle can switch to arbitrary lanes to overtake the other vehicles. During policy customization, we enforce an additional preference to stay on the rightmost lane whenever possible. This reproduces driving habits that human drivers may have in real life. For example, drivers may prefer to drive on the rightmost lane to avoid missing the exits or when they want to drive the car slowly. We compute the average normalized lane index over an episode to validate if the desired behavior emerges, which is defined as ¯ I lane = ∑ T t =1 I lane ,t / 2 T , where I lane ,t ∈ { 0 , 1 , 2 } is the lane index with the leftmost lane as the lane 0 and the rightmost lane as the lane 2.

Parking: In the parking environment, the basic task is to park the car in the target parking slot. During policy customization, we add an additional requirement to avoid touching the boundaries of the parking slots during parking, which is considered good practice to avoid scratching other cars and to leave sufficient space for neighboring parking slots. We compute the non-violation rate to monitor if the requirement is met, which is defined as the percentage of episodes where the constraint is not violated, denoted as γ no -violation .

Baselines. In our experiments, we mainly compare the performance of four categories of policies:

RL prior policy : We train an RL policy optimizing the basic reward. It serves as the prior policy for policy customization. For environments with discrete action space (i.e., CartPole and Highway ), we train the RL prior policy using soft Q-learning. For environments with continuous action space (i.e., Continuous Mountain Car and Parking ), soft actor-critic is used. At test time, we also evaluate its performance on the overall task without customization, which serves as a baseline to show the effectiveness of the proposed algorithms.

IL prior policy : We train another IL prior policy that imitates the RL prior with GAIL [22]. As discussed in Sec. 2, the proposed RQL algorithms are primarily motivated to customize imitative policy without knowledge of its inherent reward. The IL prior policy serves as a baseline for comparison with the residual-Q policy customized from the IL prior policy. In GAIL, we use soft Q-learning as the policy learning algorithm when the action space is discrete, and use soft actor-critic when the action space is continuous.

Residual-Q customized policies (Ours) : In each environment, we train two residual-Q customized policies leveraging the RL and IL prior policies respectively. They are mainly compared against the corresponding prior policies to validate the effectiveness of policy customization. We apply residual soft Q-learning for environments with discrete action space and apply the residual soft actor-critic for environments with continuous action space. Besides, we evaluate the zero-shot customization ability of the proposed residual maximum-entropy MCTS algorithm in the Highway environment.

RL with total reward : We also train an RL policy with the total reward (i.e., r + ωr R ). Note that we do not intend to treat it as a baseline for fair comparison under the policy customization setting. We aim to compare the residual-Q policies with it to validate that the customized policies indeed solve the overall task ˆ M .

Metric. We are mainly interested in validating the performance of the proposed policy customization methods from two perspectives: 1) whether the policy is customized toward the add-on reward; 2) whether the customized policy still possesses the characteristics of the prior policy, in other words, maintains the same level of performance on the basic task. Hence, we use the average basic reward and the success rate of completing the basic task as metrics to evaluate the policies' performance on

the basic task. Meanwhile, we use the average add-on reward and task-specific metrics introduced above as the metrics for evaluating the policies' performance with respect to the customized objective.

5.2 RL Offline Customization

The experimental results of RL offline customization are summarized in Table 1. In all the environments and with either RL or IL prior policies, the customized policies significantly outperform the prior policies on the add-on task objectives. Meanwhile, the customized policies maintain the same level of performance as the prior policies on the basic tasks. It validates that the proposed residual-Q algorithms can tailor the policies for customized requirements while maintaining their original characteristics. One interesting observation is that the policy customization improves the performance of the IL policy on the basic task in the Parking environment. One factor is that GAIL does not perform well in the first place. The Parking environment is indeed more challenging than the other environments given the continuous action space and, more importantly, the nature of the task-navigation in tight space under the non-holonomic constraint. Another aspect we think can explain this phenomenon is that the add-on reward implicitly guides the policy to improve its basic task performance. As shown in Appendix G, it is difficult for the imitative policy to stop the vehicle accurately at the target parking slot. Since GAIL merely learns to match the trajectory distributions under the expert and learned policies without the knowledge of the actual task objective, it is challenging for the imitative policy to precisely localize such a narrow parking space. In contrast, the add-on boundary violation constraint confines the vehicle trajectory within the parking space once the vehicle approaches the target parking slot. Hence, the policy is able to reach the goal point faster and receives higher returns even on the basic parking task.

In Table 1, we also compare the customized policies against the RL policies trained with the total reward function, named RL Full Policy in the table. In environments with discrete action spaces, the residual soft-Q learning policies behave very similarly to the RL full policies. Meanwhile, the residual soft actor-critic agents have different reward patterns compared to the RL full policies. We think the behavior difference is mainly due to the approximation errors in value and policy networks. The proposed residual-Q framework relies on the prior policy to acquire knowledge of the basic task. Thus, a suboptimal prior will affect the performance of the customized policy, especially on the basic task. It also explains the performance difference between the policies customized from RL and IL priors. Inspired by the observation mentioned above on IL policy customization, we think a practical solution to mitigate this issue is partially incorporating some basic task objectives into the add-on reward if available. It is reasonable for many practical use cases even with an imitative prior. While it is difficult to precisely describe the underlying reward function of human experts, we can comfortably assume that they follow some obvious commonsense objectives (e.g., collision avoidance). In this way, our method can be considered a more principled framework for the general synergy between IL and RL beyond policy customization where the reward functions are completely decomposed.

5.3 MCTS Online Customization

We choose the Highway environment to demonstrate the zero-shot online customization ability of residual maximum-entropy MCTS. We implement residual maximum-entropy MCTS based on the MCTS planner provided by the highway-env environment. The planner leverages the ground-truth dynamics model of the environment for roll-out simulation. The results are summarized in Table 1. MCTS online customization is able to achieve similar performance as the RL offline counterpart.

5.4 Ablation Study: Comparison with RL Fine-tuning

In our RQL framework, we formulate policy customization as an MDP whose reward function balances the utilities of the imitative and add-on tasks. It provides the theoretical foundation to determine the synergy between the IL and RL objectives in a principled manner. As discussed in Sec. 1 and Sec. 4, different approaches have been proposed in the RL fine-tuning literature to combine IL and RL objectives. We investigate two representative methods under the context of policy customization. We compare them against RQL and summarize the experimental results in this subsection. Please refer to Appendix F for the complete ablation study.

Greedy Reward Decomposition. One common approach is regularizing the divergence between the trained RL and prior policies during policy updates. It has been widely used in offline RL algorithms

to address the distributional shift issue [30, 54], and Advantage Weighted Actor-Critic (AWAC) [40] demonstrates the benefits of such policy regularization in stabilizing and accelerating RL fine-tuning. As shown in AWAC [40], when using KL-divergence to measure the distance between policies, the regularized optimal policy becomes a maximum-entropy policy with the advantage weighted by the prior policy. If we directly adapt it to solve the policy customization problem, the customized policy is essentially defined as the solution to the following optimization problem at each policy update step:

where ˜ Q R,t is the soft Q-function of the MDP with only the add-on reward r R ( s , a ) , which is defined iteratively with the following update rule:

The closed-form solution to the optimization problem is given as:

We can reduce our RQL policy to the same form by simply replacing the residual Q-function in Eqn. (6) with the soft Q-function ˜ Q R,t . The resulting policy solves the optimization problem in Eqn. (15) but with an entropy weight of ˆ α -λ in the objective function. If we consider it for the policy customization problem, it is essentially equivalent to heuristically estimating the optimal Q-function as the sum of the optimal Q-functions corresponding to the basic and add-on rewards. Hence, the synthesized policy is not the optimal solution to the target MDP but a greedy approximation [52, 26]. We compare RQL with this greedily customized policy on the Continuous Mountain Car and Parking environments. The results can be found in Appendix F.1. We found that this greedy customized policy performed much worse than RQL in the most challenging Parking environment. Compared to the prior policy, greedy customization compromises the policy's success rate on the basic parking task to customize them towards satisfying the add-on objective. However, the greedy policy still has a significantly lower non-violation rate compared to the RQL policy. It validates that our framework is a more principled method to jointly optimize the IL and RL objectives for policy customization than divergence-based regularization.

Augmenting Reward with Policy KL-Divergence. Alternatively, some works [54, 61] have explored adding the policy divergence to the reward function. It has been shown to outperform policy regularization in offline RL setting [54]. To adopt this line of approaches for policy customization, we formulate the reward function of the target MDP as the sum of the add-on reward and a policy divergence regularization term. In particular, we follow [61] to augment the add-on reward with a penalty whose expectation regularizes the KL-divergence between the customized and prior policies. Formally, the reward function is defined as:

where ˇ π t denotes the customized policy after the t th iteration of policy update, and β is a hyperparameter to balance the add-on reward and the policy regularization term. We can then apply standard RL algorithms to customize the policy toward this reward function. However, we found that it made the learning task much more difficult in our policy customization problem-as shown in Appendix F.2, soft actor-critic, for example, failed to find a customized policy that at least performed well on the basic parking task. We believe that one crucial factor contributing to the failure of divergence-based regularization is that the basic and add-on rewards are mostly orthogonal in our setting. The add-on reward only encodes additional task requirements; thus, we heavily rely on the prior policy to embed the desirable behavior on the basic task. Merely regularizing the KL-divergence is therefore insufficient-being close to the prior policy in terms of probabilistic distance does not necessarily imply behavior that accomplishes the basic task objective. Besides, incorporating the policy likelihood into the reward function makes it more challenging to update the Q-function in practice. For example, in Appendix F.3, we show that our RQL algorithm is theoretically equivalent to directly solving an MDP whose reward function is a weighted sum of the add-on reward and the log-likelihood of the prior policy. However, similar to the case of divergence-augmented reward, soft actor-critic failed to find a solution in Parking environment.

The evaluation results are computed over 4000 episodes for model-free policies and 100 episodes for maximumentropy MCTS (ME-MCTS). The statistics with ± are in the format of mean ± std .

6 Limitations

As discussed in Sec. 5.2, one limitation of our current algorithms is that the performance of policy customization is bottlenecked by the imitative prior policy. Policy customization not only requires it to perform well on the trajectory distribution under the imitative prior policy itself but, more importantly, we need the imitative policy to accurately indicate desired behavior with respect to the basic task objective on the trajectory distribution under the customized policy. Hence, without careful design, our algorithms would suffer from the distributional shift issue. While, in practice, this issue can be mitigated by partially incorporating some basic task objectives into the add-on reward if available, a more principled and fundamental solution is to obtain a diverse imitative prior that is best suited for customization. Generative models, such as normalizing flows [48] and, more recently, diffusion models [23, 6, 20], have been shown promising in constructing such a diverse behavior prior. Besides, incorporating human experts in the loop has also been shown to be effective in improving the generalizability of imitation learning [27, 38, 41, 33]. We will investigate their applications in the policy customization problem in future work. Also, we only demonstrate MCTS online customization with an ideal perfect dynamics model in our current experiments. In future work, we will combine residual maximum-entropy MCTS with offline learned predictive models to achieve zero-shot online customization in more practical scenarios.

References

M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In Proceedings of Robotics: Science and Systems , FreiburgimBreisgau, Germany, June 2019.

R. Bhattacharyya, B. Wulfe, D. J. Phillips, A. Kuefler, J. Morton, R. Senanayake, and M. J. Kochenderfer. Modeling human driving behavior through generative adversarial imitation learning. IEEE Transactions on Intelligent Transportation Systems , 2022.

R. P. Bhattacharyya, D. J. Phillips, C. Liu, J. K. Gupta, K. Driggs-Campbell, and M. J. Kochenderfer. Simulating emergent properties of human driving behavior using multi-agent reward augmented imitation learning. In 2019 International Conference on Robotics and Automation (ICRA) , pages 789-795. IEEE, 2019.

M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316 , 2016.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 , 2023.

F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 4693-4700. IEEE, 2018.

R. Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In Computers and Games: 5th International Conference, CG 2006, Turin, Italy, May 29-31, 2006. Revised Papers 5 , pages 72-83. Springer, 2007.

P. Englert, N. A. Vien, and M. Toussaint. Inverse kkt: Learning cost functions of manipulation tasks from demonstrations. The International Journal of Robotics Research , 36(13-14):14741488, 2017.

J. L. V. Espinoza, A. Liniger, W. Schwarting, D. Rus, and L. Van Gool. Deep interactive motion prediction and planning: Playing games with motion prediction models. In Learning for Dynamics and Control Conference , pages 1006-1019. PMLR, 2022.

S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y . Chai, B. Sapp, C. R. Qi, Y . Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9710-9719, 2021.

J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.

S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-learn: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems , volume 34, pages 4028-4039, 2021.

A. Gleave, M. Taufeeque, J. Rocamonde, E. Jenner, S. H. Wang, S. Toyer, M. Ernestus, N. Belrose, S. Emmons, and S. Russell. imitation: Clean imitation learning implementations. arXiv preprint arXiv:2211.11972 , 2022.

T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning , pages 1352-1361. PMLR, 2017.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573 , 2023.

T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems , volume 29, 2016.

M. Janner, Y. Du, J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991 , 2022.

S. Jauhri, J. Peters, and G. Chalvatzaki. Robot learning of mobile manipulation with reachability behavior priors. IEEE Robotics and Automation Letters , 7(3):8399-8406, 2022.

K. Judah, A. Fern, P. Tadepalli, and R. Goetschalckx. Imitation learning with demonstrations and shaping rewards. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, 2014.

Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez. Explainable reinforcement learning via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence , 2019.

M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8077-8083. IEEE, 2019.

P. Klink, C. D'Eramo, J. Peters, and J. Pajarinen. Boosted curriculum reinforcement learning. In International Conference on Learning Representations , 2021.

I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

E. Leurent. An environment for autonomous driving decision-making. GitHub repository , 2018.

C. Li, T. Trinh, L. Wang, C. Liu, M. Tomizuka, and W. Zhan. Efficient game-theoretic planning with prediction heuristic for socially-compliant autonomous driving. IEEE Robotics and Automation Letters , 7(4):10248-10255, 2022.

J. Li, C. Tang, M. Tomizuka, and W. Zhan. Dealing with the unknown: Pessimistic offline reinforcement learning. In Conference on Robot Learning , pages 1455-1464. PMLR, 2022.

Y. Li, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in Neural Information Processing Systems , 30, 2017.

Y. Lu, J. Fu, G. Tucker, X. Pan, E. Bronstein, B. Roelofs, B. Sapp, B. White, A. Faust, S. Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. arXiv preprint arXiv:2212.11419 , 2022.

A. Mandlekar, D. Xu, R. Martín-Martín, S. Savarese, and L. Fei-Fei. Learning to generalize across long-horizon tasks from human demonstrations. arXiv preprint arXiv:2003.06085 , 2020.

A. Mandlekar, D. Xu, R. Martín-Martín, Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in-the-loop imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733 , 2020.

A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning , pages 1678-1690. PMLR, 2022.

A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Z. Peng, Q. Li, C. Liu, and B. Zhou. Safe driving via expert guided policy optimization. In Conference on Robot Learning , pages 1554-1563. PMLR, 2022.

A. Raffin. Rl baselines3 zoo. GitHub repository , 2020.

A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 , 2017.

N. Rhinehart, R. McAllister, and S. Levine. Deep imitative models for flexible inference, planning, and control. In International Conference on Learning Representations , 2020.

O. Scheel, L. Bergamini, M. Wolczyk, B. Osi´ nski, and P. Ondruska. Urban driver: Learning to drive from real-world demonstrations using policy gradients. In Conference on Robot Learning , pages 718-728. PMLR, 2022.

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10024 , 2020.

E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012.

S. Tosatto, M. Pirotta, C. d'Eramo, and M. Restelli. Boosted fitted q-iteration. In International Conference on Machine Learning , pages 3434-3443. PMLR, 2017.

I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. K. Fu, C. Ma, J. Jiao, S. Levine, and K. Hausman. Jump-start reinforcement learning. In NeurIPS 2021 Robot Learning Workshop, RSS 2022 Scaling Robot Learning Workshop , 2022.

H. Van Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, and J. Tsang. Hybrid reward architecture for reinforcement learning. Advances in Neural Information Processing Systems , 30, 2017.

B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493 , 2023.

Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

C. Xiao, R. Huang, J. Mei, D. Schuurmans, and M. Müller. Maximum entropy monte-carlo planning. Advances in Neural Information Processing Systems , 32, 2019.

Z. Xue, Z. Peng, Q. Li, Z. Liu, and B. Zhou. Guarded policy optimization with imperfect online demonstrations. In The Eleventh International Conference on Learning Representations , 2022.

W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems , 34:25476-25488, 2021.

W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann, J. Kummerle, H. Konigshof, C. Stiller, A. de La Fortelle, et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. arXiv preprint arXiv:1910.03088 , 2019.

T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 5628-5635. IEEE, 2018.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a problem setting called policy customization, which involves training a policy that inherits the characteristics of a prior policy while satisfying additional requirements imposed by a target downstream task. They present a novel approach to interpreting and determining the two task objectives' trade-offs. Specifically, they formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines the inherent reward of the demonstration and the additional reward specified by the downstream task. The authors introduce a new framework called Residual Q-learning that leverages the prior policy to solve the formulated MDP without knowledge of the prior policy's inherent reward or value function. They derive a family of residual Q-learning algorithms capable of offline and online policy customization, demonstrating their effectiveness in accomplishing policy customization tasks in various environments.\n\nSTRENGTHS:\n1. The paper introduces a novel problem setting, policy customization, which addresses an important issue in imitation learning. This innovative approach better meets the personalized requirements of imitation policies in practical applications.\n\n2. The paper provides experimental validation by conducting experiments in various environments, demonstrating the effectiveness of the proposed algorithms in achieving policy customization for real-world tasks. These experiments verify the practicality and adaptability of the methodology, providing strong support for further research and applications.\n\n3. The proposed methodology in the paper is shown to be feasible and scalable. It does not rely on specific assumptions or constraints, making it applicable to different scenarios and tasks. The scalability of the approach allows for its extension and adaptation to more complex and diverse environments.\n\nWEAKNESSES:\n1. The derivation process in Equation 5 lacks clarity, particularly the transition from eq. 5b to eq. 5c and eq. 5d. It is necessary to provide a more detailed and step-by-step derivation to enhance the understanding of the readers. A comprehensive explanation of the intermediate steps and the mathematical transformations involved would greatly improve the transparency and rigor of the derivations.\n\n2. While the proposed setting in the paper is intriguing, there is a need for further clarification on the rationale behind its adoption. It is unclear why the chosen measure is employed, and it would be beneficial to provide a clear justification for this decision. Additionally, if a prior policy is available, elaborating on how policy expansion can be utilized to address the problem posed in the paper would help readers grasp the underlying reasoning.\n\n3. The experimental evaluation in this paper is limited in its scope. To substantiate the claims made, it is recommended to conduct more comprehensive experiments and provide extensive results. Expanding the experimental analysis with diverse scenarios, benchmark datasets, and evaluation metrics would contribute to a more thorough evaluation and provide stronger evidence to support the proposed approach.\n\n4. The derived forms of Residual SQL and SAC in Eq. 7-10 appear to be similar to the original formulations. It is necessary to provide a clear explanation of how to interpret and understand the derived results. Highlighting the differences, if any, and elucidating the implications of the derived forms compared to the original ones would enhance the clarity and significance of the proposed modifications.\n\nQUESTIONS:\nPlease refer to the Weakness.\n\nLIMITATIONS:\nPlease refer to the Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new problem setting called pulicy customization, where the goal is to train a policy that inherits the characteristics of a prior policy while satisfying some additional requirements imposed by a downstream task. To solve this problem, this paper uses a novel and principled framework by formulating it as a MDP with a reward function that combines the inherent reward of the demonstration and the add-on reward of the downstream task. The framework is consistent with many prior RL algorithms, and leads to residual Q-learning, residual SAC, and residual ME MCTS. A comprehensive evaluation of the proposed residual Q-learning framework is conducted on various types of environments, and algorithms with residual Q outperform the prior RL or IL policies in all of them.\n\nSTRENGTHS:\n- It introduces a novel and general problem setting of policy customization that can be applied to various domains and tasks.\n- It proposes a principled and flexible framework of residual Q-learning that can customize a policy without knowing the reward or value function of the prior policy, which can be easily added to existing algorithms.\n- Following the previous point, it provides a family of algorithms that can learn the residual Q-function and customize the policy.\n- For experiments, it demonstrates the effectiveness and robustness of the residual Q-learning framework on both discrete and continuous tasks, showing that the customized policies can inherit the characteristics of the prior policy while satisfying the downstream task requirements.\n\nWEAKNESSES:\nWhile this paper has some vague weaknesses and limitations, they are fully discussed in Section 6 (Discussion and Limitations). So I would like to leave blank here.\n\nQUESTIONS:\nThere is no direct questions but I am curious about the following points.\n- Does the performance of residual-Q highly rely on the performance of the prior policy? For example, if the prior policy is learned from a mediocre dataset in an offline manner, will the residual Q-learning solve the following downstream task?\n- What is the relationship between the policy customization problem and the continuous learning problem? Is the policy customization a special case of continuous RL or transfer RL?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n<< I have read the authors' rebuttal and have raised my score based on the discussion >>\n\nThis paper introduces a novel problem setting, named 'policy customization,' which seeks to train a new policy that inherits the properties of an existing pre-trained policy and meets additional requirements from a given downstream task. The policy customization is formulated as a Markov Decision Process (MDP), with a reward function that combines the inherent reward of the demonstration and an additional reward defined by the downstream task. To address the challenge of solving the MDP without knowledge of the underlying reward of the imitative policy, the paper proposes a novel Residual Q-Learning framework. This includes defining a residual Q-function that, when combined with the log-likelihood of the prior policy, can construct the maximum-entropy policy for the target MDP. The paper presents two model-free policy customization algorithms, namely, 'residual soft Q-learning' and 'residual soft actor-critic' and empirically demonstrates that the proposed residual Q-learning algorithms effectively customize the policies toward the additional task objective, while preserving their performance on the original tasks for which the prior policies were trained.\n\nSTRENGTHS:\nS1. Clarity of Presentation: The paper is commendably articulated, offering clear motivations for the research and a thorough explanation of the proposed solution. The technical details are presented in a digestible manner, which significantly enhances the reading experience.\n\nS2. Technical Soundness: The paper exhibits robust technical soundness. The equations and underlying logic are laid out explicitly, providing a comprehensive understanding of the objectives. The problem formulation adheres to rigorous technical standards.\n\nS3. Novelty: The paper contributes original research to a specific problem domain - policy customization - and I have not seen papers targeting this problem. As such, the solution proposed is novel.\n\nOverall, the paper provides a compelling read and I found the research intriguing. However, some concerns about the experimental setup were noted (refer to the 'Weaknesses' section for details).\n\nWEAKNESSES:\nDespite the innovative approach presented in this paper, certain aspects of the experimental setup present potential limitations:\n\nW1. The selection of relatively simple problems for testing constrains the perceived impact of the work. With more complex environments like Atari and Procgen games now serving as standard test suites, the study's relevance might be diminished due to the primitive nature of the chosen experimental domains.\n\nW2. The paper lacks a comparison with a directly relevant baseline — RL fine-tuning combined with the IL objective of the prior policy used as heuristics. Given that RL or IL prior policies are trained on objectives distinct from the downstream task, their performance on the latter is likely to be sub-optimal. A head-to-head comparison with this method, which is a natural consideration for the policy customization problem, would better establish the strengths and limitations of the proposed Residual Q-learning framework.\n\nQUESTIONS:\nQ1. This work proves beneficial when the downstream task's objective is orthogonal to the prior policy's objective. However, what happens when the objectives conflict? For instance, suppose a prior policy for robot navigation is conservatively trained to minimize speed and avoid collision. If the downstream task necessitates rapid completion, wouldn't the prior policy potentially impede the optimization of the downstream task's objective?\n\nQ2. If possible, could you share the results of experiments where the Residual Q-learning framework is compared with RL fine-tuning combined with the IL objective of the prior policy used as heuristics? Such comparisons would provide a more comprehensive evaluation of the proposed approach.\n\nQ3. Based on the results, \"RL with total reward\" consistently outperforms the Residual Q-learning framework. Could you elucidate potential real-world scenarios where the Residual Q-learning framework would be more advantageous or preferred over the \"RL with total reward\" approach?\n\nLIMITATIONS:\nThe authors discussed the limitations of their work. No discussion needed regarding potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new problem setting called policy customization, where the goal is to train a new policy that inherits the characteristics of a prior policy while satisfying additional requirements imposed by a downstream task. The authors propose a novel Residual Q-Learning framework to solve this problem, which leverages the prior policy without knowing its inherent reward or value function. The authors derive a family of residual Q-learning algorithms for offline and online policy customization and show their effectiveness in various environments. The empirical results on four tested environments seem promising.\n\nSTRENGTHS:\n- The paper introduces a new problem setting called policy customization, which addresses the need to customize a pre-trained policy to meet additional requirements imposed by a downstream task.\n- The Residual Q-Learning framework proposed in this paper provides a principled approach to interpret and design the policy customization objective, enabling the joint optimization of imitative and downstream task performance.\n\nWEAKNESSES:\n- The evaluation for the proposed method is limited. It would be beneficial to see more extensive comparisons and discussions with more baselines in the field, such as EGPO [1] and TS2C [2]. Also it would be beneficial to include some larger-scale RL environments, such as MuJoCo or MetaDrive [3].\n- A more comprehensive discussion with related works should be presented. There is a whole line of research that assumes the availability of an online expert policy to imitate [1,2,4,5,6]. With these methods, it will be natural to perform policy customization by adding task-specific rewards to the learning process.\n\nQUESTIONS:\n- What is the advantage of Residual Q-Learning when compared with inverse-RL? One can learn the inherent reward function from interactions and combine it with the additional task-specific reward.\n- What does it mean by “principled way”? Why is Residual Q-Learning better than related works?\n- How is the prior policy $\\pi$ used? Are you assuming you have an “online” policy to imitate? Then you may also need to compare with those papers focusing on RL with shared control [1,2,6].\n\nLIMITATIONS:\nReferences\n\n[1] Peng, Zhenghao, et al. \"Safe driving via expert guided policy optimization.\" Conference on Robot Learning. PMLR, 2022.\n\n[2] Xue, Zhenghai, et al. \"Guarded Policy Optimization with Imperfect Online Demonstrations.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[3] Li, Quanyi, et al. \"Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning.\" IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3461-3475.\n\n[4] Kelly, Michael, et al. \"Hg-dagger: Interactive imitation learning with human experts.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n\n[5] Mandlekar, Ajay, et al. \"Human-in-the-loop imitation learning using remote teleoperation.\" arXiv preprint arXiv:2012.06733 (2020).\n\n[6] Li, Quanyi, Zhenghao Peng, and Bolei Zhou. \"Efficient learning of safe driving policy via human-ai copilot optimization.\" arXiv preprint arXiv:2202.10341 (2022).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel concept policy customization, which trains a new policy based on a pre-trained policy so as to solve the downstream tasks. Compared to RL fine-tuning algorithms, policy customization directly optimize the imitative objectives and downstream RL objectives jointly. The authors propose a new approach, Residual Q-learning, for policy customization. The proposed algorithm has shown effectiveness across 4 different environments in this work, suggesting their potential for efficient policy customization.\n\nSTRENGTHS:\n1. The paper tackles an important problem in RL, learning new tasks based on given policy priors.\n2. The formulation and theorem are sound.\n3. The approach is based on a simple concept \"residual optimization\", yet appears to generate some benefits, especially in harder tasks.\n\nWEAKNESSES:\n1. The author should clarify the significance of policy customization and make more arguments about the difference between policy customization and RL fine-tuning methods. The main difference between them seems that policy customization jointly trains the reward from the inherent rewards and add-on rewards, while RL fine-tuning methods learn from inherent rewards more intrinsically. Why is policy customization deserved to study, or why jointly combining the reward significant? Furthermore, I haven't noticed a detailed argument about the necessity of the proposed novel concept.\n2. The experimental environments are simple. More complex tasks are better. Moreover, the authors should compare with some RL fine-tuning methods mentioned in related works.\n\nQUESTIONS:\nThe optimization is based on the assumption that the optimal policy pre-trained with IL follows a Boltzmann distribution. The imitation policies in experiments are from stable-baselines' library. But for some more practical or complex envs, such as CARLA and MuJoCo, this assumption is hard to satisfy since the expert demonstrations have more and larger bias. I'm wondering the Eq (6) cannot be satisfied, how to optimize the Residual Q-Learning problems?\n\nLIMITATIONS:\n1. The statement about the definition and significance of policy customization is poor. \n2. The experiment environments are simple, lacking convincing.\n3. The assumption in Eq 1 is strong.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}