Residual-MPPI: Online Policy Customization for Continuous Control

## Abstract

Abstract Policies learned through Reinforcement Learning (RL) and Imitation Learning (IL) have demonstrated significant potential in achieving advanced performance in continuous control tasks. However, in real-world environments, it is often necessary to further customize a trained policy when there are additional requirements that were unforeseen during the original training phase. It is possible to fine-tune the policy to meet the new requirements, but this often requires collecting new data with the added requirements and access to the original training metric and policy parameters. In contrast, an online planning algorithm, if capable of meeting the additional requirements, can eliminate the necessity for extensive training phases and customize the policy without knowledge of the original training scheme or task. In this work, we propose a generic online planning algorithm for customizing continuous-control policies at the execution time which we call Residual-MPPI . It is able to customize a given prior policy on new performance metrics in few-shot and even zero-shot online settings. Also, Residual-MPPI only requires access to the action distribution produced by the prior policy, without additional knowledge regarding the original task. Through our experiments, we demonstrate that the proposed Residual-MPPI algorithm can accomplish the few-shot/zero-shot online policy customization task effectively, including customizing the champion-level racing agent, Gran Turismo Sophy (GT Sophy) 1.0, in the challenging car racing scenario, Gran Turismo Sport (GTS) environment. Demo videos are available on our website: https://sites.google.com/view/residual-mppi

## 1 Introduction

Policy learning algorithms such as Reinforcement Learning (RL) and Imitation Learning (IL) have been widely employed to synthesize parameterized control policies for a wide range of real-world motion planning and decision-making problems, such as navigation [, ], manipulation [, , , ] and locomotion [, , ]. In practice, real-world applications often impose additional requirements on the trained policies beyond those established during training, which can include novel goals [], specific behavior preferences [], and stringent safety criteria []. Retraining a new policy network whenever a new additional objective is encountered is both expensive and inefficient, as it may demand extensive training efforts. To enable the flexible deployment of embodied AI systems, it is thereby crucial to develop sample-efficient algorithms for synthesizing new policies that meet additional objectives while preserving the characteristics of the original policy [, ].

Recently, Li et al. [] introduced a new problem setting termed policy customization, which provides a principled approach to address the aforementioned challenge. In policy customization, the objective is to develop a new policy given a prior policy, ensuring that the new policy: 1) retains the properties of the prior policy, and 2) fulfills additional requirements specified by a given downstream task. As an initial solution, the authors proposed the Residual Q-learning (RQL) framework. For discrete action spaces, RQL can leverage maximum-entropy Monte Carlo Tree Search [] to customize the policy online, meaning at the execution time of the policy. In contrast, for continuous action spaces, RQL offers a solution based on Soft Actor-Critic (SAC) [] to train a customized policy leveraging the prior policy before the real execution. While SAC-based RQL is more sample-efficient than training a new policy from scratch, the additional training steps that are required may still be expensive and time-consuming.

In this work, we propose *online policy customization* for *continuous action spaces* that eliminates the need for additional policy training. We leverage Model Predictive Path Integral (MPPI) [], a sampling-based model predictive control (MPC) algorithm. Specifically, we propose Residual-MPPI, which integrates RQL into the MPPI framework, resulting in an online planning algorithm that customizes the prior policy at the execution time. The policy performance can be further enhanced by iteratively collecting data online to update the dynamics model. Our experiments in MuJoCo demonstrate that our method can effectively achieve zero-shot policy customization with a provided offline trained dynamics model. Furthermore, to investigate the scalability of our algorithm in complex environments, we evaluate Residual-MPPI in the challenging racing game environment, Gran Turismo Sport (GTS), in which we successfully customize the driving strategy of the champion-level racing agent, GT Sophy 1.0 [], to adhere to additional safety constraints.

Figure: Figure 1: Overview of the proposed algorithm. In each planning loop, we utilize the prior policy to generate samples and then evaluate them through both the log likelihood of the prior policy and an add-on reward to obtain the customized actions. More details are in Sec. 3. In the experiments, we demonstrate that Residual-MPPI can accomplish the online policy customization task effectively, even in a challenging GTS environment with the champion-level racing agent, GT Sophy 1.0.
Refer to caption: x1.png

## 2 Preliminaries

In this section, we provide a concise introduction to two techniques that are used in our proposed algorithm: RQL and MPPI, to establish the foundations for the main technical results.

### 2.1 Policy Customization and Residual Q-Learning

We consider a discrete-time MDP problem defined by a tuple $\mathcal{M}=(\mathcal{X},\mathcal{U},r,p)$, where $\mathcal{X}\subseteq\mathcal{R}^{n}$ is a continuous state space, $\mathcal{U}\subseteq\mathcal{R}^{m}$ is a continuous action space, $r:\mathcal{X}\times\mathcal{U}\rightarrow\mathbb{R}$ is the reward function, and $p:\mathcal{X}\times\mathcal{U}\times\mathcal{X}\rightarrow[0,\infty)$ is the state transition probability density function. The prior policy $\pi$ is fully trained as an optimal maximum-entropy policy to solve this MDP problem by performing the distribution of action $\boldsymbol{u}_{t}$ with a given current state $\boldsymbol{x}_{t}$.

Meanwhile, the add-on task is specified by an add-on reward function $r_{R}:\mathcal{X}\times\mathcal{U}\rightarrow\mathbb{R}$. Therefore, the full task becomes a new MDP defined by a tuple $\hat{\mathcal{M}}=(\mathcal{X},\mathcal{U},\omega r+r_{R},p)$, where the reward is defined as a weighted sum of the basic reward $r$ and the add-on reward $r_{R}$. Thus, finding a policy that solves this new MDP accomplishes the policy customization task.

Li et al. [] proposed the residual Q-learning framework to solve the policy customization task. Given the prior policy $\pi$, RQL is able to find this customized policy without knowledge of the prior reward $r$, which ensures broader applicability across different prior policies obtained through various methods, including those solely imitating demonstrations. In particular, as shown in their appendix [], when we have access to the prior policy $\pi$, finding the maximum-entropy policy solving the full MDP problem $\hat{\mathcal{M}}=(\mathcal{X},\mathcal{U},\omega r+r_{R},p)$ is equivalent to solving an augmented MDP problem $\mathcal{M}^{\mathrm{aug}}=(\mathcal{X},\mathcal{U},\omega^{\prime}\log\pi(
\boldsymbol{u}|\boldsymbol{x})+r_{R},p)$, where $\omega^{\prime}$ is a hyper-parameter that balances the optimality between original and add-on tasks.

### 2.2 Model Predictive Path Integral

Model Predictive Path Integral (MPPI) [] is a sampling-based model predictive control (MPC) algorithm, which approximates the optimal solution of an (infinite-horizon) MDP through receding-horizon planning.
MPPI evaluates the control inputs $U=(\boldsymbol{u}_{0},\boldsymbol{u}_{1},\cdots,\boldsymbol{u}_{T-1})$ with an accumulated reward function $S_{\boldsymbol{x}_{0}}(U)$ defined by a reward function $r$ and a terminal value estimator $\phi$:

$$ $S_{\boldsymbol{x}_{0}}(U)=\sum_{t=0}^{T-1}r(\boldsymbol{x}_{t},\boldsymbol{u}_ {t})+\phi(\boldsymbol{x}_{T}),\\$ (1) $$

where the intermediate state $\boldsymbol{x}_{t}$ is calculated by recursively applying the transition dynamics on $\boldsymbol{x}_{0}$.

By applying an addictive noise sequence $\mathcal{E}=(\boldsymbol{\epsilon}_{0},\boldsymbol{\epsilon}_{1},\dots,
\boldsymbol{\epsilon}_{T-1})$ to a nominal control input sequence $\hat{U}$, MPPI obtains a disturbed control input sequence as $U=\hat{U}+\mathcal{E}$ for subsequent optimization, where $\mathcal{E}$ follows a multivariate Gaussian distribution with its probability density function defined as
$p(\mathcal{E})=\prod_{t=0}^{T-1}\left((2\pi)^{m}|\Sigma|\right)^{-\frac{1}{2}}
\exp\left(-\frac{1}{2}\boldsymbol{\epsilon}_{t}^{T}\Sigma^{-1}\boldsymbol{
\epsilon}_{t}\right)$, where $m$ is the dimension of the action space.

As shown by Williams et al. [], the optimal action distribution that solves the MDP is

$$ $q^{*}(U)=\frac{1}{\eta}\exp\left(\frac{1}{\lambda}S_{\boldsymbol{x}_{0}}(U) \right)p(\mathcal{E}),$ (2) $$

where $\eta$ is the normalizing constant and $\lambda$ is a positive scalar variable.
MPPI approximates this distribution by assigning an importance sampling weight $\omega(\mathcal{E}^{k})$ to each noise sequence $\mathcal{E}^{k}$ to update the nominal control input sequence.

$$ $\displaystyle\boldsymbol{u}_{t}=\hat{\boldsymbol{u}}_{t}+\sum_{k=1}^{K}\omega( \mathcal{E}^{k})\boldsymbol{\epsilon}^{k}_{t},$ (3) $$

where $K$ is the number of samples, and $\omega(\mathcal{E}^{k})$ could be calculated as

$$ $\displaystyle\omega(\mathcal{E}^{k})=\frac{1}{\mu}\left(S_{\boldsymbol{x}_{0}} (\hat{U}+\mathcal{E}^{k})-\frac{\lambda}{2}\sum_{t=0}^{T-1}\hat{\boldsymbol{u} }^{T}_{t}\Sigma^{-1}(\hat{\boldsymbol{u}}_{t}+2\boldsymbol{\epsilon}^{k}_{t}) \right),$ (4) $$

where $\mu$ is the normalizing constant.

## 3 Method

Our objective is to address the policy customization challenge under the online setting for general continuous control problems. We aim to leverage a pre-trained prior policy $\pi$, along with a dynamics model $F$, to approximate the optimal solution to the augmented MDP problem $\mathcal{M}^{\mathrm{aug}}$, in an online manner. To address this problem, we propose a novel algorithm, Residual Model Predictive Path Integral (Residual-MPPI), which is broadly applicable to any maximum-entropy prior policy with a dynamics model of the environment. The proposed algorithm is summarized in Algorithm 1 and Figure 1.
In this section, we first establish the theoretical foundation of our approach by verifying the maximum-entropy property of MPPI. We then refine the MPPI method with the derived formulation to approximate the solution of the $\mathcal{M}^{\mathrm{aug}}$.
Lastly, we discuss the dynamics learning and fine-tuning method used in our implementation.

Figure: Algorithm 1 Residual-MPPI

### 3.1 Residual-MPPI

MPPI is a widely used sampling-based MPC algorithm that has demonstrated promising results in various continuous control tasks. To achieve online policy customization, i.e. solving the augmented MDP $\mathcal{M}^{\mathrm{aug}}$ efficiently during online execution, we utilize MPPI as the foundation of our algorithm.

As shown in the preliminary, RQL requires the planning algorithm to comply with the principle of maximum entropy. To validate that MPPI adheres to the principle of maximum-entropy, we note the Theorem 1 where a step-by-step breakdown proof can be found in Appendix A

###### Theorem 1 .

Given an MDP defined by $\mathcal{M}=(\mathcal{X},\mathcal{U},r,p)$, with a deterministic state transition $p$ defined with respect to a dynamics model $F$ and a discount factor $\gamma=1$, the distribution of the action sequence $q^{*}(U)$ at state $\boldsymbol{x}_{0}$ in horizon $T$, where each action $\boldsymbol{u}_{t},t=0,\cdots,T-1$ is sequentially sampled from the maximum-entropy policy [] with an entropy weight $\alpha$ is

$$ $\displaystyle q^{*}(U)$ $\displaystyle=\frac{1}{Z_{\boldsymbol{x}_{0}}}\exp\left(\frac{1}{\alpha}\left( \sum_{t=0}^{T-1}r(\boldsymbol{x}_{t},\boldsymbol{u}_{t})+V^{*}(\boldsymbol{x}_ {T})\right)\right),$ (5) $$

where $V^{*}$ is the soft value function [] and $\boldsymbol{x}_{t}$ is defined recursively from $\boldsymbol{x}_{0}$ and $U$ through the dynamics model $F$ as $\boldsymbol{x}_{t+1}=F(\boldsymbol{x}_{t},\boldsymbol{u}_{t}),t=0,\cdots,T-1$.

If we have $\lambda=\alpha$ and let $V^{*}$ as the terminal value estimator, The distribution in Eq. (5) is equivalent to the one in Eq. (2), that is the optimal distribution that MPPI tries to approximate, but under the condition that the $p(\mathcal{E})$ is a Gaussian distribution with infinite variance, i.e. a uniform distribution. It suggests that MPPI can well approximate the maximum-entropy optimal policy with a discount factor $\gamma$ close to $1$ and a large noise variance. We can then derive Residual-MPPI straightforwardly by defining the evaluation function $S_{\boldsymbol{x}_{0}}(U)$ in MPPI as

$$ $S^{\mathrm{aug}}_{\boldsymbol{x}_{0}}(U)=\sum_{t=0}^{T-1}\gamma^{t}\cdot\left( r_{R}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})+\omega^{\prime}\log\pi( \boldsymbol{u}_{t}|\boldsymbol{x}_{t})\right),$ (6) $$

to solve the $\mathcal{M}^{\mathrm{aug}}$, therefore approximates the optimal customized policy online.

Additionally, the performance and sample-efficiency of MPPI depend on the approach to initialize the nominal input sequence $\hat{U}$. In our context, the prior policy serves as a natural source for initializing the nominal control inputs. As shown at line 2 in Algorithm 1, by recursively applying the prior policy and the dynamics, we could initialize a nominal trajectory with a tunable exploration noise to construct a Gaussian prior distribution for sampling. During implementation and experiments, we found that using the nominal action sequence itself as a candidate sequence for evaluation can effectively increase sampling stability.

###### Theorem 1 .

### 3.2 Dynamics Learning and Fine-tuning

In scenarios where an effective dynamics model is unavailable in a prior, it is necessary to develop a learned dynamics model. To this end, we established a dynamics training pipeline utilizing the prior policy for data collection. In our implementation, we employed three main techniques to enhance the model’s capacity for accurately predicting environmental states as follows:

Multi-step Error:
The prediction errors of an imperfect dynamics model accumulate over time steps. In the worst case, compounding multi-step errors can grow exponentially []. To ensure the accuracy of the dynamics model over the long term, we use a multi-step error $\sum_{t=0}^{T}\gamma^{t}\times(s_{t}-\hat{s_{t}})^{2}$ as the loss function for dynamics training, where $s_{t}$ and $\hat{s_{t}}$ are the ground-truth and predicted state.

Exploration Noise:
The prior policy’s behavior sticks around the optimal behavior to the prior task, which means the roll-out samples collected using the prior policy concentrated to a limited region in the state space. It limits the generalization performance of the dynamics model under the policy customization setting. Therefore, during the sampling process, we add Gaussian exploration noises to the prior policy actions to enhance sample diversity.

Fine-tuning with Online Data:
Since the residual-MPPI planner solves the customized task objective, its behavior is different from the prior policy. Thus, the residual-MPPI planner may encounter states that are not contained in the dynamics training dataset collected using the prior policy. Therefore, the learned dynamics model could be inaccurate on these unseen states. In this case, we can follow the common model-based RL routine to iteratively collect data using the residual-MPPI planner and update the dynamics model using the collected in-distribution data.

## 4 MuJoCo Experiments

**Table 1: Experimental Results of Zero-shot Residual-MPPI in MuJoCo**
| Env. | Policy | Full Task | Basic Task | Add-on Task |  |
| --- | --- | --- | --- | --- | --- |
| $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{|\theta|}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |  |  |
| Half<br>Cheetah | Prior Policy | $1004.79\pm 38.88$ | $2452.30\pm 23.55$ | $0.14\pm 0.00$ | $-1447.50\pm 31.29$ |
| Full-MPPI | $-3590.68\pm 315.50$ | $-1172.52\pm 142.35$ | $0.24\pm 0.03$ | $-2418.16\pm 314.14$ |  |
| Guided-MPPI | $1821.19\pm 285.44$ | $2137.76\pm 179.70$ | $0.03\pm 0.01$ | $-316.57\pm 108.01$ |  |
| Residual-MPPI | $\mathbf{1948.14\pm 34.56}$ | $2189.45\pm 28.87$ | $\mathbf{0.02\pm 0.00}$ | $\mathbf{-241.31\pm 28.86}$ |  |
| Env | Policy | $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{|\theta|}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |
| Swimmer | Prior Policy | $-245.01\pm 5.95$ | $346.24\pm 3.44$ | $0.59\pm 0.01$ | $-591.24\pm 6.40$ |
| Full-MPPI | $-1677.68\pm 124.86$ | $13.46\pm 6.27$ | $1.69\pm 0.12$ | $-1691.13\pm 124.17$ |  |
| Guided-MPPI | $-147.77\pm 5.54$ | $294.28\pm 3.87$ | $0.44\pm 0.01$ | $-442.05\pm 7.61$ |  |
| Residual-MPPI | $\mathbf{-61.00\pm 5.20}$ | $276.21\pm 3.49$ | $\mathbf{0.34\pm 0.01}$ | $\mathbf{-337.21\pm 7.67}$ |  |
| Env. | Policy | $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{z}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |
| Hopper | Prior Policy | $7250.67\pm 49.39$ | $3575.02\pm 9.35$ | $1.37\pm 0.00$ | $3675.02\pm 48.21$ |
| Full-MPPI | $21.09\pm 3.52$ | $3.72\pm 0.81$ | $1.24\pm 0.00$ | $17.37\pm 2.93$ |  |
| Guided-MPPI | $6146.35\pm 1605.06$ | $3073.04\pm 682.71$ | $1.36\pm 0.03$ | $3073.31\pm 927.26$ |  |
| Residual-MPPI | $\mathbf{7398.29\pm 103.92}$ | $3560.97\pm 19.04$ | $\mathbf{1.38\pm 0.01}$ | $\mathbf{3837.32\pm 105.61}$ |  |
| Env | Policy | $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{v}_{y}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |
| Ant | Prior Policy | $6299.17\pm 957.16$ | $6133.41\pm 891.45$ | $0.17\pm 0.22$ | $165.76\pm 221.35$ |
| Full-MPPI | $-2792.20\pm 154.94$ | $-2779.91\pm 120.28$ | $-0.01\pm 0.08$ | $-12.29\pm 87.81$ |  |
| Guided-MPPI | $5485.85\pm 1773.90$ | $5277.66\pm 1670.04$ | $0.21\pm 0.26$ | $208.19\pm 260.70$ |  |
| Residual-MPPI | $\mathbf{6808.38\pm 599.48}$ | $5969.61\pm 490.13$ | $\mathbf{0.84\pm 0.20}$ | $\mathbf{838.77\pm 198.08}$ |  |

In this section, we evaluate the performance of the proposed algorithms in different environments selected from MuJoCo []. In Sec. 4.1, we provide the configurations of our experiments, including the settings of policy customization tasks in different environments, baselines, and evaluation metrics. In Sec. 4.2, we present and analyze the experimental results. Please refer to the appendices for detailed experiment configurations, implementations, and visualizations.

### 4.1 Experiment Setup

Environments. In each environment, we design the basic and add-on rewards to illustrate a practical application scenario, where the basic reward corresponds to some basic task requirements and the add-on reward reflects customized specifications such as behavior preference or additional task objectives. The detailed configurations of the environments can be found in Appendix B.1.

Baselines. In our experiments, we mainly compare the performance of the proposed residual-MPPI algorithm against three baselines:

- -
Prior Policy:
We utilize SAC to train the prior policy on the basic task for policy customization. At test time, we evaluate its performance on the overall task without customization, which serves as a baseline to show the effectiveness of the proposed algorithm.
- -
Guided-MPPI:
We introduced Guided-MPPI from the idea proposed in the literature [, ] that leverages the prior policy to guide planning. The samples are obtained from the same prior policy used in Residual-MPPI. Note that Guided-MPPI utilizes the unbiased ground truth reward (i.e., $\omega r+r_{R}$). By comparing Residual-MPPI against Guided-MPPI, we aim to validate the necessity of Residual-MPPI under this ideal setting.
- -
Full-MPPI:
Additionally, we apply the MPPI with no prior on the MDP problem of the full task (i.e., $\omega r+r_{R}$). We aim to compare Residual-MPPI against it to validate that our proposed algorithm can effectively leverage the prior policy to boost online planning.

Metric. We aim to validate a policy’s performance from two perspectives: 1) whether the policy is customized toward the add-on reward; 2) whether the customized policy still maintains the same level of performance on the basic task. Therefore, we use the average basic reward as the metric to evaluate a policy’s performance on the basic task. Meanwhile, we use the average add-on reward and task-specific metrics as the metrics for evaluating a policy’s performance for the customized objective. Furthermore, we use the total reward to evaluate the joint optimality of the solved policy.

### 4.2 Results and Discussions

The experimental results, summarized in Table 1, demonstrate the effectiveness of Residual-MPPI for online policy customization. Across all the tested environments, the customized policies achieve significant improvements over the prior policies in meeting the objectives of the add-on tasks. Simultaneously, these customized policies maintain a similar level of performance as the prior policies on the basic tasks. In comparing the performance with baseline approaches, it is evident that the Guided-MPPI policies, despite utilizing the same sampling strategy and having full access to the rewards of both basic and add-on tasks, underperform in all environments relative to the proposed Residual-MPPI. Furthermore, the Full-MPPI policies, which also have the full reward information, fail in all tasks. This outcome highlights a prevalent challenge in planning-based methods: effective planning necessitates ample samples and a sufficiently long planning horizon or an accurate terminal value estimator to correctly evaluate the optimality of actions. While Guided-MPPI improves the estimation of optimal actions through more efficient sampling, it remains hindered by its limited ability to account for the long-term impacts of actions within finite planning horizons. In contrast, Residual-MPPI implicitly inherits the original task reward through the prior policy log likelihood, which is informed by the Q functions optimized over an infinite horizon or demonstrations.

## 5 Customizing Champion-level Autonomous Racing Policy

With the effective results in standard benchmark environments, we are further interested in whether the proposed algorithm can be applied to effectively customize an advanced policy for a more sophisticated continuous control task. Gran Turismo Sport (GTS) is a realistic racing simulator with high-fidelity vehicle dynamics. GT Sophy 1.0 [], a DRL-based racing policy, has shown to outperform the world’s best drivers among the global players. To further investigate the scalability of our algorithm and its robustness when dealing with more complex environments and advanced policies, we carried out further customization experiments on GT Sophy 1.0 in the GTS environment.

### 5.1 Experiment Setup

Though GT Sophy 1.0 is a highly advanced system with superhuman racing skills, it tends to exhibit too aggressive route selection as the well-trained policy has the ability to keep stable on the simulated grass. However, in real-world racing, such behaviors would be considered dangerous and fragile because of the time-varying physical properties of real grass surfaces and tires. Therefore, we establish the task as customizing the policy to a safer route selection. Formally, the customization goal is to help the GT Sophy 1.0 drive *on course* while maintaining its superior racing performance. This kind of customization objective could potentially be used to foster robust sim-to-real transfer of agile control policies for many related problem domains.

Notably, GTS runs on an independent PS5 platform with a fixed running frequency of $60$Hz, which imposes real-time requirements for online control, making it closer to real-world settings. As the dynamics model is not available in a prior, we adopt a simple MLP architecture to design a dynamics model and train it using the techniques introduced in Sec. 3.2.
The configurations of the environments and implementation details can be found in Appendix C.

As the pre-trained GT Sophy 1.0 policy is constructed with complex rewards and regulations,
we leverage the average lap time and number of off-course steps as the metrics. In addition to zero-shot Residual-MPPI, we also evaluate a few-shot version of our algorithm, in which we iteratively update the dynamics with the customized planner’s trajectories to further improve the performance. We also consider Residual-SAC [] as another baseline to validate the advantage of online Residual-MPPI against RL-based solutions in the challenging racing problem.

**Table 2: Experimental Results of Residual-MPPI in GTS**
|  | Policy | GT Sophy 1.0 | Zero-shot MPPI | Few-shot MPPI | Residual-SAC |
| --- | --- | --- | --- | --- | --- |
|  | Lap Time | $117.77\pm 0.08$ | $121.99\pm 0.57$ | $\mathbf{120.75\pm 0.37}$ | $130.00\pm 0.13$ |
|  | Off-course Steps | $93.13\pm 1.98$ | $43.53\pm 7.40$ | $\mathbf{36.60\pm 6.04}$ | $0.87\pm 0.78$ |

### 5.2 Results and Discussions

Figure: Figure 2: Typical Route Selection of Policies
Refer to caption: x2.png

The experimental results, summarized in Table 2, demonstrate that Residual-MPPI significantly enhances the safety of GT Sophy 1.0 by reducing its off-course steps, albeit with a marginal increase in lap time. Further improvements are observed after employing data gathered during the customization process to fine-tune the dynamics under a few-shot setting. This few-shot version of Residual-MPPI outperforms the zero-shot version in terms of lap time and off-course steps. Though the customized policies can not eliminate all the off-course cases, it is worth noting that these cases are significantly less severe than those in GT Sophy 1.0. The average and maximum off-course distance of GT Sophy 1.0 in a typical run is $0.69$m and $3.21$m, while for Few-shot Residual-MPPI are $0.37$m and $1.13$m.

In contrast, Residual-SAC yields a very conservative customized policy. As shown in Figure 2, it is obvious that the Residual-SAC policy behaves sub-optimally and overly yields to the on-course constraints. It is worth noting that over 80,000 laps of roll-outs are collected to achieve the current performance of Residual-SAC. In contrast, the data used to train GTS dynamics for Residual-MPPI, along with the online data for dynamics model fine-tuning, amounted to only approximately 2,000 and 100 laps respectively. As discussed in safe RL and curriculum learning methods [, ], when training with constraints, it is easy for RL to yield to an overly conservative behavior.

## 6 Related Works

In this section, we provide a concise introduction to the related works of the proposed method. Please refer to the appendices for detailed discussion. Here we briefly discuss the literature in three topics:
1) *Model-based RL:* Many works focused on combining learning-based agents with classical planning methods to enhance performance, especially in various model-based RL approaches [, ]. However, those methods mainly address the planning within the same task and have full reward information of it. As for policy customization, it requires jointly optimizing both the original task of the prior policy without access to the reward information and add-on tasks, which makes it unsolvable by these methods.
2) *Planning with Prior Policy:* Some work in the literature address the similar customization requirements of the pre-trained prior policy [, , ]. Li et al. [] consider modeling the reward with hand-designed features, which introduce further bias and constrain the overall performance. Rhinehart et al. [] formulate the additional requirements as goals, which are not flexible to specify. Residual-MCTS [] applies to discretized action, which makes it difficult to extend to continuous and more complex tasks.
3) *RL Fine-tuning:* There have been numerous works considering the topic of fine-tuning policies [, , , ]. However, these methods still require additional training for each new customization goal, which hinders flexible deployment.

## 7 Limitations and Future Work

Prior Policy and Dynamics:
Currently, Residual-MPPI is *bottlenecked* by the prior policy and dynamics.
The proposed algorithm requires its prior policy to accurately indicate the desired behavior respect to the original task.
Policies without careful modeling or sufficient training could mislead the evaluation step of the proposed algorithm and result in poor planning outcomes.
Also, the proposed algorithm relies on an accurate dynamics model to correctly rollout the states.
In the future, we can introduce more advanced methods in policies and dynamics training, such as diffusion policies [, , ] and world models [, ], to improve the prior policies and dynamics.

Terminal Value Estimation:
Similar to most online planning methods, our algorithm also needs a sufficiently long horizon to reduce the error brought by the absence of a terminal value estimator. Though utilizing prior policy partially addresses the problem as we discussed in Sec. 4.2, it can not fully eliminate the error.
Techniques like Jump-start RL [] and RL-driven MPPI [] could be incorporated in the proposed framework in the few-shot setting by using the collected data to learn a Residual Q-function online as a terminal value estimator, which will be one of our future works.

## Acknowledgment

The authors thank Ce Hao for valuable discussions and kind help.
This work was supported by Sony AI, and Polyphony Digital Inc., which provided the Gran Turismo Sport framework.

## Appendix A Derivation

In this section, we provide a detailed proof of Theorem 1. We start with introducing a lemma to expand the action distribution $q^{*}(U)$.

###### Lemma 1 .

Given an MDP defined by $\mathcal{M}=(\mathcal{X},\mathcal{U},r,p)$, with a deterministic state transition $p$ defined with respect to a dynamics model $F$, the distribution of the action sequence $q^{*}(U)$ at state $\boldsymbol{x}_{0}$ in horizon $T$, where each action $\boldsymbol{u}_{t},t=0,\cdots,T-1$ is sequentially sampled from the maximum-entropy policy [] is

$$ $\displaystyle q^{*}(U)=\ \prod_{t=0}^{T-1}\pi^{*}(\boldsymbol{u}_{t}| \boldsymbol{x}_{t}).$ (7) $$

###### Proof.

Let $\pi^{*}(\boldsymbol{u}_{0},\boldsymbol{u}_{1},\cdots,\boldsymbol{u}_{T-1}|
\boldsymbol{x}_{0})$ denote the expanded notation of $q^{*}(U)$ by substituting $u_{t}$ and maximum-entropy policy $\pi^{*}$. Firstly, we apply the conditional probability formula to expand the equation:

$$ $\pi^{*}(\boldsymbol{u}_{0},\boldsymbol{u}_{1},\cdots,\boldsymbol{u}_{T-1}| \boldsymbol{x}_{0})=\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\int_{ \mathcal{X}}\pi^{*}(\boldsymbol{u}_{1},\cdots,\boldsymbol{u}_{T-1}|\boldsymbol {x}_{0},\boldsymbol{u}_{0},\boldsymbol{x}_{1}^{\prime})p(\boldsymbol{x}_{1}^{ \prime}|\boldsymbol{x}_{0},\boldsymbol{u}_{0})d\boldsymbol{x}_{1}^{\prime}.\\$ (8) $$

Considering the Markov property of the problem,

$$ $\pi^{*}(\boldsymbol{u}_{k},\cdots,\boldsymbol{u}_{k+N}|\boldsymbol{x}_{0}, \boldsymbol{x}_{1}^{\prime},\cdots,\boldsymbol{x}_{k-1}^{\prime},\boldsymbol{u }_{0},\boldsymbol{u}_{k-1},\boldsymbol{x}^{\prime}_{k})=\pi^{*}(\boldsymbol{u} _{k},\cdots,\boldsymbol{u}_{k+N}|\boldsymbol{x}^{\prime}_{k}),$ (9) $$

Eq. (8) could be further expanded recursively till the end:

$$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0},\boldsymbol{u}_{1},\cdots,\boldsymbol{ u}_{T-1}|\boldsymbol{x}_{0})=$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\int_{\mathcal{X}} \pi^{*}(\boldsymbol{u}_{1},\cdots,\boldsymbol{u}_{T-1}|\boldsymbol{x}_{0}, \boldsymbol{u}_{0},\boldsymbol{x}_{1}^{\prime})p(\boldsymbol{x}_{1}^{\prime}| \boldsymbol{x}_{0},\boldsymbol{u}_{0})d\boldsymbol{x}_{1}^{\prime}.$ (10) $\displaystyle=$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\int_{\mathcal{X}} \pi^{*}(\boldsymbol{u}_{1},\cdots,\boldsymbol{u}_{T-1}|\boldsymbol{x}_{1}^{ \prime})p(\boldsymbol{x}_{1}^{\prime}|\boldsymbol{x}_{0},\boldsymbol{u}_{0})d \boldsymbol{x}_{1}^{\prime}$ $\displaystyle=$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\int_{\mathcal{X}} \pi^{*}(\boldsymbol{u}_{1}|\boldsymbol{x}^{\prime}_{1})p(\boldsymbol{x}_{1}^{ \prime}|\boldsymbol{x}_{0},\boldsymbol{u}_{0})$ $\displaystyle\int_{\mathcal{X}}\pi^{*}(\boldsymbol{u}_{2},\cdots,\boldsymbol{u }_{T-1}|\boldsymbol{x}_{1}^{\prime})p(\boldsymbol{x}_{2}^{\prime}|\boldsymbol{ x}_{1},\boldsymbol{u}_{1})d\boldsymbol{x}_{1}^{\prime}d\boldsymbol{x}_{2}^{\prime}$ $\displaystyle=$ $\displaystyle\cdots$ $\displaystyle=$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\int_{\mathcal{X}}p (\boldsymbol{x}_{1}^{\prime}|\boldsymbol{x}_{0},\boldsymbol{u}_{0})\pi^{*}( \boldsymbol{u}_{1}|\boldsymbol{x}_{1}^{\prime})$ $\displaystyle\int\cdots\int_{\mathcal{X}}\prod_{t=2}^{T-1}\pi^{*}(\boldsymbol{ u}_{t}|\boldsymbol{x}_{t}^{\prime})p(\boldsymbol{x}_{t}^{\prime}|\boldsymbol{x }^{\prime}_{t-1},\boldsymbol{u}_{t-1})d\boldsymbol{x}_{1}^{\prime}\cdots d \boldsymbol{x}_{T-1}^{\prime}.$ $$

Since the state transition $p$ is a deterministic function

$$ $p\left(\boldsymbol{x}_{t+1}^{\prime}|\boldsymbol{x}_{t},\boldsymbol{u}_{t})= \delta(\boldsymbol{x}_{t+1}^{\prime},F(\boldsymbol{x}_{t},\boldsymbol{u}_{t}) \right),$ (11) $$

the integral signs could be eliminated by defining $\boldsymbol{x}_{t+1}=F(\boldsymbol{x}_{t},\boldsymbol{u}_{t})$:

$$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0},\boldsymbol{u}_{1},\cdots,\boldsymbol{ u}_{T-1}|\boldsymbol{x}_{0})=$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\int_{\mathcal{X}}p (\boldsymbol{x}_{1}^{\prime}|\boldsymbol{x}_{0},\boldsymbol{u}_{0})\pi^{*}( \boldsymbol{u}_{1}|\boldsymbol{x}_{1}^{\prime})$ (12) $\displaystyle\int\cdots\int_{\mathcal{X}}\prod_{t=2}^{T-1}\pi^{*}(\boldsymbol{ u}_{t}|\boldsymbol{x}_{t}^{\prime})p(\boldsymbol{x}_{t}^{\prime}|\boldsymbol{x }^{\prime}_{t-1},\boldsymbol{u}_{t-1})d\boldsymbol{x}_{1}^{\prime}\cdots d \boldsymbol{x}_{T-1}^{\prime}$ $\displaystyle=$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\int_{\mathcal{X}} \delta\left(\boldsymbol{x}_{t+1}^{\prime},F(\boldsymbol{x}_{t},\boldsymbol{u}_ {t})\right)\pi^{*}(\boldsymbol{u}_{1}|\boldsymbol{x}_{1}^{\prime})$ $\displaystyle\int\cdots\int_{\mathcal{X}}\prod_{t=2}^{T-1}\pi^{*}(\boldsymbol{ u}_{t}|\boldsymbol{x}_{t}^{\prime})p(\boldsymbol{x}_{t}^{\prime}|\boldsymbol{x }^{\prime}_{t-1},\boldsymbol{u}_{t-1})d\boldsymbol{x}_{1}^{\prime}\cdots d \boldsymbol{x}_{T-1}^{\prime}$ $\displaystyle=$ $\displaystyle\pi^{*}(\boldsymbol{u}_{0}|\boldsymbol{x}_{0})\pi^{*}(\boldsymbol {u}_{1}|\boldsymbol{x}_{1})\int_{\mathcal{X}}p(\boldsymbol{x}_{2}^{\prime}| \boldsymbol{x}_{1},\boldsymbol{u}_{1})\pi^{*}(\boldsymbol{u}_{2}|\boldsymbol{x }_{2}^{\prime})$ $\displaystyle\int\cdots\int_{\mathcal{X}}\prod_{t=3}^{T-1}\pi^{*}(\boldsymbol{ u}_{t}|\boldsymbol{x}_{t}^{\prime})p(\boldsymbol{x}_{t}^{\prime}|\boldsymbol{x }^{\prime}_{t-1},\boldsymbol{u}_{t-1})d\boldsymbol{x}_{2}^{\prime}\cdots d \boldsymbol{x}_{T-1}^{\prime}$ $\displaystyle=$ $\displaystyle\cdots$ $\displaystyle=$ $\displaystyle\prod_{t=0}^{T-1}\pi^{*}(\boldsymbol{u}_{t}|\boldsymbol{x}_{t}).$ $$

∎

With Lemma 1, now we provide a detailed proof of Theorem 1.

###### Theorem 1 .

Given an MDP defined by $\mathcal{M}=(\mathcal{X},\mathcal{U},r,p)$, with a deterministic state transition $p$ defined with respect to a dynamics model $F$ and a discount factor $\gamma=1$, the distribution of the action sequence $q^{*}(U)$ at state $\boldsymbol{x}_{0}$ in horizon $T$, where each action $\boldsymbol{u}_{t},t=0,\cdots,T-1$ is sequentially sampled from the maximum-entropy policy [] with an entropy weight $\alpha$ is

$$ $\displaystyle q^{*}(U)$ $\displaystyle=\frac{1}{Z_{\boldsymbol{x}_{0}}}\exp\left(\frac{1}{\alpha}\left( \sum_{t=0}^{T-1}r(\boldsymbol{x}_{t},\boldsymbol{u}_{t})+V^{*}(\boldsymbol{x}_ {T})\right)\right),$ (13) $$

where $V^{*}$ is the soft value function [] and $\boldsymbol{x}_{t}$ is defined recursively from $\boldsymbol{x}_{0}$ and $U$ through the dynamics model $F$ as $\boldsymbol{x}_{t+1}=F(\boldsymbol{x}_{t},\boldsymbol{u}_{t}),t=0,\cdots,T-1$.

###### Proof.

The maximum-entropy policy with an entropy weight $\alpha$ solves the problem $\mathcal{M}$ following the Boltzmann distribution:

$$ $\pi^{*}(\boldsymbol{u}_{t}|\boldsymbol{x}_{t})=\frac{1}{Z_{\boldsymbol{x}_{t}} }\exp\left(\frac{1}{\alpha}Q^{*}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})\right),$ (14) $$

where $Z_{\boldsymbol{x}_{t}}$ is the normalization factor defined as $\int_{\mathcal{U}}\exp\left(\frac{1}{\alpha}Q^{*}(\boldsymbol{x}_{t},
\boldsymbol{u}_{t})\right)d\boldsymbol{u}_{t}$ and $Q^{*}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})$ is the soft Q-function as defined in []:

$$ $\displaystyle Q^{*}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=r(\boldsymbol{x}_{t },\boldsymbol{u}_{t})+\alpha\log\int_{\mathcal{U}}\exp\left(\frac{1}{\alpha}Q^ {*}\left(\boldsymbol{x}_{t+1},\boldsymbol{u}_{t+1}\right)\right)d\boldsymbol{u}.$ (15) $$

With Lemma 1, the optimal action distribution could be rewritten as:

$$ $q^{*}(U)=\prod_{t=0}^{T-1}\pi^{*}(\boldsymbol{u}_{t}|\boldsymbol{x}_{t})$ (16) $$

where each $\boldsymbol{x}_{t}$ is defined recursively from $\boldsymbol{x}_{0}$ and $U$ through the dynamics model $F$ as $\boldsymbol{x}_{t+1}=F(\boldsymbol{x}_{t},\boldsymbol{u}_{t}),t=0,\cdots,T-1$.
By substituting Eq. (14) and Eq. (LABEL:eq:_soft_Bellman_equation), Eq. (16) could be further expanded:

$$ $\displaystyle q^{*}(U)$ (17a) $\displaystyle=\frac{1}{\prod_{t=0}^{T-1}Z_{\boldsymbol{x}_{t}}}\exp\left(\frac {1}{\alpha}\sum_{t=0}^{T-1}Q^{*}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})\right)$ (17b) $\displaystyle=\frac{1}{\prod_{t=0}^{T-1}Z_{\boldsymbol{x}_{t}}}\exp\left(\frac {1}{\alpha}\sum_{t=0}^{T-1}\left(r(\boldsymbol{x}_{t},\boldsymbol{u}_{t})+ \alpha\log\int_{\mathcal{U}}\exp\left(\frac{1}{\alpha}Q^{*}\left(\boldsymbol{x }_{t+1},\boldsymbol{u}_{t+1}\right)\right)d\boldsymbol{u}\right)\right)$ (17c) $\displaystyle=\frac{1}{\prod_{t=0}^{T-1}Z_{\boldsymbol{x}_{t}}}\exp\left(\frac {1}{\alpha}\sum_{t=0}^{T-1}r(\boldsymbol{x}_{t},\boldsymbol{u}_{t})+\log\left( \prod_{t=1}^{T-1}Z_{\boldsymbol{x}_{t}}\right)+\log Z_{\boldsymbol{x}_{T}}\right)$ (17d) $\displaystyle=\frac{\prod_{t=1}^{T-1}Z_{\boldsymbol{x}_{t}}}{\prod_{t=0}^{T-1} Z_{\boldsymbol{x}_{t}}}\exp\left(\frac{1}{\alpha}\left(\sum_{t=0}^{T-1}r( \boldsymbol{x}_{t},\boldsymbol{u}_{t})+\alpha\log Z_{\boldsymbol{x}_{T}}\right )\right)$ (17e) $\displaystyle=\frac{1}{Z_{\boldsymbol{x}_{0}}}\exp\left(\frac{1}{\alpha}\left( \sum_{t=0}^{T-1}r(\boldsymbol{x}_{t},\boldsymbol{u}_{t})+V^{*}(\boldsymbol{x}_ {T})\right)\right),$ (17f) $$

where Eq. (17b) results from substituting Eq. (14) and Eq. (17c) results from substituting Eq. (LABEL:eq:_soft_Bellman_equation).

In Eq. (17f), $V^{*}(\boldsymbol{x}_{T})=\alpha\log Z_{\boldsymbol{x}_{T}}$ is the soft value function defined in [] at the terminal step. Each $\boldsymbol{x}_{t}$ is defined recursively from $\boldsymbol{x}_{0}$ and $U$ through the dynamics model $F$ as $\boldsymbol{x}_{t+1}=F(\boldsymbol{x}_{t},\boldsymbol{u}_{t}),t=0,\cdots,T-1$.
∎

###### Lemma 1 .

###### Proof.

###### Theorem 1 .

###### Proof.

## Appendix B Implementation Details in MuJoCo

All the experiments were conducted on
Ubuntu 22.04 with
Intel Core i9-9920X CPU @ 3.50GHz $\times$ 24,
NVIDIA GeForce RTX 2080 Ti,
and 125 GB RAM.

### B.1 MuJoCo Environment Configuration

In this section, we introduce the detailed configurations of the selected environments, including the basic tasks, add-on tasks, and the corresponding rewards. The action and observation space of all the environments follow the default settings in gym[mujoco]-v3.

Half Cheetah. In the HalfCheetah environment, the basic goal is to apply torque on the joints to make the cheetah run forward (right) as fast as possible.
The state and action space has 17 and 6 dimensions, and the action represents the torques applied between links.

The basic reward consists of two parts:

$$ $\displaystyle\mathrm{Forward\ Reward}$ $\displaystyle:r_{\mathrm{forward}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})= \frac{\Delta x}{\Delta t}$ (18) $\displaystyle\mathrm{Control\ Cost}$ $\displaystyle:r_{\mathrm{control}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=-0.1 \times\ ||\boldsymbol{u}_{t}||_{2}^{2}$ $$

During policy customization, we demand an additional task that requires the cheetah to limit the angle of its hind leg.
This customization goal is formulated as an add-on reward function defined as:

$$ $\displaystyle r_{R}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=-10\times|\theta_{ \mathrm{hind\ leg}}|$ (19) $$

Hopper.
In the Hopper environment, the basic goal is to make the hopper move in the forward direction by applying torques on the three hinges connecting the four body parts.
The state and action space has 11 and 3 dimensions, and the action represents the torques applied between links.

The basic reward consists of three parts:

$$ $\displaystyle\mathrm{Alive\ Reward}$ $\displaystyle:r_{\mathrm{alive}}=1$ (20) $\displaystyle\mathrm{Forward\ Reward}$ $\displaystyle:r_{\mathrm{forward}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})= \frac{\Delta x}{\Delta t}$ $\displaystyle\mathrm{Control\ Cost}$ $\displaystyle:r_{\mathrm{control}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=-0.0 01\times||\boldsymbol{u}_{t}||_{2}^{2}$ $$

The episode will terminate if the z-coordinate of the hopper is lower than 0.7, or the angle of the top is no longer contained in the closed interval $[-0.2,0.2]$, or an element of the rest state is no longer contained in the closed interval $[-100,100]$.

During policy customization, we demand an additional task that requires the hopper to jump higher along the z-axis.
This customization goal is formulated as an add-on reward function defined as:

$$ $\displaystyle r_{R}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=10\times(z-1)$ (21) $$

Swimmer.
In the Swimmer environment, the basic goal is to move as fast as possible towards the right by applying torque on the rotors.
The state and action space has 8 and 2 dimensions, and the action represents the torques applied between links.

The basic reward consists of two parts:

$$ $\displaystyle\mathrm{Forward\ Reward}$ $\displaystyle:r_{\mathrm{forward}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})= \frac{\Delta x}{\Delta t}$ (22) $\displaystyle\mathrm{Control\ Cost}$ $\displaystyle:r_{\mathrm{control}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=-0.0 001\times||\boldsymbol{u}_{t}||_{2}^{2}$ $$

During policy customization, we demand an additional task that requires the agent to limit the angle of its first rotor.
This customization goal is formulated as an add-on reward function defined as:

$$ $\displaystyle r_{R}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=-1\times|\theta_{ \mathrm{first\ rotor}}|$ (23) $$

Ant.
In the Ant environment, the basic goal is to coordinate the four legs to move in the forward (right) direction by applying torques on the eight hinges connecting the two links of each leg and the torso (nine parts and eight hinges).
The state and action space has 27 and 8 dimensions, and the action represents the torques applied at the hinge joints.

The basic reward consists of three parts:

$$ $\displaystyle\mathrm{Alive\ Reward}$ $\displaystyle:r_{\mathrm{alive}}=1$ (24) $\displaystyle\mathrm{Forward\ Reward}$ $\displaystyle:r_{\mathrm{forward}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})= \frac{\Delta x}{\Delta t}$ $\displaystyle\mathrm{Control\ Cost}$ $\displaystyle:r_{\mathrm{control}}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=-0.5 \times||\boldsymbol{u}_{t}||_{2}^{2}$ $$

The episode will terminate if the z-coordinate of the torso is not in the closed interval $[0.2,1]$. During experiments, we set the upper bound to $\inf$ for both prior policy training and planning experiments as it benefits both performances.

During policy customization, we demand an additional task that requires the ant to move along the y-axis. This customization goal is formulated as an add-on reward function defined as:

$$ $\displaystyle r_{R}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=\frac{\Delta y}{ \Delta t}$ (25) $$

### B.2 RL Prior Policy Training

The prior policies were constructed using Soft Actor-Critic (SAC) with the StableBaseline3 [] implementation. The training was conducted in parallel across 32 environments. The hyperparameters used for RL prior policies training are shown in Table 3. Since this parameter setting performed poorly in the Swimmer task, we used the official benchmark checkpoint of Swimmer from StableBaseline3. Note that the prior policies do not necessarily need to be synthesized using RL. We report the experiment results with GAIL prior policies in Appendix F.

**Table 3: RL Prior Policy Training Hyperparameters**
| Hyperparameter | Value |
| --- | --- |
| Hidden Layers | $(256,256,256)$ |
| Activation | $ReLu$ |
| $\gamma$ | $0.9$ |
| Target Update Interval | $50$ |
| Learning Rate | $1e-4$ |
| Gradient Step | $1$ |
| Training Frequency | $1$ |
| Batch Size | $128$ |
| Optimizer | $Adam$ |
| Total Samples | $6400000$ |
| Capacity | $1000000$ |
| Sampling | $Uniform$ |

### B.3 Offline Dynamics Training

The hyperparameters used for offline dynamics training are shown in Table 4.

**Table 4: MuJoCo Offline Dynamics Training Hyperparameters**
| Hyperparameter | Value |
| --- | --- |
| Hidden Layers | $(256,256,256,256)$ |
| Activation | $Mish$ |
| learning rate | $1e-5$ |
| Training Frequency | $10$ |
| Optimizer | $Adam$ |
| Batch Size | $256$ |
| Horizon | $8$ |
| $\gamma$ | $0.9$ |
| Total Samples | $200000$ |
| Capacity | $50000$ |
| Sampling | $Uniform$ |

### B.4 Planning Hyperparameters

The hyperparameters used for online planning are shown in Table 5. At each step, the planners computes the result based on the current observation. Only the first action of the computed action sequence is sent to the system.

**Table 5: Planning Hyperparameter in MuJoCo Tasks**
| Hyperparameter | Half Cheetah | Ant | Swimmer | Hopper |  |
| --- | --- | --- | --- | --- | --- |
| Horizon | $2$ | $5$ | $5$ | $8$ |  |
| Samples | $10000$ | $5000$ | $5000$ | $10000$ |  |
| Noise std. | $0.017$ | $0.005$ | $0.02$ | $0.005$ |  |
| $\omega^{\prime}$ | $1e-7$ | $1e-2$ | $1e-4$ | $2e-7$ |  |
| $\gamma$ | $0.9$ | $0.9$ | $0.9$ | $0.9$ |  |
| $\lambda$ | $5e-5$ | $5e-3$ | $1e-4$ | $1e-5$ |  |

## Appendix C Implementation Details in GTS

All the GTS experiments were conducted on PlayStation 5 (PS5) and
Ubuntu 20.04 with
12th Gen Intel Core i9-12900F $\times$ 24,
NVIDIA GeForce RTX 3090,
and 126 GB RAM.
GTS ran independently on PS5 with a fixed frequency of $60$Hz.
The communication protocol returned the observation and received the control input with a frequency of $10$Hz.

### C.1 GTS Environment Configuration

The action and observation spaces follow the configuration of GT Sophy 1.0 []. The reward used for GT Sophy 1.0 training was a handcrafted linear combination of reward components computed on the transition between the previous and current states, which included course progress, off-course penalty, wall penalty, tire-slip penalty, passing bonus, any-collision penalty, rear-end penalty, and unsporting-collision penalty []. During policy customization, we demand an additional task that requires the GT Sophy 1.0 to drive on course. This customization goal is formulated as an add-on reward function defined as:

$$ $\displaystyle r_{R}(\boldsymbol{x}_{t},\boldsymbol{u}_{t})=-1000000\times ReLu (d_{\mathrm{center}}^{2}-7.3^{2}),$ (26) $$

where $d_{\mathrm{center}}$ is the distance from the vehicle to the course center.

### C.2 Dynamics Design

We employed three main techniques to help us get an accurate dynamics model.

Historical Observations To address the partially observable Markov decision processes (POMDP) nature of the problem, we included historical observations in the input to help the model capture the implicit information.

Splitting State Space We divided the state space into two parts: the dynamic states and map information. We adopted a neural network with an MLP architecture to predict the dynamic states. In each step, we utilized the trained model to predict the dynamic states and leveraged the known map to calculate the map information based on the dynamic states.

Physical Prior
Some physical states in the dynamic states, such as wheel load and slip angle, are intrinsically difficult to predict due to the limited observation. To reduce the variance brought by these difficult states, two neural networks were utilized to predict them and other states independently.

Table 6 shows the hyperparameters used for training these two MLPs to predict dynamic states.

**Table 6: GTS Offline Dynamics Training Hyperparameters**
| Hyperparameter | Value |
| --- | --- |
| History Length | $8$ |
| Hidden Layers | $(2048,2048,2048)$ |
| Activation | $Mish$ |
| Learning Rate | $1e-5$ |
| Training Steps | $200000$ |
| Training Frequency | $5$ |
| Batch Size | $256$ |
| Optimizer | $Adam$ |
| Capacity | $2000000$ |
| Sampling | $Uniform$ |

### C.3 Planning Hyperparameters

To stabilize the planning process, we further utilized another hyperparameter $topratio$ to select limited action sequence candidates with top-tier accumulated reward. The hyperparameters used for online planning are shown in Table 7. At each step, the planners computed the result based on the current observation. Only the first action of the computed action sequence was sent to the system.

**Table 7: Planning Hyperparameter in GTS**
| Hyperparameter | Horizon | Samples | Noise std. | Top Ratio | $\omega^{\prime}$ | $\gamma$ | $\lambda$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Value | $4$ | $3000$ | $0.035$ | $0.002$ | $3$ | $0.8$ | $0.5$ |

### C.4 Residual-SAC Training

In addition to the Residual-SAC algorithm, we adopted the idea of residual policy learning [] to learn a policy that corrects the action of GT Sophy 1.0 by outputting an additive action, in which the initial policy was set as GT Sophy 1.0. As proposed in the same paper, the weights of the last layer in the actor network were initiated to be zero. The randomly initialized critic was trained alone with a fixed actor first. And then, both networks were trained together. The pipeline was developed upon the official implementation of RQL []. The training hyperparameters are shown in Table 8.

**Table 8: Residual-SAC Training Hyperparameters**
| Hyperparameter | Value |
| --- | --- |
| Hidden Layers | $(2048,2048,2048)$ |
| Activation | $ReLu$ |
| Learning Rate | $1e-4$ |
| Target Update Interval | $50$ |
| Gradient Step | $1$ |
| Training Frequency | $1$ |
| Batch Size | $256$ |
| Optimizer | $Adam$ |
| $\gamma$ | $0.9$ |
| Total Samples | $10000000$ |
| Capacity | $1000000$ |
| Sampling | $Uniform$ |

## Appendix D Extended Related Works

In this section, we provide a detailed discussion of the related works.

Model-based RL.
Many works focused on combining learning-based agents with classical planning methods to enhance performance, especially in various model-based RL approaches. MuZero [] employs its learned Actor as the search policy during the MCTS process and utilizes the Critic as the terminal value at the search leaf nodes to reduce the depth of the required search; TD-MPC2 [] follows a similar approach and extends it to continuous task by utilizing MPPI as the continuous planner. RL-driven MPPI [] chooses the distributional soft actor-critic (DSAC) algorithm as the prior policy and also adopts the MPPI as the planner. However, those methods are mainly addressing the planning within the same task and has full reward information of it. As for policy customization, it requires jointly optimizing both the original task of the prior policy without access to the reward information and add-on tasks, which makes it unsolvable by these methods.

Planning with Prior Policy. Some work in the literature addresses the similar customization requirements of the pre-trained Prior policy.
Efficient game-theoretic planning [] tries to use a planner to construct human behavior within the pre-trained prediction. However, it considers modeling the human reward with hand-designed features, which introduces further bias and constrains the overall performance.
Deep Imitative Model [] aims to direct an imitative policy to arbitrary goals during online usage. It is formulated as a planning problem solved online to find the trajectory that maximizes its posterior likelihood conditioned on the goal. Compared to theirs, we formulate the additional requirements as rewards instead of goals, which are more flexible to specify.
Residual-MCTS proposed in residual Q-learning [] addresses customization tasks in a similar setting to ours. However, its discrete nature makes it difficult to extend to continuous and more complex tasks.

RL Fine-tuning.
There have been numerous works considering on the topic of fine-tuning policies.
Jump-Start RL [] (JSRL) uses a pre-trained prior policy as the guiding policy to form a curriculum of starting states for the exploration policy. However, it still requires the complete reward information to construct the task.
Advantage Weighted Actor Critic [] (AWAC) combines sample-efficient dynamic programming with maximum likelihood policy updates to leverage large amounts of offline data for quickly performing online fine-tuning of RL policies. Nevertheless, like all these fine-tuning methods, it still requires additional training for each new customization goal, which contradicts the goal of flexible deployment in customization tasks.

## Appendix E Visualization

### E.1 MuJoCo Experiments

We visualize some representative running examples from the MuJoCo environment in Fig. 3. As shown in the plot, the customized policies achieved significant improvements over the prior policies in meeting the objectives of the add-on tasks.

Figure: Figure 3: (a) The angle of the Half Cheetah’s hind leg vs. the environmental steps. (b) The angle of the Swimmer’s first rotor vs. the environmental steps. (c) The trajectory of the Hopper robot on the $x$ and $z$ axis.(d) The trajectory of the Ant robot on the $x$ and $y$ axis.
Refer to caption: x3.png

### E.2 GTS Experiments

Figure: Figure 4: Route Selection Visualization at Different Cases. Different colors denote the different policies. In all cases, Residual-MPPI are able to customize the behavior to drive more on course. In (a) and (c), due to inaccurate dynamics, Zero-shot MPPI exhibits off-course behavior. In (b) and (d), although Few-shot MPPI and Zero-shot MPPI both drive on course, Few-shot MPPI tends to select a better route closer to the boundary with more accurate dynamics.
Refer to caption: x4.png

We visualize the racing line selected by each policy at four typical corners to illustrate the differences among policies. As shown in Fig. 4, GT Sophy 1.0 exhibited aggressive racing lines that tend to be off course. Both Zero-shot MPPI and Few-shot MPPI were able to customize the behavior to drive more on the course, while the Few-shot MPPI chose a better racing line. In contrast, the Residual-SAC yields to an overly conservative behavior and keeps driving in the middle of the course.

Figure: Figure 5: Typical complete trajectories of all policies. (a) The trajectory of GT Sophy 1.0. It finishes the lap in $117.762$s with $93$ steps off course. (b) The trajectory of Residual-SAC. It finishes the lap in $131.078$s with $2$ steps off course. (c) The trajectory of Zero-shot MPPI. It finishes the lap in $121.991$s with $42$ steps off course. (d) The trajectory of Few-shot MPPI. It finishes the lap in $120.542$s with $34$ steps off course.
Refer to caption: x5.png

We provide four typical complete trajectories of all policies in Fig. 5. Meanwhile, we visualize the difference between the actions selected by GT Sophy 1.0 and Residual-MPPI at each time step in Fig. 6.
When driving at the corner, our method exerted notable influences on the prior policy, as shown in Fig. 6 (a). When driving at the straight, our method exerted minimal influences on the prior policy since most forward-predicted trajectories remain on course, as shown in Fig. 6 (b).

Figure: Figure 6: Additive Action of Residual-MPPI at Different Case. The additive throttle and steering are all linear normalized to $-1$ to $1$.
Refer to caption: x6.png

## Appendix F Additional Experimental Results

Residual-MPPI is applicable to any prior policies trained based on the maximum-entropy principle, not limited to RL methods. In this section, we conduct additional experiments upon the same MuJoCo environments with IL prior policies. The IL prior policies are obtained through Generative Adversarial Imitation Learning (GAIL) with expert data generated by the RL prior in the previous experiments.
The hyperparameters used for training the IL prior policies are shown in Table 9, while the hyperparameters used for the learners are the same as the corresponding RL experts.

**Table 9: Hyperparameters of GAIL Imitated Polices**
| Hyperparameter | Half Cheetah | Ant | Swimmer | Hopper |
| --- | --- | --- | --- | --- |
| expert_min_episodes | $1000$ | $100$ | $1000$ | $2000$ |
| demo_batch_size | $512$ | $25000$ | $1024$ | $1024$ |
| gen_replay_buffer_capacity | $8192$ | $50000$ | $2048$ | $2048$ |
| n_disc_updates_per_round | $4$ | $4$ | $4$ | $4$ |

The results are summarized in Table 10. Similar to the experiment results with the RL priors, the customized policies achieved significant improvements over the prior policies in meeting the objectives of the add-on tasks, which demonstrates the applicability of Residual-MPPI with IL priors.

**Table 10: Experimental Results of Zero-shot Residual-MPPI with IL Prior in MuJoCo**
| Env. | Policy | Full Task | Basic Task | Add-on Task |  |
| --- | --- | --- | --- | --- | --- |
| $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{|\theta|}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |  |  |
| Half<br>Cheetah | Prior Policy | $-935.10\pm 643.57$ | $2322.33\pm 680.72$ | $0.33\pm 0.01$ | $-3257.43\pm 96.61$ |
| Full-MPPI | $-3612.30\pm 331.95$ | $-1140.82\pm 169.74$ | $0.25\pm 0.04$ | $-2471.48\pm 395.98$ |  |
| Guided-MPPI | $-679.68\pm 236.98$ | $2177.39\pm 271.87$ | $0.29\pm 0.01$ | $-2857.07\pm 124.82$ |  |
| Residual-MPPI | $\mathbf{-405.97\pm 231.53}$ | $2408.07\pm 197.93$ | $\mathbf{0.28\pm 0.01}$ | $\mathbf{-2814.04\pm 71.57}$ |  |
| Env | Policy | $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{|\theta|}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |
| Swimmer | Prior Policy | $-343.32\pm 3.18$ | $329.63\pm 1.54$ | $0.67\pm 0.00$ | $-672.95\pm 1.86$ |
| Full-MPPI | $-1690.20\pm 101.46$ | $12.48\pm 9.07$ | $1.70\pm 0.10$ | $-1702.67\pm 100.81$ |  |
| Guided-MPPI | $-122.06\pm 5.31$ | $223.25\pm 4.83$ | $0.35\pm 0.01$ | $-345.31\pm 6.66$ |  |
| Residual-MPPI | $\mathbf{-35.31\pm 7.19}$ | $232.82\pm 3.95$ | $\mathbf{0.27\pm 0.01}$ | $\mathbf{-268.14\pm 7.93}$ |  |
| Env. | Policy | $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{z}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |
| Hopper | Prior Policy | $6786.03\pm 39.16$ | $3438.86\pm 13.37$ | $1.33\pm 0.00$ | $3347.17\pm 27.67$ |
| Full-MPPI | $21.02\pm 3.07$ | $3.73\pm 0.81$ | $1.24\pm 0.00$ | $17.29\pm 2.40$ |  |
| Guided-MPPI | $6720.80\pm 543.55$ | $3393.86\pm 234.91$ | $1.34\pm 0.01$ | $3326.95\pm 313.20$ |  |
| Residual-MPPI | $\mathbf{6850.97\pm 321.10}$ | $3411.72\pm 133.23$ | $\mathbf{1.35\pm 0.01}$ | $\mathbf{3439.25\pm 188.67}$ |  |
| Env | Policy | $\mathrm{Total\ Reward}$ | $\mathrm{Basic\ Reward}$ | $\bar{v}_{y}$ | $\mathrm{Add}$-$\mathrm{on\ Reward}$ |
| Ant | Prior Policy | $4153.89\pm 1425.00$ | $4770.24\pm 1595.86$ | $-0.62\pm 0.22$ | $-616.35\pm 224.59$ |
| Full-MPPI | $-2759.45\pm 179.90$ | $-2760.16\pm 114.51$ | $0.00\pm 0.11$ | $0.71\pm 107.86$ |  |
| Guided-MPPI | $2792.82\pm 2158.99$ | $2853.79\pm 2175.37$ | $-0.06\pm 0.13$ | $-60.97\pm 131.36$ |  |
| Residual-MPPI | $\mathbf{5175.55\pm 1459.31}$ | $4847.04\pm 1379.12$ | $\mathbf{0.33\pm 0.13}$ | $\mathbf{328.51\pm 132.60}$ |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I am curious about the performance difference caused by the inclusion of $\log \pi$. How does the trajectory change when the hyperparameter $\omega'$ is increased or decreased?
- When performing policy customization without a given reward, how can the proposed Residual-MPPI be applied if only expert data or instructions are provided instead?
- I am willing to raise the score if the authors respond to the weaknesses and questions.
soundness: 3
strengths: - The paper is well-written and easy to follow.
- The methodology is novel and well-structured.
- The approach is validated in complex environments, including MuJoCo and GTS.
summary: The paper introduces Residual-MPPI, a method for online policy customization in continuous control tasks. Residual-MPPI enables immediate policy adaptation at execution time by utilizing only the action distribution of a pre-trained policy, without needing additional training data or detailed knowledge of the original policy. Experiments demonstrate its effectiveness in adapting the champion-level racing agent GT Sophy 1.0 to custom constraints within the challenging Gran Turismo Sport (GTS) simulator environment, as well as in the robotic MuJoCo environment.
weaknesses: - The paper lacks SOTA baselines for policy customization. For instance, methods like those in [1, 2] demonstrate few-shot adaptability to new environments without an additional parameter training phase. A comparison with such method would strengthen the evaluation.

[1] Xu, Mengdi, et al. "Prompting decision transformer for few-shot policy generalization." *international conference on machine learning*. PMLR, 2022.

[2] Liu, Jinxin, et al. "Ceil: Generalized contextual imitation learning." *Advances in Neural Information Processing Systems* 36 (2023): 75491-75516.

- The performance difference between Residual-MPPI and the comparison baseline, Greedy-MPPI, is not sufficiently analyzed. In Table 1, Residual-MPPI and Greedy-MPPI show very similar performance, yet the paper only briefly mentions on Line 430 that “Greedy-MPPI, as mentioned above, also leads to severe failure in complex GTS tasks” without further explanation. A more detailed analysis of Off-course Steps across different baselines would add clarity to the comparison.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.  In the experiment part, why the customised the policy need to maintain the same level of performance on the basic tasks? I understand that the full task is given by $wr+r_R$ and the add-on task is given by $r_R$. It makes sense to report results of customized polices on these two tasks. But what’s the motivation of comparing the results on task defined by $r$? since it is irrelevant to the customization objective. 
2. The paper formulated the policy customization as an augmented MDP, with the reward to be defined as $w’\log \pi(u|x) + r_R$. I’m wondering, in Table 1, the total reward refers to $ w’\log \pi(u|x) + r_R$ or $wr+r_R$? 
3. Re Theorem 1, I can’t understand why such theorem is needed? And what does it imply?
soundness: 2
strengths: * [Originality] The paper attempts to integrate the residual Q learning into the Model Predictive Path Integral (MPPI) and show some promising results of such attempt. 
* [Quality and clarity] The paper is well written, and the structure is good in general. It is good to see the empirical evaluation has been conducted on multiple different domains and a relatively thorough results are reported. 
* [Significance] The proposed method considers an important problem of adapting a policy to new setting, which can be very useful for RL to be applied to many real-world applications.
summary: The paper studies the problem of online-fining pre-trained policies and proposes to combine the residual q learning and Model Predictive Path Integral (MPPI) together to form a new method, called Residual-MPPI, to address the problem. The paper provides a theoretical analysis of the proposed formulation and a comprehensive empirical evaluation of the method on Mujoco and GTS domains. The results are reported to be promising and the proposed method turns out to be effective.
weaknesses: ### [Empirical evaluation appears to be weak] 
1. The empirical performance of the proposed method, Residual-MPPI, is quite similar to the heuristic modification of MPPI, i.e., Greedy-MPPI. On the main metrics, Full-task and Add-on Task, the reported performance of Residual-MPPI is approximately the same as that of Greedy-MPPI. The paper pointed out one difference on Ant Full Task. But it is unclear how the total reward was computed here and why this total reward should matter more than the add-on reward, given that Greedy-MPPI yields higher rewards than Residual-MPPI on add-on task. So please **provide a more detailed analysis of the differences between Residual-MPPI and Greedy-MPPI, particularly in cases where their performance is similar**. Also please **clarify how the total reward is calculated and why it's considered more important than the add-on reward in certain cases, especially for the Ant Full Task**. 

2. The baseline methods only cover some variants of MPPI. The paper only compared its proposed method with different variants of MPPI. However, as mentioned in the related work, there are many more baseline methods that had been discussed in contrast to the proposed method. So the paper should at least provide convincing argument that why these mentioned methods in related work should not be included as baselines. I think at least RL-driven MPPI, JSRL, and AWAC can be applied to the Mujoco environment to solve the online fine-tuning tasks. So please **include RL-driven MPPI, JSRL, and AWAC (if appropriate) to the baseline methods as they are discussed in the related work as important previous works**, or please **explain why these preivous methods in related work could not be included in the experiment**. 

3. Some metrics adopted in the experiments are debatable. The paper adopted two metrics, add-on reward and basic task reward, as the main measures of the required performance. However, I’m not sure if I understand it correctly: should we always focus on the augmented reward (or the add-on reward) as the sole measure? Since this is the main objective in the policy customization. See my questions in the Question Section. 

4. Results on GT Sophy and sim-to-sim experiments seem to be incomplete. It’s good to see the paper evaluates the proposed method on GT Sophy and sim-to-sim domains. But the results are relatively incomplete (e.g., no quantitative results are reported for Full, Guided, Greedy and Valued variants of MPPI). Given that the Greedy-MPPI was reported to perform similarly to the proposed method, Residual-MPPI, it would be necessary to see how Greedy performs in other domains.  So please **provide quantitative results for the missing MPPI variants in the GT Sophy and sim-to-sim experiments**. 

### [More clarifications and ablations are needed to understand the method]
In section 3.2, the paper introduced three main techniques to improve the learning of the transition dynamics function. However, the paper did not explicitly describe how these techniques are used in the method. At least, it’s not apparent to find them in the algorithm 1 (can’t find the multi-step error used as a loss function). Neither is clear about fine-tuning with online data. So please **provide a more detailed explanation of how the three techniques (multi-step error, exploration noise, and fine-tuning with online data) are integrated into the algorithm, possibly by updating Algorithm 1 or providing additional pseudocode.**

Further, there are no ablation studies on many design choices reported by the paper. Specifically,  it is not reported that how the above three techniques of learning the transition dynamics can affect the empirical performance of Residual-MPPI. Also, comparing Greedy-MPPI against Residual-MPPI, it can be hard to say it is necessary to include $\log\pi$ as a reward in Residual-MPPI since the two perform very similarly on HalfCheetah, Swimmer, and Hopper. 

### Clarity and the rigorousness 
Some sections lacks clarity. Section 3.2: DYNAMICS LEARNING AND FINE-TUNING appear to be disjoint to Section 3.1 since the proposed techniques in Section 3.2 cannot be found in Section 3.1 or the algorithm. Re Section 5.2.2, it is vague as to how the evaluation was conducted and what quantities had been compared against to draw the conclusions there.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: As mentioned in my previous points, how does Residual-MPPI performs in comparison to training a prior policy on the full task (initial task + add-on reward)?



**Summary of the Review**:

The work is well written, the method is clearly presented and well motivated, the evaluation setup is also interesting. I do believe the work could improve in some directions, particularly adding a few more baselines and highlighting some important contributions. Nonetheless, I believe it is already in a good shape for acceptance.
soundness: 3
strengths: - The work is well written and structured, easy to follow. The adopted motivation and proposed method are also clear;

- The evaluation setup is interesting: it brings MuJoCo benchmarks (which, although toy problems, are not trivial for continuous control) and also on GTS (which is a more complex setup for control). Therefore, the presented results are grounded in solid benchmarks.

- The discussion in Section 4.2 is very elucidative, and convincingly justifies the failures of the baselines in comparison to the proposed method. This helps understand the strengths of Residual-MPPI.

- Noticing that it is possible to replace the terminal value estimator for the log probabilities of the policy (into the MaxEnt framework) sounds particularly clever and also justifies well the proposed method.
summary: This work approaches the challenge of policy customization: given a prior policy (trained on a task, represented by a reward function), how should one adapt it at test-time to account for a different objective (add-on reward)? To tackle this problem, the paper introduces Residual-MPPI, which bridges Residual Q-Learning (an RL algorithm for policy customization) and MPPI (a planning method for model predictive control). The work firstly introduces a theoretical motivation for this mix of techniques (Section 3.1) and then provides experiments in continuous control tasks on MuJoCo and the GTS environment. Empirical results demonstrate Residual-MPPI is capable of adapting the policy with fewer samples than prior work, and not considerably compromising performance on the prior task.
weaknesses: While my concerns are not major, there are some suggestions that could improve the clarity of the results in the paper:

- In Section 5.1, the paper argues that the metrics to evaluate on GTS are lap time and off-course steps since the rewards are complex. Nonetheless, I believe the work should provide both the prior task reward and the add-on reward for each method, as in the MuJoCo case, to understand how the method influences the returns that are optimized. Perhaps the considered small changes in lap time are associated with big changes in the returns, which would mean that Residual-MPPI is actually considerably deviating from the prior task designed by the environment.

- Similarly, the work could provide results for Residual SAC in the MuJoCo environments, as the most direct comparison baseline in the literature. Currently, the only results are in GTS. Providing MuJoCo results should also give a better understanding about this direct comparison.

- Another baseline that is important is training the “prior policy” on the full task (prior reward + add-on reward). Since MPPI is just a local search around the prior policy, perhaps it is too regularized towards the prior task. By providing the prior policy on the full task, the work provides a potential upper bound on joint performance, which contextualizes the performance of Residual-MPPI (as a policy customization method) to standard full RL training. This also helps clarify the trade-off of retraining a policy from scratch (expensive, but potentially better asymptotic performance) OR performing policy optimization (cheap, but potentially suboptimal).

- Lastly, the work should probably highlight more the results in terms of sample efficiency. I understand that sample efficiency is key to policy customization, particularly in the physical world. The work just mentions that MPPI leverages 2100 laps while Residual-SAC leverage 80000 laps, which is a big difference. From my understanding, this is a big contribution of the method and should be highlighted. Some suggestions are either bringing a temple of sample analysis to the main paper or at least highlighting these results in the text. 


Typo: Section 5.2.1: “Compration”

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: - How were the MuJoCo add-on tasks chosen?
- How was a dynamics model obtained for Residual-MPPI for these environments?
- Why does Greedy-MPPI only degrade the performance for the Ant environment? It is stated that ignoring the log $\pi$ would degrade performance when the add-on reward is orthogonal to the basic reward in the discussion which makes sense, but why is this only the case for the Ant environment?
- Table 1 states that the evaluation results are over 500 episodes, but does not provide the number of training seeds/independent runs. How many training seeds were used? If only one seed was used, it should be stated that the std is over the evaluation period, not over different training runs, and this should be justified. 
- Why are two decimal places used in Table 1? This would be clearer to the nearest integer (except the undefined last but one column).
soundness: 3
strengths: - **Motivation / Significance**
  - The problem of customizing a policy at execution time without knowledge of the original objective, and while minimising additional online data collection/fine-tuning, is an interesting and important problem.
  - This problem is well-motivated in the paper.
- **Approach / Originality** 
  - The use of MPPI as a model-based approach to customize the policy at execution time seems sensible and original.
- **GTS Application**
  - The GTS results provide a clear use-case for the use of the method.
  - The videos and results appear to show a clear improvement in the policy as a result.
summary: This paper introduces an algorithm, Residual-MPPI, for customizing continuous control policies at execution time, without fine-tuning, in order to perform better at an auxiliary task while maintaining performance on the original task. Residual-MPPI does not require knowledge of the original task reward or value function; only the prior policy. However, the algorithm requires learning a dynamics model if one is not available, requiring additional online data to train and possibly iteratively improve as the policy changes (although apparently less data is required than the primary external baseline for policy customization, residual Q-learning). The approach is benchmarked against other variants of the algorithm on MuJoCo, with code provided, before being evaluated on the motivating use-case in Gran-Turismo Sport (GTS).

In summary, the paper has a strong problem motivation and proposed solution, but the quality and clarity of the paper could be significantly improved. As a result, this paper does not feel ready for publication yet, but I would be happy to increase my rating if these issues were improved before the end of the discussion period.
weaknesses: - **Quality**

  - The quality of the work should ideally be improved before the work is published. The full 10 pages are used, but there are many aspects which are unclear/undefined (see below), while there are large sections of text for which the key point is not clear. For example:
    - The discussion (first paragraph of section 4.2) should be made more concise by breaking it up into the key conclusions. 
    - The abstract could also be made more concise e.g. 'Residual-MPPI can be used to improve new performance metrics at execution time, given access to the prior policy alone.' 
    - The related work could similarly be made more concise by focusing on the key difference with each work.
  - Citations should be in brackets, not highlighted. 
  - The appendix should come after the references in the main paper, not in the supplementary materials.
  - Other small typos and mistakes such as:
    - L416: 'Compration'
    - L361: 'apperant'

- **Clarity / Novelty of Theory**

  - The relevance and novelty of the theory is unclear to me. The approach seems to be a direct application of MPPI to Resdiaul-SAC. What is the intended contribution of Theorem 1 to the paper? It is derived in the Appendix, but the resulting Equation 5 just appears to be a direct result from [1] and it is mentioned that reference [2] already combines MPPI with SAC. Calling it a Theorem seems unnecessary if it is not a new theoretical result. The purpose of this theorem and the logic behind the derivation of Residual-MPPI could be made clearer.
  - 'Few-shot' and 'zero-shot' are mentioned throughout the paper, but not defined until Section 5.1 and it is not clear what these terms refer to until this point. These should be defined in this context before use.

- **Experiments / Significance**

  - The add-on rewards used for the MuJoCo tasks are not stated in the main paper, only the Appendix. Some examples or at least the kind of add-on tasks considered should be included in the main paper to help the reader intuitively understand what kind of customization is being considered, and to define the last but one column of Table 1, which is meaningless otherwise. 

  - There are no external baselines for the MuJoCo experiements. Could residual-SAC be used as a baseline here? It is claimed that these results are zero shot, but Residual-MPPI would need a dynamics model for this to be the case. Where does the dynamics model come from for these experiments? If it is learned from data, then could Residual-SAC be trained on this data? The experimental evaluation would be strengthened by the inclusion of external baselines, or justification as to why these are not possible.

  - Residual-MPPI seems to fall between the baseline GT Sophy and Residual-SAC in terms of both Lap Time and Off-course Steps in Table 2, so Residual-SAC performs more conservatively as claimed. However, the authors mention that Residual-SAC takes much more data (80k laps) compared to Residual-MPPI (~2k laps). What would happen if Residual-SAC was trained on the equivalent smaller amount of data? I wonder if it might customize the policy less and therefore perform less conservatively on the original task, better matching the performance of Residual-MPPI? I believe this would provide a fairer comparison of the two methods so would be helpful to include if possible.

**References:**

[1] *Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor*, Haarnoja et al., 2018

[2] *Information Theoretic Model Predictive Q-Learning*, Bhardwaj et al., 2020

Meta-review: The paper proposes Residual-MPPI, a method for online policy customization in continuous control tasks. The proposed method bridges model predictive path integral (MPPI) control and residual Q-Learning to adapt a pre-trained policy to new objectives at execution time without additional training on the prior task. The experiments conducted in the MuJoCo and Gran Turismo Sport (GTS) environments demonstrate sample-efficient adaptation.

Reasons to accept
- Generally well-written and structured, with a clear problem statement and motivation.
- This work tackles the challenging problem of policy customization at execution time without fine-tuning or detailed knowledge of the original reward function.
- The proposed method combines MPPI and residual Q-Learning in a novel way for policy customization.
- The evaluation is comprehensive, covering diverse benchmarks (MuJoCo and GTS).
- The experimental results demonstrate significantly improved sample efficiency of the proposed method compared to baselines.

Reasons to reject
- Missing ablation studies on many design choices. (mostly addressed by the rebuttal)
- Residual-MPPI performs similarly to Greedy-MPPI in many tasks, but there is insufficient analysis of these results. (addressed by the rebuttal)
- Missing results for a "prior policy" trained on the full task, which could serve as an upper performance bound. (addressed by the rebuttal)
- Terms like "few-shot" and "zero-shot" are introduced late without adequate explanation.
- Details on how dynamics models are trained and used are sparse. (mostly addressed by the rebuttal)

Overall, this paper studies a promising research direction and presents an interesting and effective approach. Many of the concerns and questions raised by the reviewers were sufficiently addressed by the author's rebuttal and in the revised paper. Consequently, I recommend accepting the paper.

Decision: Accept (Poster)