Residual-MPPI: Online Policy Customization for Continuous Control
Authors: Pengcheng Wang
Chenran Li
Catherine Weaver
Kenta Kawamoto
Masayoshi Tomizuka
Chen Tang
Wei Zhan
Venue: ICLR 2025 Poster (2025)
Keywords: Policy customization
Combination of learning- and planning-based approaches
Model predictive control
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a generic online planning algorithm for customizing continuous-control policies on new performance metrics in few-shot and even zero-shot online settings.

Abstract: Policies developed through Reinforcement Learning (RL) and Imitation Learning (IL) have shown great potential in continuous control tasks, but real-world applications often require adapting trained policies to unforeseen requirements. While fine-tuning can address such needs, it typically requires additional data and access to the original training metrics and parameters.
In contrast, an online planning algorithm, if capable of meeting the additional requirements, can eliminate the necessity for extensive training phases and customize the policy without knowledge of the original training scheme or task. In this work, we propose a generic online planning algorithm for customizing continuous-control policies at the execution time, which we call Residual-MPPI. It can customize a given prior policy on new performance metrics in few-shot and even zero-shot online settings, given access to the prior action distribution alone. Through our experiments, we demonstrate that the proposed Residual-MPPI algorithm can accomplish the few-shot/zero-shot online policy customization task effectively, including customizing the champion-level racing agent, Gran Turismo Sophy (GT Sophy) 1.0, in the challenging car racing scenario, Gran Turismo Sport (GTS) environment. Code for MuJoCo experiments is included in the supplementary and will be open-sourced upon acceptance. Demo videos are available on our website: https://sites.google.com/view/residual-mppi.

Published as a conference paper at ICLR 2025
RRSIDUAL-MPPI: ONLINE POLICY CUSTOMIZATION FOR CONTINUOUS CONTROL
Pengcheng Wang"", Chenran Li",, Catherine Weaver Kenta Kawamoto Masayoshi Tomizuka', Chen Tang"t, and Wei Zhan'
Deparrmennt of Mechanical Engineering, University of California, Berkeley. Department of Computer Science, University of Texas at Austin. SSoy Research Inc., Japan.
ABSTRACT
Policies developed through Reinforcement Learning (RL) and Imitation Learning (IL.) have shown great potential in continuous control tasks, but real-world appli- cations often require adapting trained policies unforeseen requirements. While fine-tuning can address such needs, it typically requires additional data and access the original training metries and parameters. In contrast, an online planning algorithm, if capable of meeting the additional requirements, can eliminate the necessity for extensive training phases and customize the policy without knowl- edge of the original training scheme or task. In this work, we propose generic online planning algorithm for customizing continuous-contre policies at the ex- ecution time, which we call Residual-MPPH It customize given prior pol- icy on new performance metrics in few-shot and even zero-shot online settings, given access the prior action distribution alone. Through experiments, we demonstrate that the proposed Residual-MPPI algorithm can accomplish the few- shot/zero-shor online policy customization task effectively, including customizing the champion-level racing agent, Gran Turismo Sophy (GT Sophy) 1.0, in the challenging car racing scenario, Gran Turismo Sport (GTS) environment. Code for MuJoCo experiments is included in the supplementary and will be open- sourced upon acceptance. Demo videos and code are available on our website: www./oww....//e...oo...ooo.o...e www.//iewww/eeeeeeeeeeeee wwwp://lessiaaaaaeeeeeeaaaalllla
/ttps://sitee google. com
INTRODUCTION
Policy learning algorithms such Reinforcement Learning (RL) and Imitation Learning (IL) have been widely employed to synthesize parameterized control policies for a wide range of real-world motion planning and decision-making problems (Tang al. 2024). such as navigation (Mirowski
ct al.]6611] Francis et al. 2020), manipulation (Andrychowicz et al. Zhang et al. 20.8 Man-
dlekar al. 2021 Rajeswaran al. (017) and locomotion (Peng & Van De Panne 201 Gan-
gapurwala et al. 2022). In practice, real-world applications often impose additional requirements on the trained policies beyond those established during training, which can include novel goals (Rhine-
hart et al. 2018)) specific behavior preferences (Ziegler et al. 2019). and stringent safety criteria (Lu al. 2023). Retraining new policy network whenever new additional objective encountered both expensive and inefficient, as it may demand extensive training efforts. To enable flexible de- ployment, it is thereby crucial develop sample-efficies algorithms for synthesizing new policies
that meet additional objectives while preserving the characteristics of the original policy (Lu et al.)
2023 Harmel et al. 2023).
Recently, Li et al. 2024) introduced a new problem setting termed policy customization, which provides a principled approach to address the aforementioned challenge. In policy customization, the objective is to develop new policy given prior policy, ensuring that the new policy: 1) retains the
Equal Contribution
Corresponding Author

---

Published as a conference paper at ICLR 2025
Online Customization
Redidual-MPPI
((NiR))
Pylor Palky
Arfie Sampler
Action Evaluation
Castanine Palkyy
Figure I: Overview of the proposed algorithm. In each planning loop, utilize the prior policy to generate samples and then evaluate them through both the log likelihood of the prior policy and an add-on reward to obtain the customized actions. More details in Sec. [ In the experiments, demonstrate that Residual-MPPI can accomplish the online policy customization task effectively, even in challenging GTS environment with the champion-level racing agent, GT Sophy 1.0.
properties of the prior policy, and 2) fulfills additional requirements specified by given downstream task. As an initial solution, the authors proposed the Residual Q-learning (RQL) framework. For discrete action spaces, RQL can leverage maximum-entroppy Monte Carlo Tree Search (Xiao al., 2019) to customize the policy online, meaning at the inference time without training. In contrast, for continuous action spaces, RQL offers solution based on Soft Actor-Critic (SAC) (Haarnoja ct al. 2018) to train a customized policy leveraging the prior policy before the real execution. While SAC-based RQL is more sample-efficient than training new policy from scratch, the additional training steps that are required may still be expensive and time-consuming.
In this work, we propose online policy customization for continuous action spaces that eliminates the need for additional policy training. We leverage Model Predictive Path Integral (MPPI) (Williams) al. 2017). a sampling sameling-based model predictive control (MPC) algorithm. Specifically, we propose Residual-MPPL, which integrates RQL into the MPPI framework, resulting in an online planning algorithm that customizes the prior policy at the execution time. The policy performance can fur- ther enhanced by iteratively collecting data online to update the dynamics model. Our experiments in MuJoCo demonstrate that our method can effectively achieve zero-shot policy customization with provided offline trained dynamics model. Furthermore. to investigate the scalability of our algo- rithm in complex environments, we evaluate Residual-MPPI in the challenging racing game, Gran Turismo Sport (GTS), in which successfully customize the driving strategy of the champion-level racing agent, GT Sophy 1.0 (Wurman et al., (022), to adhere to additional constraints.
PRELIMINARIES
In this section, we provide a concise introduction to two techniques that are used in our proposed algorithm: RQL and MPPI, to establish the foundations for the main technical results.
2.1 POLICY CUSTOMIZATION AND RESIDUAL Q-LEARNING
We consider discrete-time MDP problem defined by tuple M (X, where X R is continuous state space, U4 C Rm is a continuous action space, X' x 2 - IR is the reward function, and p: X' x x X' > [0, oc) is the state transition probability density function. The prior policy π X + U is trained as optimal maximum-entropp policy to solve this MDP problem.
Meanwhile, the add-on task specified by an add-on reward function TR X x U - R. The full task becomes a new MDP defined by a tuple M = (X, + +RR.P). p), where the reward is defined as weighted sum of the basic reward and the add-on reward FR. The policy customization task is to find a policy that solves this new MDP. Li et al. (024) proposed the residual Q-learning framework to solve the policy customization task. Given the prior policy æ. RQL is able find this customized policy without knowledge of the prior reward r, which ensures broader applicability across different prior policies obtained through various methods, including those solely imitating demonstrations.
2

---

Published as a conference paper at ICLR 2025
In particular, shown in their appendix (Li et al. 2024). when we have access to the prior policy æ, finding the maximum-entropy policy solving the full MDP problem M (X U, wr rn, p) is equivalent to solving an augmented MDP problem M = (X,4, w/ log π T u2 æ + rn.P), where w' is hyper-parameter that balances the optimality between original and add-on tasks.
2.2 MODEL PREDICTIVE PATH INTEGRAL
Model Predictive Path Integral (MPPI) (Willims et al. 2017) is sampling-based model predictive control (MPC) algorithm, which approximates the optimal solution of an Lininite-horizon, MDP through receding-horizon planning. MPPI evaluates the control inputs U (uo, W1.-.. , .T--) -) with an accumulated reward function Sao (U) defined by a reward function r and a terminal value estimator db
T-1 Sw, (U) - Σ r(æ,, u.) + #(xT),
(1)
where the intermediate state 2; is calculated by recursively applying the transition dynamics on æo-
By applying an addictive noise sequence (Eo, El...., ,.....,,ETTII- ET nominal control input sequence
Ü. MPPI obtains disturbed control input sequence as U Ü - & for subsequent optimization, where & follows a multivariate Gaussian distribution with its probability density function defined as
p(E) = - ((2x) exp (- 1 x=-Ee). where m the dimension of the action space.
As shown by Williams et al. (2017), the optimal action distribution that solves the MDP is
1 q- (U) = exp (~5e0 P(E),
(2)
7
where 17 is the normalizing constant and À is a positive scalar variable. MPPI approximates this distribution by assigning importance sampling weight w(E)) to each noise sequence E to update the nominal control input sequence:
K
u. = úr +
Σ
we
(3)
k=1
where K is the number of samples, and w(E)) could be calculated as
w(E ( (0 + 8 - Σ af'=''c (û, + 2c()
X T-1
where μ is the normalizing constant.
METHOD
Our objective is to address the policy customization challenge under the online setting for continuous control. We aim to leverage a pre-trained prior policy æ with a dynamics model F, to approximate the optimal solution to the augmented MDP problem Maug in online manner. To address this problem, we propose a novel algorithm, Residual Model Predictive Path Integral (Residual-MPPP) which broadly applicable any maximum-entropy prior policy with a dynamics model. The proposed algorithm is summarized in Algorithm [ and Figure [] In this section, we first establish the theoretical foundation of our approach by verifying the We then refine the MPPI method with the derived formulation to approximate the solution of the Maug Lastly, we discuss the dynamics learning and fine-tuning method used implementation.
-entrop property of MPPI.
RESIDUAL-MPPI
MPPI is widely used campling-basedd MPC algorithm that has demonstrated promising results in various continuous control tasks. To achieve online policy customization, i.c., solving the augmented MDP MMss efficiently during online execution, utilize MPPI as the foundation of our algorithm.
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 Residual-MPPI
Input: Current state zu; Output: Action Sequence Ü (ūo, ū1,".. ur -1)-
Require: System dynamics F; Number of samples K; Plamning horizon T:: Prior policy क, Distur-
bance covariance matrix 2; Add-on reward TR: Temperature scalar X; Discounted factor ? Initialize the action sequence from the prior policy
do 2: it f arg max T (xx maxT(aaAa 3: J1+1 F(x,, úi))
for = ...,....
4: end for
5: for l....: K do
D Evaluate the sampled action sequence
6: Sample noise 8
{ c/.
+
7: for t O..... T do
8:
B++1 F F ú +
9:
S(E)) 7t (FR(0r, ún + tt) + w log (ûg + Auo E-
10: end for
11: end for
12: 3 = ming S(E
Update the action sequence
n + Σ exp
(S (E*) - -3)
14: for k 1 K do
15: w(E)) 1 exp (1 (S (E*) - B))
16: end for
17: for 1.- = 0 ..... T - do
18: ú,+ k- w(E))e
K
19: end for
return U
As shown in Sec. 2.1 RQL requires the planning algorithm, MPPI, to comply with the principle of maximum entropy. Note that this has been shown in [Bhardwaj al. (2020), where this result established the foundation for their Q-learning algorithm integrating MPPI and model-free soft Q- learning. In Residual-MPPI, this result serves as preliminary step to ensure that MPPI can be employed as an online planner to solve the augmented MDP M in policy customization. To better serve our purpose, provide a self-contained and concise notation of this observation in Proposition ! Proposition Given an MDP defined by M (X,U,r. with a deterministic state transition defined with respect to dynamics model F' and discount factor ? = 1, the distribution of the action sequence qº (U) at state By horizon T, where each action u.,t 0,-.. ,T sequentially sampled from the maximum-entie entropp policy (Haarnoja et al. 2017 with an entropy weight a
Its step-by-step breakdown proof can be found in Appendix A
(
T-1 r(( u.) V* V*(xT)
1
a Σ
Q'(U)
exp
(5)
Zao
where V* is the soft value function (Haarnoja et al. 2017) and xs is defined recursively from wo and U through the dynamics model F I++1 F(x,, uz), ..... . 1. 1.
we have X = a and let V be the terminal value estimator, the distribution equation 5 equivalent to the one in equation _ that is the optimal distribution that MPPI tries to approximate, Gaussian distribution with infinite variance, i.c. a uni-
but under the condition that the p(E) form distribution. It suggests that MPPI can well approximate the maximum-entroo optimal policy with a discount factor 7 close to and a large noise variance. We can then derive Residual-MPPI straightforwardly by defining the evaluation function Sæ, (U) in MPPI as
T-1
SauR (U)
Σ
(rn(#,, u.) + w/ log T (u)))))))
(6)
to solve the M therefore approximates the optimal customized policy online.
The performance and sample efficiency of MPPI depends on the approach to initialize the nominal input sequence Ú. In policy customization, the prior policy serves as a natural source for initial- izing the nominal control inputs. As shown line T in Algorithm T by recursively applying the
4

---

Published as a conference paper at ICLR 2025
prior policy, we could initialize nominal trajectory with tunable exploration noise to construct Gaussian prior distribution for sampling. During experiments, we found that including the nominal action sequence as a candidate evaluation sequence effectively increase sampling stability.
3.2 DYNAMICS LEARNING AND FINE-TUNING
In scenarios where an effective dynamics model is unavailable in a prior, it is necessary to develop a learned dynamics model. To this end, we established dynamics training pipeline utilizing the prior policy for data collection. In our implementation, we employed three main techniques to enhance the model's capacity for accurately predicting environmental states, as follows. The pseudocode of the complete Residual-MPPI deployment pipeline can be found in Algorithm 2 in Appendix [
The prediction errors of the imperfect dynamics model accumulate over time steps. In the worst case, compounding multi-step errors can grow exponentially (Venkatraman 2013). To ensure the accuracy of the dynamics over the long term, we use multi-step emror Ei=07"(s- 61)2 as the loss for training, where 8, and & are the ground-truth and prediction.
Multi-step Error:
The prior policy's behavior sticks around the optimal behavior to the prior task, which means the roll-out samples collected using the prior policy concentrated to limited region in the state space. It limits the generalization performance of the dynamics model under the policy customization setting. Therefore, during the sampling process, we add Gaussian exploration noises to the prior policy actions enhance sample diversity.
Exploration Noise:
Since the residual-MPPI planner solves the customized task objec- tive, its behavior is different from the prior. Thus, the planner may encounter states not contained in the dynamics training dataset collected by prior policy, on which the learned dynamics could be inaccurate. In this case, we can iteratively collect data with Residual-MPPI and update the dynam- ies model using the in-distribution data. We refer to the variant with online dynamics fine-tuning as few-shot Residual-MPPI and the variant without fine-tuning as zero-shot Residual-MPPI
Fine-tuning with Online Data:
4 MUJoCo EXPERIMENTS
In this section, we evaluate the performance of the proposed algorithms in different environments selected from MuJoCo (Todrovv al. 2012). In Sec. [.1] we provide the configurations of our ex- periments, including the settings of policy customization tasks in different environments, baselines, and evaluation metrics. In Sec. Se2. we present and analyze the experimental results. Please refer to the appendices for detailed experiment configurations, implementations, and visualizations.
4.1 EXPERIMENT SETUP
Environments. In each environment, design the add-on rewards to illustrate customized spec- ifications such as behavior preference or additional task objectives in the practical application sce- narios. In HalfCheetah and Swimmer, we apply an add-on penalty on the angle of certain joint, which is common issue in deployment if the corresponding motor is broken; In Hopper, apply an extra reward on height; In Ant, we apply an add-on reward for moving along the y-axis. The configurations of environments and training parameters can be found in Appendix
Baselines. In our experiments, we compare the performance of the proposed residual-MPPI algo- rithm against seven baselines, including the prior policy, four alternative MPPI variants, and two RL-based baselines. Note that except for Greedy-MPPI, the remaining MPPI baselines have access to the underlying reward value function of the prior policy. These baselines show that Residual- MPPI is still the ideal choice, even with privileged access to additional reward or value information.
We utilize SAC to train the prior policy the basic task for policy customization. At test time, we evaluate its performance on the overall task without customization, which serves as a baseline to show the effectiveness of policy customization.
Prior Policy:
Greedy- Firstly, we introduce Greedy-MPPI, which samples the action sequences from the prior policy but only optimizes the control actions for the add-on reward rg. i.e., removing the log π reward term of the proposed Residual-MPPP Through comparison with Greedy-MPPi, aim to show the necessity and effect of including log reward in Residual-MPPi.
5

---

Published as a conference paper at ICLR 2025
Full-MPPI: Next, we apply the MPPI with no prior the MDP of the full task (i.e., wTx) We aim to compare Residual-MPPI against it to validate that our proposed algorithm can effectively leverage the prior policy to boost online planning.
GuudeddMPPI: Furthermore, we introduce Guided-MPPI, which samples the control actions from the prior policy and also has privileged access to the full reward information, i.c., WT+ By comparing it against Guided-MPPI, we aim to show the advantages and the necessity of Residual- MPPI even with granted access to the prior reward.
Valued-MPPI: While it is not compatible with problem setting, we further consider the case where the value function of the prior policy is accessible. We construct a variant of Guided-MPPI with this value function a terminal value estimator like Argenson & Dulac-Amold (2020).
Residual-SAC: We also include the Residual-SAC
RL-based baseline to set an upper bound performance of the policy customization task without knowing the prior reward. We used the final checkpoint (4M steps) as well checkpoints with an equivalent amount of data to dynamics training (200K steps) to illustrate the superior sample efficiency of desidual-MPPI.
Fulltask-SAC: Finally, we report the SAC policy trained on the full task as a reference to set an upper bound for the policy customization task's performance.
Metric. We aim to validate a policy's "siicy'spean performance with total reward, i.e., -T When the add-on task contradicts the basic task, the optimal policy may trade off its performance on the basic task to maximize the total reward. Therefore, we further measure the basic and add-on rewards separately to monitor the trade-off achieved by the customized policy, i.e., how well the performance on the basic task is maintained and how much the policy is optimized toward the add-on task.
4.2 RESULTS AND DISCUSSIONS
The experimental results, summarized in Table | demonstrate the effectiveness of Residual-MPPI for online policy customization. The ablation results of the planning parameters in MuJoCo and visualization can be found in Appendix F22 and Appendix E.. Also, we conduct experiments upon the same MuJoCo environments with IL prior policies, whose results are summarized in Table IO] in Appendix F.I Across the tested environments, the customized policies achieve significant im- provements over the prior policies in meeting the objectives of the add-on tasks while maintaining a similar level of performance in terms of the basic task objectives. aln particular, we observe three key advantages of Residual- over baseline approaches.
First, Residual-MFP MMPPI demonstrates significantly higher sample-effciincy over the RL-based base- lines. While Fulltask-SAC and Residual-SAC achieve better performance than planning- based ap- proaches through extensive online exploration (4M steps), our focus on online policy customiza- tion. Thus, we consider their performance upper bounds in our online setting. With a sub- stantially smaller dataset for dynamics model training (2K steps), Residual-MPPI achieves total rewards comparable to Residual-SAC in the HalfCheetah, Hopper, and Ant environments. In con- trast, Residual-SAAt performs significantly worse when trained on the same limited amount of data. Second, Residual-MPPI can strike better trade-off' between the basic and add-on rewards than Greedy-MPPI. Greedy-MPPI optimizes the objective of a biased MDP Madd = (X, U, FR, p). relies solely on sampling from the prior policy to heuristically regularize the optimized action se- quences to remain close to the prior policy. While this heuristic approach helps maintain perfor- mance on the basic task objective, it is prone suboptimality in tasks where the add-on reward is sparse or orthogonal to the basic reward. For example, in the Ant environment, the agent is rewarded for making progress along the x-axis in the basic task, while the add-on reward encourages progress along the y-axis. Since these reward terms are orthogonal, exclusively optimizing for progress along the y-axis compromises the progress along the x-axis, leading to a noticeable performance gap for MPPI. This limitation is further amplified in the more complex racing task, will demonstrate in the next section.
Finally, Residual- outperforms MPPI variants that have privileged access to the prior policy's reward or value functions. Despite having full reward knowledge. Full-MPPI fails to complete the task. Without sampling control actions from the prior policy, Full-MPPI suffers from poor sample efficiency and achieves subpar performance. Guided-MPPI can achieve better performance, given its access to action samples from the prior policy. However, it still underperforms Residual-MPPI

---

Published as a conference paper at ICLR 2025
Table 1: Experimental Results of Zero-shot Residual-MPPI in MuJoCo
Full Task
Basic Task
Add-on Task
Env.
Policy
Total Reward
Basic Reward
[0]
Add-on Reward
Prior Policy
1000.7 88.8
2449.8 52.3
0.14 0.00
1449.1 45.3
Greedy-MPPP
1939.9 9 + 134.7
2180.9 87.3
0.02 0.01
-241. 50.3
Full-MPPI
-3595.1 - 322.7
-1167.3 + 144.0
0.24 0.03
-2427.7
320.3
Half Cheetah
Guided-MPPI
1849.6 + 151.0
215416 95.7
0.03 - 0.01
-305.0 58.7
Valued-MPPI
1760.7 478.8
2201.8
258.3
0.04
-441.0 222.5
Residual-MPPI
1936.2 + 109.3
2178.6 71.9
0.02 0.00
-242.3 40.5
Residual-SAC (200K)
263.0 = 919.0
455.4 + 678.6
0.07 = 0.03
720.4 251.8
Residual-SAC (4M)
2184.5 29.7
2233.7 29.3
0.00 = 0.00
19.2 + 1.7
Fulltask-SAC
2149.9 + 28.6
2214.5 27.2
0.01 = 0.00
64.5 + 2,4
Env
Policy
Total Reward
Basic Reward
[0]
Add-on Reward
Prior Policy
245.2 5.6
345.8 3.2
0.59 + 0.01
-591.0 + 5.8
Greedy-MPPI
55.9 +:
275.8
0.33 - 0.01
333.77 7.4
Full-MPPI
1686.6 + 106.7
14.1 6.3
1.70 0.11
-1700.7 106.2
Guided-MPPI
149.0 5.6
292.9 = 3.8
0.44 0.01
-441.9 7.2
Swimmer
Valued-MPPI
205.8 6.3
335.1
1.04
-540.9 6.3
Residual-MPPH
-60.0 5.2
275.8 3.4
4.44
-335.9 7.6
Residual-SASe SAC
209.0 67.6
2.1 15.5
0.21 = 0.07
221.1 72.7
Residual-SAC (4M)
10.5
-1.5 16.9
0.01 = 0.02
-9.0 16.6
Fulltask-SAC
-4.2+17.1
2.1 2.1+17. 17.6
0.01 0.00
-6.3+3.0 3.0
Env.
Policy
Total Reward
Basic Reward
5
Add-on Reward
Prior Policy
7252.7 + 49.2
3574.5 + 9.7
1.37 = 0.00
3678.2 = 48.3
Greedy-MPPP
7367.0 + 199.4
3553.0 58.4
1.38 + 0.01
3814.0 156.8
Full-MPPI
20.5 3.0
3.6 0.7
1,24 0.00
16.9 2.4
Guided-MPPI
6121.3 + 1590.1
3067.8 679.0
1.35
3053.4 917.7
Hopper
Valued-MPPI
T243.9 + 75.7
3562.7 14.5
1.01
3681.2 = 71.6
Residual-MPPI
7363.0 +: 254.9
3547.6 78.0
1.38 0.01
3815.4 186.
Residual-SAC (200K)
3543.1 478.9
1019.8 94.3
1,27 0.01
2523.2 405.5
Residual-SAC (4M)
7682.5 178.2
2310.4 106.8
1.54 0.01
5372.0 75.8
Fulltask- SAC
7825.3
2934.5 27.6
1.49 + 0.00
4890.8
Env
Policy
Total Reward
Basic Reward
Add-on Reward
Prior Policy
6333.7 + 753.9
6177.1 703.7
0.16 + 0.22
156.6 + 200.5
Greedy-MPPI
6104.2 + 1532.0 0
5092.8 + 1305.2
1.01 + 0.27
1011.3 1 277.7
Full-MPPI
-2767.7 154.0
-2764.1 114.2
-0.00 0.11
-3.3 + 108.0
Guided-MPPI
5160.9 + 1963.0
4999.8 + 1887.9
0.16 - 0.22
161.2 + 217.7
Ant
Valued-MPPI
6437.0 + 1021.9
6230.7 + 959.0
0.21 + 0.20
206.3 + 196.3
Residual-MPPI
6846.7 + 647.8
5984.8 + 541.5
0.86 0.19
861.8 +: 189.8
Residual-SAA SAC (200K)
-1175.5 + 157.3
-1178.3 156.4
0.00 0.00
2.7 3.9
Residual-SAC (4M)
6962.9 + 342.9
5710.2 252.0
1,25 1.25
1252.7 127.3
Fulltask SAC
7408.6 + 312.0
3100.3 184.4
4.31 0.21
4308.3 209.2
The evaluation results are in the form of mean d: std over the 500 running episodes. The total reward is calculated on full lask, whose reward is wr Tg-
across all tasks, as constrained by its limited ability to reason the long-term effects of actions within its finite planning horizons. Conversely, Residual-MPP1 implicitly incorporates the original task reward through the prior policy log-likelihood The log æ term also serves as a proxy the prior policy's Q-function, informing the planner of the impact of the action sequences beyond the planning horizon. While Valued-MPPI addresses long-term impacts by incorporating the prior Q-function, struggles to fully align with the complete task reward. Valued-MPPi considers the long-tern impact by incorporating the prior Q-function but fails to match the full task reward setting, resulting in better performance the basic task but suboptimum on the full task.

---

Published as a conference paper at ICLR 2025
Case
Clove 2
GT Sophy 1.0
Zero-shot MPPI
Few-shot MPP1
Residull-SCC
Figure 2: In-game screen shots of Policy Behavior on Different Road Sections
CUSTOMIZING CHAMPION-LEVEL AUTONOMOUS RACING POLICY
With the effective results in standard benchmark environments, are further interested in whether the proposed algorithm can be applied to effectively customize an advanced policy for a more so- phisticated continuous control task. Gran Turismo Sport (GTS), realistic racing simulator that sim- ulates high-fidelityy racing dynamics, complex environments at an independent, fixed frequency with realistic latency as real-world control systems, serves as an ideal test bed. GT Sophy 1.0 (Wur- man et al., 2022). DRL-based racing policy, has been shown to outperform the world's best drivers among the global GTS player community. To further investigate the scalability of our algorithm and its robustness when dealing with complex environments and advanced policies, carried out further experiments on GT Sophy 1.0 in the GTS environment.
5.1 EXPERIMENT SETUP
Though GT Sophy 1.0 is highly advanced system with superhuman racing skills, it tends to hibit too aggressive route selection as the well-trained policy has the ability to keep stable on simulated grass. However, in real-world racing, such behaviors would be considered dangerous and fragile because of the varying physical properties of real grass surfaces and tires. Therefore, we establish the task as customizing the policy to a safer route selection. Formally, the customization goal is to help the GT Sophy 1.0 drive on course while maintaining its superior racing performance. This kind of customization objective could potentially be used to foster robust sim-to-real transfer of agile control policies for many related problem domains.
We adopt simple MLP architecture design dynamics model and train it using the techniques introduced in Sec. 3.2 The configurations of the environments and implementation details can be found in Appendix D As the pre-trained GT Sophy 1.0 policy constructed with complex rewards and regulations, we adopt the average lap time and number of off-course steps as the metrics. In addition to zero-shot Residual-MPPI, we also evaluate few-shot version of our algorithm, which we iteratively update the dynamics with the customized planner's trajectories to further improve the performance. We also consider Residual-SAC. (Li et al.) (024) as another baseline to validate the advantage of online Residual-MPPI MPPI against RL-based solutions in the challenging racing problem. Notably, we are only given access to the policy network GT Sophy 1.0, which prevents us from evaluating Valued-MPP Greedy-MPPI and Guided-MPPI as MPPI variants for comparison in the GTS environment.
it requires access to the critic function. Therefore, we only include
5.2 RESULTS AND DISCUSSIONS
The experimental results, summarized in Table demonstrate that Residual-MPPI significantly en- hances the safety of GT Sophy 1.0 by reducing its off-course steps, albeit with a marginal increase in lap time. Further improvements process to fine-tune the dynamics under a few-shot setting. This few-shot version Residual-MPPI outperforms the zero-shot version in terms of lap time and off-course steps.
observed after employing data gathered during the customization
The ablation results of the planning parameters and visualization GTS can be found in Ap- pendix F33 and Appendix E.2 which clearly demonstrates the effectiveness of the proposed method by greatly reducing the off-course steps. the detailed route selection visualization. it can be ob- served that Few-shot MPPI chose safer and faster racing line compared Zero-shot MPPI with
8

---

Published as a conference paper at ICLR 2025
Table 2: Experimental Results of Residual-MPPI in GTS
Policy
GT Sophy 1.0
Zero-shot MPPI
Few-shot MPPI
Residual-SAC (80K laps)
Lap Time
117.77 + 0.08
083.3410.2
122.93 + 0.14
130.00 + 0.13
Off-course Steps
93.13 + 1.98
9.03 3.33
4.43 + 2.39
0.87 = 0.78
Policy
Full-MPPI
Guided-MPPI
Greedy-MPPI Residual-SAC (2K laps)
Lap Time
*Failed "Failed
*Failed
*Failed *Failed
*Failed
Off-course Steps
*Failed
*Failed
The evaluation results are in the form of mean + std over 30 laps. "Failed baseline is not able to finish
complete lap. Malued-MPPI is not available since we only have access the policy network of GT Sophy 1.0.
Gi-tephy shot Hit)
(a) Grecdy-MPPI
(b) Guidod-MPPI
Figure 3: Guided-MPPI and Greedy-MPPI Results in GTS. (a) In-game screenshots of Greedy-
MPPI; (b) In-game screenshots of Guided-MPPI. Red parts indicate off-course behaviors. Both baselines cannot drive the vehicle effectively, completely going off track at the first corner.
more accurate dynamics. Though the customized policies can not eliminate all the off-course steps, it is worth noting that these violations are minor (i.e., slightly touching the course boundaries) com- pared to GT Sophy 1.0, making the customized policies safe enough to stay on course.
Residual-SAC, compared with the Residual-MPPI, yields very conservative customized policy. As shown in Figure 2 it is obvious that the Residual-SAC policy behaves sub-optimally and overly yields to the on-course constraints. It is worth noting that over 80,000 laps of roll-outs are collected to achieve the current performance of Residual-SAC. In contrast, the data used to train GTS dynam- ics for Residual-MPPI, along with the online data for dynamics model fine-tuning, amounted to only approximately 2, 2,000 and 100 laps, respectively. As discussed in safe RL and curriculum learning methods (Mo et al.] 2021 Anzalone et al.) 2022), when training with constraints, it is easy for RL to yield to overly conservative behavior. At the checkpoint with 2k laps of training data, Residual- SAC could not finish a lap yet, which highlights the outstanding data-efficiency of Residual-MPPL
Cuided-MPPH however, cannot finish the track stably, as shown in figure 3 Similar to what we have analyzed in the MuJoCo experiments, Guided-MPPI suffers from its limited ability to account for the long-term impacts of actions given the finite planning horizon. This limitation leads more severe consequences in complex tasks that require long-term, high-level decision decision-making, such as route selection during racing, which leads failure in GTS.
GreI as mentioned above, also leads to severe failure in the complex GTS task, further emphasizing the importance of the log π term in Residual-MPPI's objective function. In GTS, the Greedy-MPPI only optimizes the trajectories by not allowing the policy to be off-course, where a straightforward local optimal but undesirable solution would be just staying still. Such a trivial policy would behave better on the add-on tasks (stay on course) but not on the full task (stay on course while maintaining racing superiority). From the theoretical perspective, the significance of the log term goes far beyond simple regularization term. It is the key factor in addressing the policy customization problem, which is inherently a part of the joint optimization objective, encoding and passing the information of the original reward in theoretically sound manner.

---

Published as a conference paper at ICLR 2025
6 RELATED WORKS
Model-based RL. Many works focused on combining learning-based agents with classical planning enhance performance, especially in various model-based RL approaches. MuZero (Schrittwieser et al., (020)) employs a learned Actor as the search policy during the MCTS process and utilizes the eritic as the terminal value at the leaf nodes; TD-MPC2 (Hansen et al. 2023) follows a similar approach and extends it to continuous task by utilizing MPPI as the continuous planner. RL-driven MPPI (Qu et al. 2024) chooses the distributional SACs as the prior policy and also adopts the MPPI as the planner. However, those methods mainly address the planning within the same task and have a full reward and value function, while policy customization requires jointly optimizing an add-on reward and the underlying reward of the prior policy, which is unknown in a priori. In general cases, the prior policy may not provide a critic, as is the case with algorithms like policy gradient (Shi et al. 2019). Moreover, the value function will change with the new reward, making it infeasible to be applied. Therefore, these methods are not able to solve policy customization tasks directly.
Policy Adaptation. There have been numerous works considering on the topic of policy adaptation. Some works require an extra demonstration dataset to adapt the policy to the new task, like Ceil (Liu al., 2023) and Prompt-DT (Xu al. 2022). However, such expert demonstrations are unavailable in the online policy customization settings. Some works focus on adaptation via RL fine-tuning: ISRL (Uchendu et al. 2023) uses a prior policy as the guiding policy to form a curriculum for the exploration policy; AWAC (Nair et al.] (020) combines sample-efficiee
dynamic programming with maximum likelihood policy updates to leverage large amounts of offline data for quickly performing online fine-tuning of RL policies. However, the main objective in RL fine-tuning is to maximize the policy return on the new task. contrast, policy customization (Li et al. 2024) aims to find a policy that can maintain the prior performance on the basic task at the same time.
Planning with Prior Policy. Some works address a similar problem of customizing pre-trained prior policies towards additional requirements. Efficient game-theoretic planning (Li et al., 2022) uses planner to construct human behavior within the pre-trained prediction model. Deep Imitative Model (Rhinehart et al.] 2018) aims direct an imitative policy to arbitrary goals during online usage. However, they require full reward modeling with hand-designed features, or the add-on task in the form of specific goals. Residual-MCTS proposed in RQL (Li et al.. 2024) solves the policy customization online via planning but is limited to tasks with discrete action spaces. MBOP (Argen-) son & Dulac-Arnole 2020) solves the offline RL problem through planning by learning dynamics, action prior, and values from an offline dataset. LOOP (Sikchi et al. (022) focused on learning off- off-policy with samples from an online planner. However, both approaches require knowledge of the basic reward function and thus cannot be directly applied to our setting.
7 LIMITATIONS AND FUTURE WORK
In this work, we propose Residual-MPPL, which integrates RQL into the MPPI framework, resulting in an online planning algorithm that customizes the prior policy at the execution time. By conducting experiments in MuloCo and GTS, we show the effectiveness of our methods against the baselines. Here, we discuss some limitations of the current method, inspiring several future research directions.
Terminal Value Estimation. Similar to most online planning methods, our algorithm also needs a sufficiently long horizon to reduce the error brought by the absence of terminal value estima- tor. Though utilizing prior policy partially addresses the problem discussed Sec. 4.2 it can not eliminate the error entirely. Techniques like JSRL (Uchendu et al. 2023) and RL-driven MPPI (Qu) jet al., 2024) could be incorporated in the proposed framework in the few-shot setting by using the collected data to learn Residual Q-function online as a terminal value estimator.
Sim2Real Transferability. Sim2real will still face several additional challenges, which mainly comes from the suboptimal prior policy and the domain gap. Prior policies without careful modeling or sufficient training could mislead the evaluation step of the proposed algorithm and result in poor planning, Also, the proposed algorithm relies on an accurate dynamics model to correctly roll out. In the future, we can introduce more advanced methods in policies and dynamics training, such as diffusion policies (Chi et al.) 2023) and world models (Micheli et al. 2022), to improve the prior policies and dynamics. We look forward to further extending residual-MPP to achieve sim-to-sim and sim-to-real for challenging agile, continuous control tasks by addressing these limitations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
The authors thank Ce Hao for valuable discussions and kind help. This work was supported by Sony AI, and Polyphony Digital Inc., which provided the Gran Turismo Sport framework.
REFERENCES
OpenAl:
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert. Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Luca Anzalone, Paola Barra, Silvio Barra, Aniello Castiglione, and Michele Nappi. An end-to-end curriculum learning approach for autonomous driving scenarios. IEEE Transactions on Intelligent Transportation Systems, 23(10):18877-19228 2022.
Arthur Argenson and Gabriel Dulac-Amold. 1xXiv:0008.055566 2020.
Model-based offline planning.
arXiv preprint
Mohak Bhardwaj, Ankur Handa, Dieter Fox, and Byron Boots. Information theoretic model predic- tive q-learning. In Learning for Dynamics and Control, pp. 840-850. PMLR, 2020.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04133 2023.
Anthony Francis, Aleksandra Faust, Hao-Tien Lewis Chiang, Jasmine Hsu, Chase Kew, Marek Fiser, and Tsang- Wei Edward Lee. Long-range indoor navigation with prm-rl. IEEE Transactions on Robotics, 36(4):1115-1133 2020.
Siddhant Gangapurwala, Mathieu Geisert, Romeo Orsolino, Maurice Fallon, and Ioannis Havoutis. Rloc: Terrain-aware legged locomotion using reinforcement learning and optimal control. IEEE Transactions on Robotics, 3865):2908-2927, 2022.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. PMLR, 2018.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- control. arXiv preprint arXiv:2310.16828, 2023.
Moritz Harmel, Anubhav Paras, Andreas Pasternak, and Gary Linscott. Scaling is all you need: Training strong policies for autonomous driving with jax-accelerated reinforcement learning. arXiv preprint 3rXiv:2312.15122 2023.
Chenran Li. Tu Trinh, Letian Wang, Changliu Liu, Masayoshi Tomizuka, and Wei Zhan. Effi- cient game-theoretic planning with prediction heuristic for socially-compliant autonomous driv- ing. IEEE Robotics and Automation Letters, 7(4):10248-10253 2022.
Chenran Li. Chen Tang, Haruki Nishimura, Jean Mercat, Masayoshi Tomizuka, and Wei Zhan. Residual q-learning Offline and online policy customization without value. Advances in Neural Information Processing Systems, 36, 2024.
Jinxin Liu, Li He, Yachen Kang, Zifeng Zhuang, Donglin Wang, and Huazhe Xu. Ceil: Generalized contextual imitation learning. Advances in Neural Information Processing Systems, 36:75491- 75516, 2023.
11

---

Published as a conference paper at ICLR 2025
Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, et al. Imitation is not enough: Robustify- ing imitation with reinforcement learning for challenging driving scenarios. In 2023 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 7553-7560. IEEE, 2023.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline 2021.
human demonstrations for robot manipulation. arXiv preprini
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world mod- els. arXiv preprint arXi::2209.0053: 2022.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.03673 2016.
Shuojie Mo, Xiaofei Pei, and Chaoxian Wu. Safe reinforcement learning for autonomous vehicle using monte carlo tree search. IEEE Transactions on Intelligent Transportation Systems, 23(7): 6766-6773, 2021.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.0953 2020.
Xue Bin Peng and Michiel Van De Panne. Learning locomotion skills using deeprl: Does the choice of action space matter? In Proceedings of the ACM SIGGRAPII/Eamgrrahhicc Symposium Computer Animation, pp. 1-13, 2017a.
Xue Bin Peng and Michiel Van De Panne. Learning locomotion skills using deeprl: Does the choice of action space matter? Proceedings of the ACM SIGGRAPH/Eurographis Symposiun on Computer Animation, pp. 1 1-13, 2017b.
Yue Qu, Hongqing Chu, Shuhua Gao, Jun Guan, Haoqi Yan, Liming Xiao, Shengbo Eben Li, and Jingliang Duan. RI-driven mppi: Accelerating online control laws calculation with offline pol- icy. IEEE Transactions on Intelligent Vehicles, 9(2):300533166 2024. doi: 10.1 109/T//...2. 3348134.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines3; Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 2017.
Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep imitative models for flexible inference, planning, and control. arXiv preprint arXiv:1810.06544 2018.
Julian Schritwwesser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-669, 2020.
Wenjie Shi, Shiji Song, and Cheng Wu. Soft policy gradient method for maximum entropy deep reinforcement learning. arXiv preprint arXiv:1909.0311 2019.
Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off-policy with online planning. In Conference Robot Learning, pp. 1622-1633, PMLR, 2022.
Tom Silver, Kelsey Allen, Josh Tenenbaum, and Leslie Kaelbling. Residual policy learning. arXiv preprint arXiv:1812.06298, 2018
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Mant and Peter Stone. Deep reinforcement learning for robotics: A survey of real- world successes. arXiv preprint arXiv:2408. 03539, 2024.
12

---

Published as a conference paper at ICLR 2025
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1 W..119//ROS..002.668888
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, et al. Jump-start reinforcement learning. In Inter- national Conference on Machine Learning, pp. 34556-345555 PMLR, 2023.
Arun Venkatraman, Martial Hebert, and Bagnell. Improving multi-step prediction of learned time series models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpe for model- reinforcement learning. In 2017 IEEE international conference robotics and automation (ICRA), pp. 1714-1721. IEEE, 2017.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Out- racing champion gran turismo drivers with deep reinforcement learning. Nature, 602 228, 2022.
Chenjun Xiao, Ruitong Huang, Jincheng Mei, Dale Schuurmans, and Martin Müller. Maximum entropy monte-carlo planning. Advances in Neural Information Processing Systems, 32, 2019.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international confer- ence on machine learning, pp. 24631-24645. PMLR, 2022.
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 5628-5635. IEEE, 2018.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.085000 2019.
13

---

Published as a conference paper at ICLR 2025
A DERIVATION
In this section, we provide detailed proof Proposition] We start with introducing a lemma to expand the action distribution (U))
Lemma 1. Given MDP with deterministic state transition the distribution of the action sequence q(U) at state Ny in horizon T, where each action Us 0, 0. is sequentially sampled from policy xt ((,, is
T ((U) = II T( we))
(7)
Proof. Let xuug u1, ur-1//20) denote the expanded notation of (U) by substituting u, and maximum-entropy policy и. Firstly, apply the conditional probability formula expand the equation:
x (u40 "1,""" " (2)) - (20)) / T(ui,".. .rrrr- 1200 uo, 2 (a( (00, uo )or'
(8)
Considering the Markov property of the problem,
π Tuu " uk+N PD, 2/,," 2//-11 uo, u1--1, x%).) = x (uks, " un -u-NNal w(()))
(9)
Eq. equation8 could be further expanded recursively till the end:
<uug U1,". ur (z0) = =
s(u). , uT Fo un x (p(2))((a)))))) loo, un dox
a(u,.... ur o )p(c ( up dar's
= =
x( (p()) [zo unn)
/ π T(u2. ... ur ur "
oo us )da da'
(10)
= = (zo) / plzr (wo, un) uo)
T
[(ull=))p do-1)do dx'r-1
t=2
Since the state transition is deterministic function
(1/441200 u.) 5( F(a (x))
(11)
14

---

Published as a conference paper at ICLR 2025
the integral signs could be eliminated by defining x++1 F(x,, ut):
/ p(z//1/ (zo (
πt uuu U1 uT- -
5I (2 (2n)
π (u (2/)p 2/ ue dar' dxcr-11
/ (1'442 Flx,, u.)) ((u1)z)))
(uo (20)
/
II (us ((/u)=//)(<<))) uu darr-1
(12)
/ plargla1, un)
(uo ao (11 (21) x
m(ulkz))pxx(l u 1)d22 dx'r-1
T-1
II t=
QED
With Lemma] 1 now provide detailed proof of Theorem
Theorem 1. Given MDP defined by M (K,U,,,P) with deterministic state transition defined with respect to dynamics model F and discount factor ? = 1. the distribution of the action sequence q" (U) at state xo horizon T, where each action u,,t 0,-..
sequentially sampled from the maximum-entre policy (Haarnoja et al. 2017 with an entropy weight a
T-1 r(a,, u.) + V*(IT)
q'(U) = Zas
cxp
Σ
(13)
where V is the soft value function (Faarrojj al. 2017) and æ, defined recursively from De and U through the dynamics model F as Øt+1 = F(x,, uz) = 0, ... 1.
Proof. The maximum entropy policy with an entropy weight a solves the problem M following the one-step Boltzmann distribution:
Zz. exp (_Q" (x) (o))
क (ulxx;)
(14)
where Zz, is the normalization factor defined as f exp (2 Q2 uz) du, and Q" (ax uz) the soft f-function as defined in (Haarnoja et al. 2017):
Q' (xt, ux) r(æ,, u.) log / exp Q''² (m1))) du.
(15)
ju
With Lemma I the optimal action distribution could be rewritten as:
T-1
(U) = II (ul
(16)
t=
where each 2; is defined recursively from 2o and U through the dynamics model F Ø[+1 F(x,, u,), = ..... ......T.... 1. By substituting Eq. equation and Eq. equation 15 Eq. equation 16 could be further expanded:
15

---

Published as a conference paper at ICLR 2025
(U)
(17a)
exp
(xr, ut)
(17b)
Ze:
[ exp
exp
r(0; u.) + a log
-0-
(# mc+1)
du
(17c)
1=0 Z
P-1
su
TT-1
exp
r(a:, ur) + log
II
Zay
+ log Zorr
(17d)
1=0 Zz
tm/
Zer
exp
r(w us) + a log Zor
(17e)
11=0 Ze
1 Zmo
Σ r(æ,, u.) + ((()) v((rr))
exp
(17f)
where Eq. equation 17b 17b results from substituting Eq. equation 14 and Eq. equation 17c results from substituting Eq. equation 15
V* (aT) a log Zæy is the soft value function defined in (Haarnoja et al, 2017) at the terminal step. Each or defined recursively from Do and U through the dynamics model F as I++1 = F Fxxz, 2 t 0, .....
In Eq. equation 17
QED
B COMPLETE DASIDUAL-MPPI DEPLOYMENT PIPELINE
The complete Rusidual-MPP deployment pipeline is shown in Algorithm
Algorithm Residual-MPPI Deployment Pipeline
Require: Prior policy π
1: Initialize a dataset < 0, dynamics Fo
2: for = 0, 1,... do
P Dynamics Training
3: u, = arg max π ( (x,) +
b Exploration Noise
4: I++1 t Environment( xt, Uz)
5: D DU Drr,, ur, 14-1)
> Multi-step Error
6:
-+ arg ming ED Et=o 7 - u)))
7:
end for
8: for /- = 0, 1, do
Zero-shot Residual-MPPI
9:
u, Residuall-MPP((ze))
10: F++1 < Environment(ar,, uz)
end for
12: for = 0, 1,.. do
> Few-shot Residual-MPPI
u, (esidual-MPPI((z.))
14:
I++1 + Environment (It, ut)
15:
D - DU (I,, Ur Tt-1)
Fine-tune with Online Data
+ arg ming ED
Etm 7 Foldr, uo)))
17: end for
C IMPLEMENTATION DETAILS IN MuJoCo
All the experiments were conducted on Ubuntu 22.04 with Intel Core i9-9920X CPU 3.50GHz 24, NVIDIA GeForce RTX 2080 Ti, and 125 GB RAM.
16

---

Published as a conference paper at ICLR 2025
C.I MuJoCo ENVIRONMENT CONFIGURATION
In this section, we introduce the detailed configurations of the selected environments, including the basic tasks, add-on tasks, and the corresponding rewards. The action and observation space of all the environments follow the default settings in gym [mu [mujoco] -v3.
Half Cheetah. In the Hal BalfCheetal ah environment, the basic goal to apply torque on the joints to make the chectah run forward (right) as fast as possible. The state and action space has 17 and 6 dimensions, and the action represents the torques applied between links.
The basic reward consists of two parts:
A.
Forward Reward Fforward (2), u4) At
(18)
Control Cost Tcontrol (Tt, U+) 0.1 x 14:12
During policy customization, we demand an additional task that requires the cheetah to limit the angle of its hind leg. This customization goal is formulated as an add-on reward function defined as:
T Un) 10 x |0aind legl
(19)
Hopper. In the Hopper environment, the basic goal is make the hopper move in the forward direction by applying torques on the three hinges connecting the four body parts. The state and action space has 11 and dimensions, and the action represents the torques applied between links.
The basic reward consists of three parts:
Alive Reward Talive
Ax
Forward Reward Tforward (Et, u.) At
(20)
Control Cost Trontral (2y u.) = 0.001 x /
The episode will terminate if the z-coordinate of the hopper is lower than 0.7, or the angle of the top no longer contained in the closed interval [-0.2, 0.2). or an element of the rest state is no longer contained in the closed interval 100, 100).
During policy customization, we demand an additional task that requires the hopper to jump higher along the z-axis. This customization goal is formulated
add-on reward function defined as:
TR (are ue) = 10 (= -
(21)
Swimmer. In the Swimmer environment, the basic goal is to move as fast as possible towards the right by applying torque on the rotors. The state and action space has and 2 dimensions, and the action represents the torques applied between links.
The basic reward consists of two parts:
Ax Forward Reward Fforward (Et, uk. At
(22)
Control Cost Teontrol (It, Uz) -0.0001 x |u.12
During policy customization, we demand an additional task that requires the agent to limit the angle of its first rotor. This customization goal is formulated an add-on reward function defined as:
T' 44) = -1 x Barst rotorl
(23)
Ant. In the Ant environment, the basic goal is coordinate the four legs (right) direction by applying torques on the eight hinges connecting the two links of each leg and the torso (nine parts and eight hinges). The state and action space has 27 and 8 dimensions, and the action represents the torques applied at the hinge joints.
in the forward
The basic reward consists of three parts:
Alive Reward Talive
Forward Reward Tforward (wt; U.) Ar
(24)
A/
Control Cost Toontrol (2r, Ue) = -05 x ||u,11
17

---

Published as a conference paper at ICLR 2025
Table 3: RL Prior Policy Training Hyperparameters
Hyperparameter
Value
Hidden Layers
(256, 256)
Activation
ReLu
0.99
Target Update Interval
50
Learning Rate
3e
Gradient Step
Training Frequency
Batch Size
256
Optimizer
Adom
Total Samples
64000000
Capacity
1000000
Sampling
Unifoorm
Table 4: MuJoCo Offline Dynamics Training Hyperparameters
Hyperparameter
Value
Hidden Layers
(256, 256, 256, 256)
Activation
Mish
learning rate
le
Training Frequency
10
Optimizer
Adam
Batch Size
256
lorizon
8
0.9
Total Samples
200000
Capacity
50000
Sumpling
Uniform
The episode will terminate if the z-coordinate of the torso is not in the closed interval [0.2, 1). During experiments, we set the upper bound to inf for both prior policy training and planning experiments as it benefits both performances.
During policy customization, we demand an additional task that requires the ant to move along the y-axis. This customization goal is formulated as add-on reward function defined as:
Ay AL
TR( Ue)
(25)
C.2 RL PRIOR POLICY TRAINING
The prior policies were constructed using Soft Actor-Critic (SAC) with the StableBaseline3 (Raffin al., (202)) implementation. The training was conducted in parallel across 32 environments. The hyperparameters used for RL prior policies training are shown in Table Since this parameter set- ting performed poorly the Swimmer task, we used the official benchmark checkpoint of Swimmer from StableBaseline3. Note that the prior policies do not necessarily need to be synthesized using RL. We report the experiment results with GAIL, prior policies in Appendix F.T
C.3 OFFLINE DYNAMICS TRAINING
The hyperparameters used for offline dynamics training are shown in Table 4 The exploration noise we utilized comes from the guassian distribution of the prior policy.
C.4 PLANNING HYPERPARAMETERS
The hyperparameters used for online planning shown in Table 5 At each step, the planners computes the result based on the current observation. Only the first action of the computed action sequence is sent to the system.
18

---

Published as a conference paper at ICLR 2025
Table 5: Planning Hyperparameter in MuJoCo Tasks
Hyperparameter
Half Chectah
Ant
Swimmer
Hopper
Horizon
Samples
10000
5000
5000
10000
Noise std.
0.017
0.005
0.02
0.005
w
le 7
le 1e-2
le
2e -
0.9
0.9
0.9
0.9
5e
D IMPLEMENTATION DETAILS IN GTS
All the GTS experiments were conducted on PlayStation 5 (PS5) and Ubuntu 20.04 with 12th Gen Intel Core 19- 12900F x 24, NVIDIA GeForce RTX 3090, and 126 GB RAM. GTS ran independently on PS5 with a fixed frequency of 60Hz. The communication protocol returned the observation and received the control input with frequency of 10Hz.
D.1 GTS ENVIRONMENT CONFIGURATION
) The reward used for GT Sophy 1.0 training was a handcrafted linear combination of reward com- ponents computed on the transition between the previous and current states, which included course progress, off-course penalty, wall penalty, tire-slip penalty, passing bonus, any-collision penalty, rear-end penalty, and unsporting-collinn tion, demand an additional task that requires the GT Sophy 1.0 to drive course. This cus- tomization goal is formulated as an add-on reward function defined as:
The action and observation spaces follow the configuration of GT Sophy 1.0 (Wurman al. 2022)
penalty (Wurman et al. 2022). During policy customiza-
( ue) 1000000 RdL
de d'map):
(26)
where doenter is the distance from the vehicle the course center and dmap the width of the course given the car's position.
D.2 DYNAMICS DESIGN
We employed three main techniques to help us get an accurate dynamics model.
Historical Observations To address the partially observable Markov decision processes (POMDP) nature of the problem, we included historical observations in the input to help the model capture the implicit information.
Splitting State Space We divided the state space into two parts: the dynamic states and map infor- mation. We adopted a neural network with an MLP architecture to predict the dynamic states. In each step, we utilized the trained model predict the dynamic states and leveraged the known map to calculate the map information based on the dynamic states.
Physical Prior Some physical states in the dynamic states, such as wheel load and slip angle, are in- trinsically difficult to predict due to the limited observation. To reduce the variance brought by these difficult states, two neural networks were utilized to predict them and other states independently.
Table 6 shows the hyperparameters used for training these two ML.ss to predict dynamic states.
D.3 PLANNING HYPERPARAMETERS
To stabilize the planning process, we further utilized another hyperparameter topratio to select limited action sequence candidates with top-tier accumulated reward. The hyperparameters used for online planning are shown in Table 7 At each step, the planners computed the result based on the current observation. Only the first action of the computed action sequence was sent the system.
D.4 RESIDUAL-SAC TRAINING
In addition to the Residual-SAC algorithm, we adopted the idea of residual policy learning (Silver al. 2018) to learn policy that corrects the action of GT Sophy 1.0 by outputting additive
19

---

Published as a conference paper at ICLR 2025
Table 6: GTS Offiine Dynamies Training Hyperparameters
Hyperparameter
Value
History Length
Hidden Layers
(2048,
2048, 2048)
Activation
Mish
Learning Rate
le
Training Steps
200000
Training Frequency
5
Batch Size
256
Horizon
5
7 Optimizer
1
Adam
Capacity
2000000
Sampling
Uniform
Table 7: Planning Hyperparameter in GTS
Hyperparameter
Horizon
Samples
Noise std.
Top Ratio
w/
A
Value
15
500
0.035
0.048
3
0.8 0.5
action, in which the initial policy was GT Sophy 1.0. As proposed in the same paper. the weights of the last layer in the actor network were initiated to be zero. The randomly initialized critic was trained alone with fixed actor first. And then, both networks were trained together. The pipeline developed upon the official implementation of RQL (Li et al. 2024). The training hyperparameters are shown Table g
Table 8: Residual-SAC Trainin Hyperparameters
Hyperparameter
Value
lidden Layers
(2048, 2048, 2048)
Activation
ReLu
I Learning Rate
le 4
Target Update Interval
50
Gradient Step
Training Frequency
Batch Size
256
Optimizer
Adam
0.9
Total Samples
10000000
Capacity Sampling
1000000
Uniforn
E VISUALIZATION
E. MUJOCo EXPERIMENTS
We visualize some representative running examples from the MuJoCo environment in Figure 4 As shown in the plot, the customized policies achieved significant improvements over the prior policies in meeting the objectives of the add-on tasks.
E.2 GTS EXPERIMENTS
We provide four typical complete trajectories of all policies in Figure 5 sim-to-sim experiment results in Figure The visualization clearly demonstrates the effectiveness of the proposed method by greatly reducing the off-course steps of champion-leve driving policy.
Also, we visualize the
We visualize the racing line selected by each policy at four typical corners to illustrate the differences among policies. As shown in Figure 7 GT Sophy 1.0 exhibited aggressive racing lines that tend to be off course. Both Zero-shot MPPI and Few-shot MPPI were able customize the behavior to drive
20

---

Published as a conference paper at ICLR 2025
0.6
Peior Polley
Roadival sonn
Pittr So
Guiced 9499
ded 19991
10-
angietrad)
anglefred)
0.0
400
600
1000
200
500
892
1020
Step
step
(a) Half Cheetah
(b) Swimmer
1.9
M
Prior Follcy Resictur-HM))
1.4
30
Plicr Pailyy
0.0
20
30
100
150
200
230
330
xtm)
x(m)
(c) Hopper
(d) Ant
Figure 4: (a) The angle of the Half Cheetah's hind leg vs. the environmental steps. (b) The angle of the Swimmer's first rotor vs. the environmental steps. (c) The trajectory of the Hopper robot on the and (d) The trajectory of the Ant robot the and y axis.
Residuii-SAC
(c) Zero-hor MPPI
(6) Fox-shot MPP)
Figure 5: Typical complete trajectories of all policies, where the red parts indicate off-course be- haviours. (a) The trajectory of GT Sophy 1.0. It finishes the lap in 117. 76.s with 93 steps off course. (b) The trajectory trajectory of Zero-shot MPPI. It finishes the lap in 123.551s with 10 steps off course. (d) The trajec- tory of Few-shot MPPI. It finishes the lap in 122.919. with 4 steps off course.
Residual-SAC. It finishes the lap 131.078s with 2 steps off course. (c) The
more on the course, while the Few-shot MPPI chose a better racing line. In contrast, the Residual- SAC yields to overly conservative behavior and keeps driving in the middle of the course.
Meanwhile, we visualize the difference between the actions selected by GT Sophy 1.0 and Residual- MPPT at each time step in Figure & When driving at the corner, method exerted notable influ-
21

---

Published as a conference paper at ICLR 2025
[a] GT Sopby 1.0 (I.)
[S) Sophy 1.0 (S.)
Zero shoc MPP
(d) Few shot MPP (S.)
Figure 6: Typical complete trajectories of all policies in Sim2Sim experiments, where the red parts indicate off-course behaviours. (a) The trajectory of GT Sophy with Race-Hard. It finishes the lap in 117.762s with 93 steps off course. (b) The trajectory of Residual-SAC with Race-Soft. It finishes the lap 116.674s with 133 steps off course. (c) The trajectory of Zero-shot MPPI with Race-Soft. It finishes the lap in 123.550s with 9 steps off course. (d) The trajectory of Few-shot MPPI with Race-Soft. It finishes the lap in 122.379s with 2 steps off course.
ences on the prior policy, as shown in Figure 8 (a). When driving at the straight, our method exerted minimal influences on the prior policy since most forward- preedcccde trajectories remain on course, as shown in Figure 8 (b).
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 RESIDUAL-MPPI MPPI WITH IL PRIOR
Residual-MPPI is applicable to any prior policies trained based on the maximum-entropy principle, not limited to RL methods. In this section, conduct additional experiments upon the same Mu- JoCo environments with IL prior policies. The IL prior policies are obtained through Generative Adversarial Imitation Learning (GAIL) with expert data generated by the RL prior in the previous experiments. The hyperparameters used for training the IL prior policies are shown in Table while the hyperparameters used for the learners the same the corresponding RL experts.
Table 9: Hyperparameters of GAIL Imitated Polices
Hyperparameter
Half Cheetah
Ant
Swimmer
Hopper
expert_min sepeed_ein_episddes
1000
100
1000
2000
biieh
512
25000
1024
1024
gen.replay.butte buffer.capaccity
8192
50000
2048
2048
n_disc_ bilsss_uddttssspeee per_ro
4
The results are summarized in Table 10 Similar to the experiment results with the RL priors, the
customized policies achieved significant improvements over the prior policies in meeting the objec- tives of the add-on tasks, which demonstrates the applicability of Residual-MPP with IL priors.
F.2 ABLATION IN MuJoCo
We conducted ablation studies for Guided-MPPI and Residual-MPPI in MuJoCo with 100 episodes for each selected setting, whose evaluation logs available in the supplementary materials.
Horizon. Regarding the question of why Guided-MPPI baseline underperforms the proposed ap- proach, as explained in the main text, while Guided-MPPI improves the estimation of optimal ac- tions through more efficient sampling, it remains hindered by its ability to account for the long-tem impacts of actions within finite planning horizons. In contrast, Residual-MPPI implicitly inherits
22

---

Published as a conference paper at ICLR 2025
Ussanca the Chans Catratinn
240
Gr-Sophy
ReaisunlSSAC
E
240
200
-10
Distaneem)
(a)
GT Sopty
Residual SAC
Zero-shat MiH
Datante to the Couree Centarte)
Foweshot MiM
Recidlol
y/mm
MPP
-5-
100
120
1920
1930
1940
1950
1960
1970
1990
1990
2000
2010
660
620
640
650
660
700
Course Dstancejm)
(b)
GT-Sashy
GT:pppy
10
Besidua
-50
Redioual-ACC
Zaro-thlt
thuanee the Chana Canarin)
-100
-150
-200
220
2600
2532
(c)
Resiaual
Orstatcc to the Course Contertio)
240
150
-10
120
4000
4840
4680
4000
4940
4950
6502
-400
-300
-300
-240
320
Course Dotancatm)
(d)
Figure 7: Route Selection Visualization at Different Cases. Different colors denote the different policies. In all cases, Residual-MPP
able to customize the behavior to drive more course. In (a), (b) and (d), due to inaccurate dynamics, Zero-shot MPPI exhibits off-course behavior. In (c), although Few-shot MPPI and Zero-shot MPPI both drive course, Few-shot MPPI tends to select a better route closer to the boundary with more accurate dynamics.
23

---

Published as a conference paper at ICLR 2025
Addiive Steering
Additive Steering
0.4
0.4
0.6
0.-
-0.4
-0.4
1000
1000
1050
11000
1130
1200
1250
1300
1350
Additve Threele
0.2
Addltivve Throttle
0.2
-02-
-022
-0.8
0.5
700
750
B02
950
1000
1000
1050
1100
1150
1200
1250
1300
1350
Distance(()
Course Distanod(m)
(a) Corner
(b) Straight
Figure 8: Additive Action of Residual-MPPI at Different Case. The additive throttle and steering all linear normalized to -1 to 1.
the information of the basic reward through the prior policy log-likelihood. And the prior policy log-likelihood is informed by the value functions optimized over an infinite horizon demonstra- tions. Therefore, our hypothesis, a longer horizon will benefit more on Guided-MPPI compared hypoth-
to Residual-MPPI. As shown in Figure 9 the performance of each algorithm matches esis. is worth noting that Guided-MPP still underperforms Residual-MPPI in all cases except in HalfCheetah. Meanwhile, in environments, overlong horizon decreases the performance. These results reflect the limitation of the planning-based methods with inaccurate dynamics.
Samples. The number of samples would have lower impact it primarily controls the sufficiency of optimization. When the number exceeds the threshold required for the optimization, further per- formance improvement can hardly observed. From the results shown Figure [O] we can see that the chosen number of samples is already sufficient for each optimization. However, the performance of Guided-MIl MPPII in Hopper and Ant reduces with more samples, which due the error from the terminal value estimation. In this situation, Residual MPPII demonstrates strong stability.
The temperature controls the algorithm's tendency average various actions or only focus on the top-tier actions. As shown in the Figure 11 a higher temperature can potentially lead instability, as observed in the Hopper and Swimmer tasks; however, can also allow more actions to be adequately considered, improving performance, observed in the Ant task.
Temperature.
Noise Std. The Noise Std. parameter controls the magnitude action noise, affecting the algo- rithm's exploration performance. As shown in Figure 12: a small noise would hinder the policy from exploring better actions, as observed in the Swimmer task; however, it can also increase stabil- ity, leading to higher overall performance, observed in the Hopper task.
Omega. Theoretically, w/ represents the trade-off between the basic task and the add-on task. As shown in Figure 13] in complex tasks that the add-on reward is orthogonal to the basic reward (e.g., Ant), a larger w' leads to behaviors close to the prior policy and higher basic reward; and a smaller w' leads to behaviors close to the Greedy-MPP and higher add-on reward but lower total reward.
F.3 ABLATION IN GTS
We also conducted the corresponding ablation studies for Few-shot MPPI in GTS with laps for each selected setting. Due to the strict experimental requirement of 10Hz control frequency, we found the maximum number of samples that satisfies this requirement for each chosen horizon and conducted experiments. The results shown in Figure 14 hypothesis.
Figure 15 and Figure I6 match our
In GTS, we also conduct an ablation w/ parameter. Theorectically, w/ represents the trade-off between the basic task and the add-on task, where larger w' should lead to a behavior closer to the prior policy with smaller lap time and larger off-course steps. On the other hand, oversmall w' would construct biased MDP, leading to suboptimal performace both on laptime and cof-course steps. The corresponding ablation results in GTS are shown in Figure 17 which also follow analysis.
24

---

Published as a conference paper at ICLR 2025
Table 10: Experimental Results of Zero-shot Residual-MPPI with II. Prior in MuJoCo
Full Task
Basic Task
Add-on Task
Env.
Policy
Total Reward
Basic Reward
[
Add-on Reward
Prior Policy
997.33 +
2288.9 725.5
0.33
0.01
-3260.3 93.5
Greedy-MPPI
-408.5 + 247.4
2397.9 + 285.3
0.28 = 0.01
-2806.4 141.3
Full-MPPI
-3589.7 353.9
-1164.4 152.0
0.24 0.03
-2415.2 + 341.3
Half
GuiPed-MPPI
-620.3 217.6
2236.3 + 219.2
0.29 = 0.01
-2856.6 93.6
Cheetah
Balued-MPPL
643.7 224.3
2231.7 202.9
0.29 0.01
-875.5
Residual-MPPI
415.2 1: 265.5
2383.7 303.6
0.28
2798.9 1 145.
Residual-SAC (200K)
-35.00 77.0
703.6 + 71.5
0.11 0.00
-1059.7 30.1
Residual-SAC (4M)
2111.5 79.1
2382.3 1: 78.0
0.03 = 0.00
-270.8 7.3
Env
Policy
Total Reward
Basic Reward
[
Add-on Reward
Prior Policy
334.55 3.3
328.2 + 1.5
0.67 0.00
-672.7 1.9
Greedy-MPPI
-35.1 = 7.1
232.7 3.8
0.27 0.01
0.01
-267.8 7.3
Full-MPPI
1685.3 1 108.1
13.5 1: 7.5
1.70 = 0.11
1698.8 = 107.4
Swimmer
Guided-MPPI
-122.6 6.9
222.1 4.8
0.34 = 0.01
-344.7 + 7.6
Valued-MPPI
-157.4
243.8 + 5.0
0.40 0.01
-401.3
ResP
35.1 6.7
232.8 1: 4.2
0.27 0.01
267.99 1: 7.2
Residual-SAC (200K)
168.1 + 113.8
-0.4 18.1
0.17 = 0.11
-667.7 114.7
Residual-SAC (4M)
-9.3 + 15.4
-0.1 14.3
0.01 + 0.01
-9.2 7.3
Env.
Policy
Total Reward
Basic Reward
Add-on Reward
Prior Policy
6790.9 40.2
3439.8 12.7
1.34 0.00
3351.1 29.4
Greedy-MPPI
6881.8 : 180.6
3423.9 1: 73.5
1.35 0.00
3457.9 1: 109.5
Full-MPPI Guided-MPPI
20.7 1: 3.2
3.6 1.61:00 0.7
1.24
17.1 = 2.6
Hopper
6793.8 301.1
3422.2 + 122.1
3370.8 183.1
Valued-MPPI
6832.2 179.6
3434.7 + 73.3
1.34 0.00
3397.4 + 108.3
Residual-MPPI
6902.8 41.0
3430.8 12.8
1.35 0.00
3472.0 = 30.7
Residual-SAC (200K)
5993.2 + 1327.7
3019.0 + 627.6
1.35 = 0.02
2974.1 = 709.0
Residual-SAC (4M)
7077.2 514.9
3445.9 + 229.2
1.37 = 0.00
3631.3 = 287.0
Env
Policy
Total Reward
Basic Reward
Add-on Reward
Prior Policy
4169.1 + 1468.3
4782.9 1639.1
-0.61 0.22
-613.7 216.4
Greedy-MPP1
4518.3 +: 1755.3
4018.8 1597.1
0.50 0500=0.1 0.18
499.4 181.6
Full-MPPI
- 153.7
-2774.4 110.1
-6,2 108.5
Ant
Guided-MPPI
3041.5 +: 2111.2
3098.6 2142.3
-0.06
55..1 139.4
Valued-MPPP
4321.5 + 1848.7
4727.7 7 1995.1
-0.41 = 0.22
1--006.2 222.6
Residual-MPPI
4877.6 1648.4
4565.9 1560.6
311.6 137.2
Residual-SAC (200K)
180.3 20.4
181.5 + 20.4
0.00 0.00
1.1 1.1
Residual-SAC (4M)
6016.9 + 1001.8
3836. + 705.3
2.18 0.30
2180.8 303.7
The evaluation results are in the form of mean = std over the 500 running episodes. The total reward calculated full tusk, whose reward is wrr + TR-
F.4 ABLATION OF DYNIMIC MODEL
To demonstrate the effectiveness of proposed dynamics training design, we conducted the cor- responding ablation study. The results in Table I show that planning based dynamics trained with a multi-step (5-step) error is more effective in both the laptime and off-course steps metrics. Moreover, using online data to fine-tune the dynamics model also yields significant performance improvements in the one-step error approach. Regarding the exploration noise, the effect of this technique in GTS is not particularly significant.
F.5 SIM-TO-SIM EXPERIMENTS
To motivate future sim-to-real deployments, we designed a preliminary sim-to-sim experiment by replacing the test vehicle's 's tires from Race-Hard (H.) Race-Soft (S.) to validate the proposed al- gorithm's 's robustness under suboptimal dynamics and prior policy. Since Residual- -PP-aa----PPP----- is a model- based herizzn control algorithm. The dynamic replanning mechanism and dynamics model
25

---

Published as a conference paper at ICLR 2025
Hopper
Swimmer
HalfCheetah
Reward
Reddiol 2011
Guidec MVM)
Choedy MP
Figure 9: Ablation Study of Horizon in MuloCo
Swimmer
HalfCheetah
Total Reward
samples
samples
Figure 10: Ablation Study of Samples in MuJoCo
adaptation could potentially enable robust and adaptive domain transfer under tire dynamics discrep- ancy. The experimental setup and parameter selection are consistent with those in Sec. 5.2
In sim-to-sim experiments, the domain gap brought by the tire leads to massively increased off- course steps in prior policy, which may lead to severe accidents in sim-to-real applications. In contrast, Residual-MPPI could still drive the safely course with minor speed loss, despite the suboptimality in prior policy and learned dynamics. Furthermore, as shown the Few-shot MPPI (S.) results, Residual-MPPI could serve safe starting point for data collection, policy finetuning, and possible future sim-to-real applications.
26

---

Published as a conference paper at ICLR 2025
Hopper
Swimmer
HalfCheetah
Reward
temperature
tamperature
temperature
temperature
Guides NV
Greedy MP
Figure 11: Ablation Study of Tempreture in MuloCo
Hopper
Reward
Add Reward
Resiuul--01 G.Mlac
Gre
Figure 12: Ablation Study of Noise Std. in MuJoCo
F.6 COMPARISON WITH PROMPT-DT
As discussed in the related work, Prompt-DT tries to solve the few-shot adaptation problem. Some readers may get interested in the differences between policy customization and few-shot adaptation. In this section, we provide detailed comparison of Prompt-DTT and Residual-MPP both in theory and experiments.
Comparing to policy customization, few-shot adaptation seeks to adapt policy solely to the new reward and requires full knowledge of the reward function. Moreover, few-shot adaptation scheme proposed in Prompt-DT relies a Decision Transformer-bbsed prior policy trained multi-task expert demonstration data via offline RL.
To make the comparison fair possible, we bring Prompt-DT to setting closer to the policy customization. We select the ANT-dir environment and test Prompt-DT with the official implemen- tation and its training parameters. Similar the Ant environment in our original experiments, define the basic reward as the progress along the x-axis, and the add-on reward as the progress along the y-axis.
First, we train a Prompt-DT model through multi-tase offline RL as the prior policy. The multi-task demonstration data consists of expert trajectories walking along various directions (see Prompt-DDT
27

---

Published as a conference paper at ICLR 2025
Hopper
Swimmer
HalfCheetah
75
1 1:111
Reward
Rosiiaa STM
Defasi Croady MPM
Delaul: Revctu M919
Figure 13: Ablation Study of w' in MuloCo
Off-course Steps
0.204
[horizon, samples)
HUse 90370
(horizon, samples)
BLRa 203m
Debaut Semitual-419
Figure 14: Ablation Study Horizon and Samples in GTS
Basic Task Demo in Table 13 the rewards the demo trajectories provided for the basic task). Prompt-DT can be directed to solve the basic task, by conditioning on trajectory prompt that is 1) sampled from the basic task demo, and 2) labeled with the basic reward function (see Prompt-DT on Basic Task in Table 13 for its performance).
We construct two types of prompts to adapt Prompt-DT the full task defined by the sum of basic and add-on rewards:
the trajectory prompt that is 1) collected with an expert policy trained on the full task, and 2) labeled with the total reward function. This prompt follows the original Prompt-DT setting. However, it has priviledged access to an expert policy and the total reward function, which not given in policy customization.
Expert Prompt:
the trajectory prompt that is 1) sampled from the basic task demo, and 2) labeled with the add-on reward function. It is the modification suggested by the reviewer to bring Prompt- DT closer to our setting.
Prior Prompt:
The performance of the customized policies is reported in Table 14 If provided with the prior prompt, Prompt-DT can hardly customize itself to the full task, with minor improvements on the add-on task. Moreover, when provided with the expert prompt, Prompt-IDT fails strike
28

---

Published as a conference paper at ICLR 2025
2300
260
off-course Steps
2400
temperature
temperature
Defout Nocihal 9111
Figure 15: Ablation Study of Temperature in GTS
4400
1350
ot couroo Steps
020
0180
0.53
640
Figure 16: Ablation Study Noise in GTS
good trade-off-iit sacrifices too much on the basic reward to get higher add-on reward. The total reward is even lower than the case with the prior prompt. In contrast, Residua-MPPI customizes the prior policy to achieve higher add-on reward with minor degradation on the basic task, leading to a much higher total reward.
29

---

Published as a conference paper at ICLR 2025
340
Off-courze Steps
DA0
1220
Detuu Resiaal MPP
Piar Palcy
Figure 17: Ablation Study of w/ in GTS
Table 11: Dynamics Design Ablation Results GTS
Modules
Add-on Task
Policy
Exploration Noise
Multi-Step Error
Finetune
Lap Time
Off-course Steps
one-step Zero-shot MPPI
124.2
13.2 + 3.8
5.3
one-step Few-shot MPPI
123.3 + 0.1
No-exp Zero-shot MPPI
123.5 1 0.3
10.0 3.7
Na-exp Few-shot MPPI
123.0 0.2
4.7 2.9
Zero-shot MPPI
123.3
9.0
Few-shot MPPI
122.9 0.1
4.4
The evaluation results the form mean +: std over 10 laps.
Table 12: Sim-to-Sin Sim Experimental Results of Residual-MPPI in GTS
Policy
GT Sophy 1.0 (H.)
GT Sophy 1.0 (S.)
Zero-shot MPPI (S.)
Few-shot MPPI (S.)
Lap Time
117.7 +: 0.1
116.8 + 0.1
123.5 = 0.2
122.6 + 0.3
Off-course Steps
93.1 2.0
131.5+22
8.3 3.6
3.9 + 2.9
The evaluation results are in the form of mean std over 10 laps.
Table 13: Evaluation of Prior Polices in Residual-MPPI and Prompt-DT
Prior Policy
Total Reward
Basic Reward
Add-on Reward
Residual-MPPI Prior
784.9 + 127.4
799.7 + 108.5
-14.8 + 58.5
Prompt-DT Basic Task Demo
779.5 + 109.0
793.0 + 61.7
-13.5 + 25.0
Prompt-DT on Basic Task
686.4 + 109.0
691.8 + 101.1
-5.4 + 45.5
The evaluation results are the form of mean 1: std over 500 episodes.
Table 14: Evaluation Customized Polices in Residual-MPPI and Prompt-DT
Customized Policy
Total Reward
Basic Reward
Add-on Reward
Prompt-DTT (Prior Prompt)
678.4 126.5
677.0 114.3
1.3 46.3
Prompt-DT (Expert Prompt)
626.0 187.8
605.6 184.8
20.4 55,5
RePi
872.0 69.0
774.8 41.8
97.1 48.5
The evaluation results are in the form of mean + std over 500 episodes.
30


Reviews:
[{"review_id": "4FR3ZxP3K0", "reviewer": "Reviewer_c2uW", "summary": "The paper introduces Residual-MPPI, a method for online policy customization in continuous control tasks. Residual-MPPI enables immediate policy adaptation at execution time by utilizing only the action distribution of a pre-trained policy, without needing additional training data or detailed knowledge of the original policy. Experiments demonstrate its effectiveness in adapting the champion-level racing agent GT Sophy 1.0 to custom constraints within the challenging Gran Turismo Sport (GTS) simulator environment, as well as in the robotic MuJoCo environment.", "questions": "- I am curious about the performance difference caused by the inclusion of $\\log \\pi$. How does the trajectory change when the hyperparameter $\\omega'$ is increased or decreased?\n- When performing policy customization without a given reward, how can the proposed Residual-MPPI be applied if only expert data or instructions are provided instead?\n- I am willing to raise the score if the authors respond to the weaknesses and questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The methodology is novel and well-structured.\n- The approach is validated in complex environments, including MuJoCo and GTS.", "weaknesses": "- The paper lacks SOTA baselines for policy customization. For instance, methods like those in [1, 2] demonstrate few-shot adaptability to new environments without an additional parameter training phase. A comparison with such method would strengthen the evaluation.\n\n[1] Xu, Mengdi, et al. \"Prompting decision transformer for few-shot policy generalization.\" *international conference on machine learning*. PMLR, 2022.\n\n[2] Liu, Jinxin, et al. \"Ceil: Generalized contextual imitation learning.\" *Advances in Neural Information Processing Systems* 36 (2023): 75491-75516.\n\n- The performance difference between Residual-MPPI and the comparison baseline, Greedy-MPPI, is not sufficiently analyzed. In Table 1, Residual-MPPI and Greedy-MPPI show very similar performance, yet the paper only briefly mentions on Line 430 that “Greedy-MPPI, as mentioned above, also leads to severe failure in complex GTS tasks” without further explanation. A more detailed analysis of Off-course Steps across different baselines would add clarity to the comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIvk64TWm2", "reviewer": "Reviewer_8CjN", "summary": "The paper studies the problem of online-fining pre-trained policies and proposes to combine the residual q learning and Model Predictive Path Integral (MPPI) together to form a new method, called Residual-MPPI, to address the problem. The paper provides a theoretical analysis of the proposed formulation and a comprehensive empirical evaluation of the method on Mujoco and GTS domains. The results are reported to be promising and the proposed method turns out to be effective.", "questions": "1.  In the experiment part, why the customised the policy need to maintain the same level of performance on the basic tasks? I understand that the full task is given by $wr+r_R$ and the add-on task is given by $r_R$. It makes sense to report results of customized polices on these two tasks. But what’s the motivation of comparing the results on task defined by $r$? since it is irrelevant to the customization objective. \n2. The paper formulated the policy customization as an augmented MDP, with the reward to be defined as $w’\\log \\pi(u|x) + r_R$. I’m wondering, in Table 1, the total reward refers to $ w’\\log \\pi(u|x) + r_R$ or $wr+r_R$? \n3. Re Theorem 1, I can’t understand why such theorem is needed? And what does it imply?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* [Originality] The paper attempts to integrate the residual Q learning into the Model Predictive Path Integral (MPPI) and show some promising results of such attempt. \n* [Quality and clarity] The paper is well written, and the structure is good in general. It is good to see the empirical evaluation has been conducted on multiple different domains and a relatively thorough results are reported. \n* [Significance] The proposed method considers an important problem of adapting a policy to new setting, which can be very useful for RL to be applied to many real-world applications.", "weaknesses": "### [Empirical evaluation appears to be weak] \n1. The empirical performance of the proposed method, Residual-MPPI, is quite similar to the heuristic modification of MPPI, i.e., Greedy-MPPI. On the main metrics, Full-task and Add-on Task, the reported performance of Residual-MPPI is approximately the same as that of Greedy-MPPI. The paper pointed out one difference on Ant Full Task. But it is unclear how the total reward was computed here and why this total reward should matter more than the add-on reward, given that Greedy-MPPI yields higher rewards than Residual-MPPI on add-on task. So please **provide a more detailed analysis of the differences between Residual-MPPI and Greedy-MPPI, particularly in cases where their performance is similar**. Also please **clarify how the total reward is calculated and why it's considered more important than the add-on reward in certain cases, especially for the Ant Full Task**. \n\n2. The baseline methods only cover some variants of MPPI. The paper only compared its proposed method with different variants of MPPI. However, as mentioned in the related work, there are many more baseline methods that had been discussed in contrast to the proposed method. So the paper should at least provide convincing argument that why these mentioned methods in related work should not be included as baselines. I think at least RL-driven MPPI, JSRL, and AWAC can be applied to the Mujoco environment to solve the online fine-tuning tasks. So please **include RL-driven MPPI, JSRL, and AWAC (if appropriate) to the baseline methods as they are discussed in the related work as important previous works**, or please **explain why these preivous methods in related work could not be included in the experiment**. \n\n3. Some metrics adopted in the experiments are debatable. The paper adopted two metrics, add-on reward and basic task reward, as the main measures of the required performance. However, I’m not sure if I understand it correctly: should we always focus on the augmented reward (or the add-on reward) as the sole measure? Since this is the main objective in the policy customization. See my questions in the Question Section. \n\n4. Results on GT Sophy and sim-to-sim experiments seem to be incomplete. It’s good to see the paper evaluates the proposed method on GT Sophy and sim-to-sim domains. But the results are relatively incomplete (e.g., no quantitative results are reported for Full, Guided, Greedy and Valued variants of MPPI). Given that the Greedy-MPPI was reported to perform similarly to the proposed method, Residual-MPPI, it would be necessary to see how Greedy performs in other domains.  So please **provide quantitative results for the missing MPPI variants in the GT Sophy and sim-to-sim experiments**. \n\n### [More clarifications and ablations are needed to understand the method]\nIn section 3.2, the paper introduced three main techniques to improve the learning of the transition dynamics function. However, the paper did not explicitly describe how these techniques are used in the method. At least, it’s not apparent to find them in the algorithm 1 (can’t find the multi-step error used as a loss function). Neither is clear about fine-tuning with online data. So please **provide a more detailed explanation of how the three techniques (multi-step error, exploration noise, and fine-tuning with online data) are integrated into the algorithm, possibly by updating Algorithm 1 or providing additional pseudocode.**\n\nFurther, there are no ablation studies on many design choices reported by the paper. Specifically,  it is not reported that how the above three techniques of learning the transition dynamics can affect the empirical performance of Residual-MPPI. Also, comparing Greedy-MPPI against Residual-MPPI, it can be hard to say it is necessary to include $\\log\\pi$ as a reward in Residual-MPPI since the two perform very similarly on HalfCheetah, Swimmer, and Hopper. \n\n### Clarity and the rigorousness \nSome sections lacks clarity. Section 3.2: DYNAMICS LEARNING AND FINE-TUNING appear to be disjoint to Section 3.1 since the proposed techniques in Section 3.2 cannot be found in Section 3.1 or the algorithm. Re Section 5.2.2, it is vague as to how the evaluation was conducted and what quantities had been compared against to draw the conclusions there.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eOk7czvMmd", "reviewer": "Reviewer_e9vs", "summary": "This work approaches the challenge of policy customization: given a prior policy (trained on a task, represented by a reward function), how should one adapt it at test-time to account for a different objective (add-on reward)? To tackle this problem, the paper introduces Residual-MPPI, which bridges Residual Q-Learning (an RL algorithm for policy customization) and MPPI (a planning method for model predictive control). The work firstly introduces a theoretical motivation for this mix of techniques (Section 3.1) and then provides experiments in continuous control tasks on MuJoCo and the GTS environment. Empirical results demonstrate Residual-MPPI is capable of adapting the policy with fewer samples than prior work, and not considerably compromising performance on the prior task.", "questions": "As mentioned in my previous points, how does Residual-MPPI performs in comparison to training a prior policy on the full task (initial task + add-on reward)?\n\n\n\n**Summary of the Review**:\n\nThe work is well written, the method is clearly presented and well motivated, the evaluation setup is also interesting. I do believe the work could improve in some directions, particularly adding a few more baselines and highlighting some important contributions. Nonetheless, I believe it is already in a good shape for acceptance.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The work is well written and structured, easy to follow. The adopted motivation and proposed method are also clear;\n\n- The evaluation setup is interesting: it brings MuJoCo benchmarks (which, although toy problems, are not trivial for continuous control) and also on GTS (which is a more complex setup for control). Therefore, the presented results are grounded in solid benchmarks.\n\n- The discussion in Section 4.2 is very elucidative, and convincingly justifies the failures of the baselines in comparison to the proposed method. This helps understand the strengths of Residual-MPPI.\n\n- Noticing that it is possible to replace the terminal value estimator for the log probabilities of the policy (into the MaxEnt framework) sounds particularly clever and also justifies well the proposed method.", "weaknesses": "While my concerns are not major, there are some suggestions that could improve the clarity of the results in the paper:\n\n- In Section 5.1, the paper argues that the metrics to evaluate on GTS are lap time and off-course steps since the rewards are complex. Nonetheless, I believe the work should provide both the prior task reward and the add-on reward for each method, as in the MuJoCo case, to understand how the method influences the returns that are optimized. Perhaps the considered small changes in lap time are associated with big changes in the returns, which would mean that Residual-MPPI is actually considerably deviating from the prior task designed by the environment.\n\n- Similarly, the work could provide results for Residual SAC in the MuJoCo environments, as the most direct comparison baseline in the literature. Currently, the only results are in GTS. Providing MuJoCo results should also give a better understanding about this direct comparison.\n\n- Another baseline that is important is training the “prior policy” on the full task (prior reward + add-on reward). Since MPPI is just a local search around the prior policy, perhaps it is too regularized towards the prior task. By providing the prior policy on the full task, the work provides a potential upper bound on joint performance, which contextualizes the performance of Residual-MPPI (as a policy customization method) to standard full RL training. This also helps clarify the trade-off of retraining a policy from scratch (expensive, but potentially better asymptotic performance) OR performing policy optimization (cheap, but potentially suboptimal).\n\n- Lastly, the work should probably highlight more the results in terms of sample efficiency. I understand that sample efficiency is key to policy customization, particularly in the physical world. The work just mentions that MPPI leverages 2100 laps while Residual-SAC leverage 80000 laps, which is a big difference. From my understanding, this is a big contribution of the method and should be highlighted. Some suggestions are either bringing a temple of sample analysis to the main paper or at least highlighting these results in the text. \n\n\nTypo: Section 5.2.1: “Compration”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "woVSVrWPkI", "reviewer": "Reviewer_VRbb", "summary": "This paper introduces an algorithm, Residual-MPPI, for customizing continuous control policies at execution time, without fine-tuning, in order to perform better at an auxiliary task while maintaining performance on the original task. Residual-MPPI does not require knowledge of the original task reward or value function; only the prior policy. However, the algorithm requires learning a dynamics model if one is not available, requiring additional online data to train and possibly iteratively improve as the policy changes (although apparently less data is required than the primary external baseline for policy customization, residual Q-learning). The approach is benchmarked against other variants of the algorithm on MuJoCo, with code provided, before being evaluated on the motivating use-case in Gran-Turismo Sport (GTS).\n\nIn summary, the paper has a strong problem motivation and proposed solution, but the quality and clarity of the paper could be significantly improved. As a result, this paper does not feel ready for publication yet, but I would be happy to increase my rating if these issues were improved before the end of the discussion period.", "questions": "- How were the MuJoCo add-on tasks chosen?\n- How was a dynamics model obtained for Residual-MPPI for these environments?\n- Why does Greedy-MPPI only degrade the performance for the Ant environment? It is stated that ignoring the log $\\pi$ would degrade performance when the add-on reward is orthogonal to the basic reward in the discussion which makes sense, but why is this only the case for the Ant environment?\n- Table 1 states that the evaluation results are over 500 episodes, but does not provide the number of training seeds/independent runs. How many training seeds were used? If only one seed was used, it should be stated that the std is over the evaluation period, not over different training runs, and this should be justified. \n- Why are two decimal places used in Table 1? This would be clearer to the nearest integer (except the undefined last but one column).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Motivation / Significance**\n  - The problem of customizing a policy at execution time without knowledge of the original objective, and while minimising additional online data collection/fine-tuning, is an interesting and important problem.\n  - This problem is well-motivated in the paper.\n- **Approach / Originality** \n  - The use of MPPI as a model-based approach to customize the policy at execution time seems sensible and original.\n- **GTS Application**\n  - The GTS results provide a clear use-case for the use of the method.\n  - The videos and results appear to show a clear improvement in the policy as a result.", "weaknesses": "- **Quality**\n\n  - The quality of the work should ideally be improved before the work is published. The full 10 pages are used, but there are many aspects which are unclear/undefined (see below), while there are large sections of text for which the key point is not clear. For example:\n    - The discussion (first paragraph of section 4.2) should be made more concise by breaking it up into the key conclusions. \n    - The abstract could also be made more concise e.g. 'Residual-MPPI can be used to improve new performance metrics at execution time, given access to the prior policy alone.' \n    - The related work could similarly be made more concise by focusing on the key difference with each work.\n  - Citations should be in brackets, not highlighted. \n  - The appendix should come after the references in the main paper, not in the supplementary materials.\n  - Other small typos and mistakes such as:\n    - L416: 'Compration'\n    - L361: 'apperant'\n\n- **Clarity / Novelty of Theory**\n\n  - The relevance and novelty of the theory is unclear to me. The approach seems to be a direct application of MPPI to Resdiaul-SAC. What is the intended contribution of Theorem 1 to the paper? It is derived in the Appendix, but the resulting Equation 5 just appears to be a direct result from [1] and it is mentioned that reference [2] already combines MPPI with SAC. Calling it a Theorem seems unnecessary if it is not a new theoretical result. The purpose of this theorem and the logic behind the derivation of Residual-MPPI could be made clearer.\n  - 'Few-shot' and 'zero-shot' are mentioned throughout the paper, but not defined until Section 5.1 and it is not clear what these terms refer to until this point. These should be defined in this context before use.\n\n- **Experiments / Significance**\n\n  - The add-on rewards used for the MuJoCo tasks are not stated in the main paper, only the Appendix. Some examples or at least the kind of add-on tasks considered should be included in the main paper to help the reader intuitively understand what kind of customization is being considered, and to define the last but one column of Table 1, which is meaningless otherwise. \n\n  - There are no external baselines for the MuJoCo experiements. Could residual-SAC be used as a baseline here? It is claimed that these results are zero shot, but Residual-MPPI would need a dynamics model for this to be the case. Where does the dynamics model come from for these experiments? If it is learned from data, then could Residual-SAC be trained on this data? The experimental evaluation would be strengthened by the inclusion of external baselines, or justification as to why these are not possible.\n\n  - Residual-MPPI seems to fall between the baseline GT Sophy and Residual-SAC in terms of both Lap Time and Off-course Steps in Table 2, so Residual-SAC performs more conservatively as claimed. However, the authors mention that Residual-SAC takes much more data (80k laps) compared to Residual-MPPI (~2k laps). What would happen if Residual-SAC was trained on the equivalent smaller amount of data? I wonder if it might customize the policy less and therefore perform less conservatively on the original task, better matching the performance of Residual-MPPI? I believe this would provide a fairer comparison of the two methods so would be helpful to include if possible.\n\n**References:**\n\n[1] *Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor*, Haarnoja et al., 2018\n\n[2] *Information Theoretic Model Predictive Q-Learning*, Bhardwaj et al., 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)