Residual-MPPI: Online Policy Customization for Continuous Control

Authors: Pengcheng Wang, Chenran Li, Catherine Weaver, Kenta Kawamoto, Masayoshi Tomizuka, Chen Tang, Wei Zhan
Primary area: applications to robotics, autonomy, planning
Keywords: Policy customization, Combination of learning- and planning-based approaches, Model predictive control

Abstract:
Policies developed through Reinforcement Learning (RL) and Imitation Learning (IL) have shown great potential in continuous control tasks, but real-world applications often require adapting trained policies to unforeseen requirements. While fine-tuning can address such needs, it typically requires additional data and access to the original training metrics and parameters.
In contrast, an online planning algorithm, if capable of meeting the additional requirements, can eliminate the necessity for extensive training phases and customize the policy without knowledge of the original training scheme or task. In this work, we propose a generic online planning algorithm for customizing continuous-control policies at the execution time, which we call Residual-MPPI. It can customize a given prior policy on new performance metrics in few-shot and even zero-shot online settings, given access to the prior action distribution alone. Through our experiments, we demonstrate that the proposed Residual-MPPI algorithm can accomplish the few-shot/zero-shot online policy customization task effectively, including customizing the champion-level racing agent, Gran Turismo Sophy (GT Sophy) 1.0, in the challenging car racing scenario, Gran Turismo Sport (GTS) environment. Code for MuJoCo experiments is included in the supplementary and will be open-sourced upon acceptance. Demo videos are available on our website: https://sites.google.com/view/residual-mppi.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729251255405

thread_type:
official_review

<thread_object>
creator: reviewer_vrbb

<content>
summary:
This paper introduces an algorithm, Residual-MPPI, for customizing continuous control policies at execution time, without fine-tuning, in order to perform better at an auxiliary task while maintaining performance on the original task. Residual-MPPI does not require knowledge of the original task reward or value function; only the prior policy. However, the algorithm requires learning a dynamics model if one is not available, requiring additional online data to train and possibly iteratively improve as the policy changes (although apparently less data is required than the primary external baseline for policy customization, residual Q-learning). The approach is benchmarked against other variants of the algorithm on MuJoCo, with code provided, before being evaluated on the motivating use-case in Gran-Turismo Sport (GTS).

In summary, the paper has a strong problem motivation and proposed solution, but the quality and clarity of the paper could be significantly improved. As a result, this paper does not feel ready for publication yet, but I would be happy to increase my rating if these issues were improved before the end of the discussion period.

strengths:
- **Motivation / Significance**
  - The problem of customizing a policy at execution time without knowledge of the original objective, and while minimising additional online data collection/fine-tuning, is an interesting and important problem.
  - This problem is well-motivated in the paper.
- **Approach / Originality** 
  - The use of MPPI as a model-based approach to customize the policy at execution time seems sensible and original.
- **GTS Application**
  - The GTS results provide a clear use-case for the use of the method.
  - The videos and results appear to show a clear improvement in the policy as a result.

weaknesses:
- **Quality**

  - The quality of the work should ideally be improved before the work is published. The full 10 pages are used, but there are many aspects which are unclear/undefined (see below), while there are large sections of text for which the key point is not clear. For example:
    - The discussion (first paragraph of section 4.2) should be made more concise by breaking it up into the key conclusions. 
    - The abstract could also be made more concise e.g. 'Residual-MPPI can be used to improve new performance metrics at execution time, given access to the prior policy alone.' 
    - The related work could similarly be made more concise by focusing on the key difference with each work.
  - Citations should be in brackets, not highlighted. 
  - The appendix should come after the references in the main paper, not in the supplementary materials.
  - Other small typos and mistakes such as:
    - L416: 'Compration'
    - L361: 'apperant'

- **Clarity / Novelty of Theory**

  - The relevance and novelty of the theory is unclear to me. The approach seems to be a direct application of MPPI to Resdiaul-SAC. What is the intended contribution of Theorem 1 to the paper? It is derived in the Appendix, but the resulting Equation 5 just appears to be a direct result from [1] and it is mentioned that reference [2] already combines MPPI with SAC. Calling it a Theorem seems unnecessary if it is not a new theoretical result. The purpose of this theorem and the logic behind the derivation of Residual-MPPI could be made clearer.
  - 'Few-shot' and 'zero-shot' are mentioned throughout the paper, but not defined until Section 5.1 and it is not clear what these terms refer to until this point. These should be defined in this context before use.

- **Experiments / Significance**

  - The add-on rewards used for the MuJoCo tasks are not stated in the main paper, only the Appendix. Some examples or at least the kind of add-on tasks considered should be included in the main paper to help the reader intuitively understand what kind of customization is being considered, and to define the last but one column of Table 1, which is meaningless otherwise. 

  - There are no external baselines for the MuJoCo experiements. Could residual-SAC be used as a baseline here? It is claimed that these results are zero shot, but Residual-MPPI would need a dynamics model for this to be the case. Where does the dynamics model come from for these experiments? If it is learned from data, then could Residual-SAC be trained on this data? The experimental evaluation would be strengthened by the inclusion of external baselines, or justification as to why these are not possible.

  - Residual-MPPI seems to fall between the baseline GT Sophy and Residual-SAC in terms of both Lap Time and Off-course Steps in Table 2, so Residual-SAC performs more conservatively as claimed. However, the authors mention that Residual-SAC takes much more data (80k laps) compared to Residual-MPPI (~2k laps). What would happen if Residual-SAC was trained on the equivalent smaller amount of data? I wonder if it might customize the policy less and therefore perform less conservatively on the original task, better matching the performance of Residual-MPPI? I believe this would provide a fairer comparison of the two methods so would be helpful to include if possible.

**References:**

[1] *Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor*, Haarnoja et al., 2018

[2] *Information Theoretic Model Predictive Q-Learning*, Bhardwaj et al., 2020

questions:
- How were the MuJoCo add-on tasks chosen?
- How was a dynamics model obtained for Residual-MPPI for these environments?
- Why does Greedy-MPPI only degrade the performance for the Ant environment? It is stated that ignoring the log $\pi$ would degrade performance when the add-on reward is orthogonal to the basic reward in the discussion which makes sense, but why is this only the case for the Ant environment?
- Table 1 states that the evaluation results are over 500 episodes, but does not provide the number of training seeds/independent runs. How many training seeds were used? If only one seed was used, it should be stated that the std is over the evaluation period, not over different training runs, and this should be justified. 
- Why are two decimal places used in Table 1? This would be clearer to the nearest integer (except the undefined last but one column).
</content>
</thread_object>

</thread>
<thread>
creation date:
1729772300451

thread_type:
official_review

<thread_object>
creator: reviewer_e9vs

<content>
summary:
This work approaches the challenge of policy customization: given a prior policy (trained on a task, represented by a reward function), how should one adapt it at test-time to account for a different objective (add-on reward)? To tackle this problem, the paper introduces Residual-MPPI, which bridges Residual Q-Learning (an RL algorithm for policy customization) and MPPI (a planning method for model predictive control). The work firstly introduces a theoretical motivation for this mix of techniques (Section 3.1) and then provides experiments in continuous control tasks on MuJoCo and the GTS environment. Empirical results demonstrate Residual-MPPI is capable of adapting the policy with fewer samples than prior work, and not considerably compromising performance on the prior task.

strengths:
- The work is well written and structured, easy to follow. The adopted motivation and proposed method are also clear;

- The evaluation setup is interesting: it brings MuJoCo benchmarks (which, although toy problems, are not trivial for continuous control) and also on GTS (which is a more complex setup for control). Therefore, the presented results are grounded in solid benchmarks.

- The discussion in Section 4.2 is very elucidative, and convincingly justifies the failures of the baselines in comparison to the proposed method. This helps understand the strengths of Residual-MPPI.

- Noticing that it is possible to replace the terminal value estimator for the log probabilities of the policy (into the MaxEnt framework) sounds particularly clever and also justifies well the proposed method.

weaknesses:
While my concerns are not major, there are some suggestions that could improve the clarity of the results in the paper:

- In Section 5.1, the paper argues that the metrics to evaluate on GTS are lap time and off-course steps since the rewards are complex. Nonetheless, I believe the work should provide both the prior task reward and the add-on reward for each method, as in the MuJoCo case, to understand how the method influences the returns that are optimized. Perhaps the considered small changes in lap time are associated with big changes in the returns, which would mean that Residual-MPPI is actually considerably deviating from the prior task designed by the environment.

- Similarly, the work could provide results for Residual SAC in the MuJoCo environments, as the most direct comparison baseline in the literature. Currently, the only results are in GTS. Providing MuJoCo results should also give a better understanding about this direct comparison.

- Another baseline that is important is training the “prior policy” on the full task (prior reward + add-on reward). Since MPPI is just a local search around the prior policy, perhaps it is too regularized towards the prior task. By providing the prior policy on the full task, the work provides a potential upper bound on joint performance, which contextualizes the performance of Residual-MPPI (as a policy customization method) to standard full RL training. This also helps clarify the trade-off of retraining a policy from scratch (expensive, but potentially better asymptotic performance) OR performing policy optimization (cheap, but potentially suboptimal).

- Lastly, the work should probably highlight more the results in terms of sample efficiency. I understand that sample efficiency is key to policy customization, particularly in the physical world. The work just mentions that MPPI leverages 2100 laps while Residual-SAC leverage 80000 laps, which is a big difference. From my understanding, this is a big contribution of the method and should be highlighted. Some suggestions are either bringing a temple of sample analysis to the main paper or at least highlighting these results in the text. 


Typo: Section 5.2.1: “Compration”

questions:
As mentioned in my previous points, how does Residual-MPPI performs in comparison to training a prior policy on the full task (initial task + add-on reward)?



**Summary of the Review**:

The work is well written, the method is clearly presented and well motivated, the evaluation setup is also interesting. I do believe the work could improve in some directions, particularly adding a few more baselines and highlighting some important contributions. Nonetheless, I believe it is already in a good shape for acceptance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730057470560

thread_type:
official_review

<thread_object>
creator: reviewer_8cjn

<content>
summary:
The paper studies the problem of online-fining pre-trained policies and proposes to combine the residual q learning and Model Predictive Path Integral (MPPI) together to form a new method, called Residual-MPPI, to address the problem. The paper provides a theoretical analysis of the proposed formulation and a comprehensive empirical evaluation of the method on Mujoco and GTS domains. The results are reported to be promising and the proposed method turns out to be effective.

strengths:
* [Originality] The paper attempts to integrate the residual Q learning into the Model Predictive Path Integral (MPPI) and show some promising results of such attempt. 
* [Quality and clarity] The paper is well written, and the structure is good in general. It is good to see the empirical evaluation has been conducted on multiple different domains and a relatively thorough results are reported. 
* [Significance] The proposed method considers an important problem of adapting a policy to new setting, which can be very useful for RL to be applied to many real-world applications.

weaknesses:
### [Empirical evaluation appears to be weak] 
1. The empirical performance of the proposed method, Residual-MPPI, is quite similar to the heuristic modification of MPPI, i.e., Greedy-MPPI. On the main metrics, Full-task and Add-on Task, the reported performance of Residual-MPPI is approximately the same as that of Greedy-MPPI. The paper pointed out one difference on Ant Full Task. But it is unclear how the total reward was computed here and why this total reward should matter more than the add-on reward, given that Greedy-MPPI yields higher rewards than Residual-MPPI on add-on task. So please **provide a more detailed analysis of the differences between Residual-MPPI and Greedy-MPPI, particularly in cases where their performance is similar**. Also please **clarify how the total reward is calculated and why it's considered more important than the add-on reward in certain cases, especially for the Ant Full Task**. 

2. The baseline methods only cover some variants of MPPI. The paper only compared its proposed method with different variants of MPPI. However, as mentioned in the related work, there are many more baseline methods that had been discussed in contrast to the proposed method. So the paper should at least provide convincing argument that why these mentioned methods in related work should not be included as baselines. I think at least RL-driven MPPI, JSRL, and AWAC can be applied to the Mujoco environment to solve the online fine-tuning tasks. So please **include RL-driven MPPI, JSRL, and AWAC (if appropriate) to the baseline methods as they are discussed in the related work as important previous works**, or please **explain why these preivous methods in related work could not be included in the experiment**. 

3. Some metrics adopted in the experiments are debatable. The paper adopted two metrics, add-on reward and basic task reward, as the main measures of the required performance. However, I’m not sure if I understand it correctly: should we always focus on the augmented reward (or the add-on reward) as the sole measure? Since this is the main objective in the policy customization. See my questions in the Question Section. 

4. Results on GT Sophy and sim-to-sim experiments seem to be incomplete. It’s good to see the paper evaluates the proposed method on GT Sophy and sim-to-sim domains. But the results are relatively incomplete (e.g., no quantitative results are reported for Full, Guided, Greedy and Valued variants of MPPI). Given that the Greedy-MPPI was reported to perform similarly to the proposed method, Residual-MPPI, it would be necessary to see how Greedy performs in other domains.  So please **provide quantitative results for the missing MPPI variants in the GT Sophy and sim-to-sim experiments**. 

### [More clarifications and ablations are needed to understand the method]
In section 3.2, the paper introduced three main techniques to improve the learning of the transition dynamics function. However, the paper did not explicitly describe how these techniques are used in the method. At least, it’s not apparent to find them in the algorithm 1 (can’t find the multi-step error used as a loss function). Neither is clear about fine-tuning with online data. So please **provide a more detailed explanation of how the three techniques (multi-step error, exploration noise, and fine-tuning with online data) are integrated into the algorithm, possibly by updating Algorithm 1 or providing additional pseudocode.**

Further, there are no ablation studies on many design choices reported by the paper. Specifically,  it is not reported that how the above three techniques of learning the transition dynamics can affect the empirical performance of Residual-MPPI. Also, comparing Greedy-MPPI against Residual-MPPI, it can be hard to say it is necessary to include $\log\pi$ as a reward in Residual-MPPI since the two perform very similarly on HalfCheetah, Swimmer, and Hopper. 

### Clarity and the rigorousness 
Some sections lacks clarity. Section 3.2: DYNAMICS LEARNING AND FINE-TUNING appear to be disjoint to Section 3.1 since the proposed techniques in Section 3.2 cannot be found in Section 3.1 or the algorithm. Re Section 5.2.2, it is vague as to how the evaluation was conducted and what quantities had been compared against to draw the conclusions there.

questions:
1.  In the experiment part, why the customised the policy need to maintain the same level of performance on the basic tasks? I understand that the full task is given by $wr+r_R$ and the add-on task is given by $r_R$. It makes sense to report results of customized polices on these two tasks. But what’s the motivation of comparing the results on task defined by $r$? since it is irrelevant to the customization objective. 
2. The paper formulated the policy customization as an augmented MDP, with the reward to be defined as $w’\log \pi(u|x) + r_R$. I’m wondering, in Table 1, the total reward refers to $ w’\log \pi(u|x) + r_R$ or $wr+r_R$? 
3. Re Theorem 1, I can’t understand why such theorem is needed? And what does it imply?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730187922863

thread_type:
official_review

<thread_object>
creator: reviewer_c2uw

<content>
summary:
The paper introduces Residual-MPPI, a method for online policy customization in continuous control tasks. Residual-MPPI enables immediate policy adaptation at execution time by utilizing only the action distribution of a pre-trained policy, without needing additional training data or detailed knowledge of the original policy. Experiments demonstrate its effectiveness in adapting the champion-level racing agent GT Sophy 1.0 to custom constraints within the challenging Gran Turismo Sport (GTS) simulator environment, as well as in the robotic MuJoCo environment.

strengths:
- The paper is well-written and easy to follow.
- The methodology is novel and well-structured.
- The approach is validated in complex environments, including MuJoCo and GTS.

weaknesses:
- The paper lacks SOTA baselines for policy customization. For instance, methods like those in [1, 2] demonstrate few-shot adaptability to new environments without an additional parameter training phase. A comparison with such method would strengthen the evaluation.

[1] Xu, Mengdi, et al. "Prompting decision transformer for few-shot policy generalization." *international conference on machine learning*. PMLR, 2022.

[2] Liu, Jinxin, et al. "Ceil: Generalized contextual imitation learning." *Advances in Neural Information Processing Systems* 36 (2023): 75491-75516.

- The performance difference between Residual-MPPI and the comparison baseline, Greedy-MPPI, is not sufficiently analyzed. In Table 1, Residual-MPPI and Greedy-MPPI show very similar performance, yet the paper only briefly mentions on Line 430 that “Greedy-MPPI, as mentioned above, also leads to severe failure in complex GTS tasks” without further explanation. A more detailed analysis of Off-course Steps across different baselines would add clarity to the comparison.

questions:
- I am curious about the performance difference caused by the inclusion of $\log \pi$. How does the trajectory change when the hyperparameter $\omega'$ is increased or decreased?
- When performing policy customization without a given reward, how can the proposed Residual-MPPI be applied if only expert data or instructions are provided instead?
- I am willing to raise the score if the authors respond to the weaknesses and questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732002054552

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author General Response to Common Concerns (Part1)

comment:
We thank the reviewers for their thoughtful feedback and suggestions. We are delighted that all the reviewers found the Residual-MPPI framework novel and the majority of the reviewers found it essential (R2, R3, R4). We also appreciate the reviewers' acknowledgment of the thorough empirical evaluation (R1, R2, R3, R4) and clarity of the presentation (R1, R2, R3). In this thread, we summarize our responses to the common concerns shared by the reviewers. For a reviewer-specific response, please refer to the rebuttal attached to each reviewer's comments.

### Clarification of Problem Formulation
Firstly, we want to emphasize that Residual-MPPI considers a novel problem setting that has not been addressed in the literature before. Specifically, Residual-MPPI aims to solve the policy customization task, where the goal is to **customize a given prior policy towards jointly maximizing an add-on reward and the underlying reward of the prior policy, which is unknown in a priori**. The policy customization problem was first formulated in RQL [12], where the authors proposed initial solutions leveraging model-free RL, as well as a model-based online solution based on MCTS for discrete action spaces. In Residual-MPPI, we aim to further solve the policy customization problem in a sampling-efficient manner online for continuous control problems. 

### 1: Baselines Comparison 

Some reviewers suggest some existing approaches in the literature that could potentially solve the policy customization task and are not compared against Residual-MPPI in our work. We would like to thank the reviewers for pointing us to these relevant papers in the literature, including those baselines already covered in the paper and those suggested by the reviewers. They are worth discussing and comparing in our context. In this section, we provide a thorough comparison between Residual-MPPI and these relevant works in the literature.


**First, the MPPI variants we covered in experiments can represent the performance of various planning approaches**. **Full-MPPI** represents the performance of the commonly used sampling-based MPC approach, MPPI [1], to maximize the total reward without a prior policy. **Guided-MPPI** further incorporates the prior policy during the sampling stage, which is similar to one ablation version of RL-driven MPPI [8]. **Valued-MPPI** assumes access to the value function together with the prior policy, representing the performance of the most commonly used SOTA approaches, such as MuZero [2], TD-MPC [3, 4], MBOP [5], and RL-driven MPPI [8]. **Greedy-MPPI**, in contrast to the previous variants, requires no knowledge of the basic reward and only maximizes the add-on reward without utilizing the $\log \pi$ term, directly serving as a baseline to evaluate the importance of the $\log \pi$ term in Residual-MPPI.

**Second, some reviewers suggested RL fine-tuning/ Few-shot adaptation works [6, 7, 9, 10] as baselines. However, we want to emphasize that their problem settings are different; thus, they are not appropriate baselines in our setting**. As stated in the problem formulation, policy customization aims to jointly optimize the unknown basic reward and the add-on reward of the full task. In contrast, RL fine-tuning algorithms, such as JSRL [9] and AWAC [10], aim to fine-tune a prior policy solely towards a new reward function. The prior policy is used to regularize and accelerate fine-tuning. The difference between policy customization and RL fine-tuning has been elaborated in the original RQL paper. Similarly, the goal of the suggested few-shot adaptation works, Ceil [6] and Prompt-DT [7] are also to adapt the prior policy solely toward a new task. Moreover, Ceil [6] and Prompt-DT [7] require an extra demonstration dataset to adapt the policy to the new task, while such expert demonstrations are unavailable in the online policy customization setting. Therefore, we believe the mentioned works are not appropriate baselines to include.


**Finally, some reviewers have suggested adding Residual-SAC and RL policies trained on Fulltask as baselines in the experiments.** In response to the requests from R3 and R4, we added Residual-SAC as a baseline in MuJoCo. Furthermore, in both MoJoCo and GTS experiments, we reported the performance of Residual-SAC trained with the same amount of data used for dynamics model training in Residual-MPPI to illustrate the superior sample efficiency of Residual-MPPI. As suggested by R3, the RL policy trained on Fulltask can serve as a performance upper bound of the policy customization task. We have included this baseline in the MoJoCo experiment (see Table 1, Section 4).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Residual-MPPI, a method for online policy customization in continuous control tasks. The proposed method bridges model predictive path integral (MPPI) control and residual Q-Learning to adapt a pre-trained policy to new objectives at execution time without additional training on the prior task. The experiments conducted in the MuJoCo and Gran Turismo Sport (GTS) environments demonstrate sample-efficient adaptation.

Reasons to accept
- Generally well-written and structured, with a clear problem statement and motivation.
- This work tackles the challenging problem of policy customization at execution time without fine-tuning or detailed knowledge of the original reward function.
- The proposed method combines MPPI and residual Q-Learning in a novel way for policy customization.
- The evaluation is comprehensive, covering diverse benchmarks (MuJoCo and GTS).
- The experimental results demonstrate significantly improved sample efficiency of the proposed method compared to baselines.

Reasons to reject
- Missing ablation studies on many design choices. (mostly addressed by the rebuttal)
- Residual-MPPI performs similarly to Greedy-MPPI in many tasks, but there is insufficient analysis of these results. (addressed by the rebuttal)
- Missing results for a "prior policy" trained on the full task, which could serve as an upper performance bound. (addressed by the rebuttal)
- Terms like "few-shot" and "zero-shot" are introduced late without adequate explanation.
- Details on how dynamics models are trained and used are sparse. (mostly addressed by the rebuttal)

Overall, this paper studies a promising research direction and presents an interesting and effective approach. Many of the concerns and questions raised by the reviewers were sufficiently addressed by the author's rebuttal and in the revised paper. Consequently, I recommend accepting the paper.