ICLR 2025

Published as a conference paper at ICLR 2025
BOOTSTRAPPED MODEL PREDICTIVE CONTROL
Yuhang Wang, Hanwei Guo, Sizhe Wang, Long Qian, Xuguang Lan∗
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence
National Engineering Research Center for Visual Information and Application
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China
{wertyuilife,clannnagisa,wangsizhe,qianlongym}@stu.xjtu.edu.cn
xglan@mail.xjtu.edu.cn
ABSTRACT
Model Predictive Control (MPC) has been demonstrated to be effective in con-
tinuous control tasks. When a world model and a value function are available,
planning a sequence of actions ahead of time leads to a better policy. Existing
methods typically obtain the value function and the corresponding policy in a
model-free manner. However, we find that such an approach struggles with com-
plex tasks, resulting in poor policy learning and inaccurate value estimation. To
address this problem, we leverage the strengths of MPC itself. In this work, we
introduce Bootstrapped Model Predictive Control (BMPC), a novel algorithm that
performs policy learning in a bootstrapped manner. BMPC learns a network pol-
icy by imitating an MPC expert, and in turn, uses this policy to guide the MPC
process. Combined with model-based TD-learning, our policy learning yields bet-
ter value estimation and further boosts the efficiency of MPC. We also introduce
a lazy reanalyze mechanism, which enables computationally efficient imitation
learning. Our method achieves superior performance over prior works on diverse
continuous control tasks. In particular, on challenging high-dimensional loco-
motion tasks, BMPC significantly improves data efficiency while also enhancing
asymptotic performance and training stability, with comparable training time and
smaller network sizes. Code is available at https://github.com/wertyuilife2/bmpc.
1
INTRODUCTION
Model-based reinforcement learning (MBRL) algorithms that incorporate online pla...