Bisimulation Metric for Model Predictive Control
Authors: Yutaka Shimizu
Masayoshi Tomizuka
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Model-based reinforcement learning
optimal control
MPC
Primary area: reinforcement learning
TL;DR: A new model-based reinforcement learning algorithm using bisimulation metric

Abstract: Model-based reinforcement learning (MBRL) has shown promise for improving sample efficiency and decision-making in complex environments. However, existing methods face challenges in training stability, robustness to noise, and computational efficiency. In this paper, we propose Bisimulation Metric for Model Predictive Control (BS-MPC), a novel approach that incorporates bisimulation metric loss in its objective function to directly optimize the encoder. This optimization enables the learned encoder to extract intrinsic information from the original state space while discarding irrelevant details. BS-MPC improves training stability, robustness against input noise, and computational efficiency by reducing training time. We evaluate BS-MPC on both continuous control and image-based tasks from the DeepMind Control Suite, demonstrating superior performance and robustness compared to state-of-the-art baseline methods.

Published as a conference paper at ICLR 2025
BISIMULATION METRIC FOR MODEL PREDICTIVE CONTROL
Yutaka Shimizu & Masayoshi Tomizuka
Mechanical Engincering University of California, Berkeley
Berkeley, CA 94720, USA
(purewater0901, tomlzukaj@Bberkeleyy
ABSTRACT
Model-based reinforcement learning has shown promise for improving sample ef- ficiency and decision-making in complex environments. However, existing meth- ods face challenges in training stability. robustness to noise, and computational efficiency. In this paper, we propose Bisimulation Metric for Model Predictive Control (BS-MPC), a novel approach that incorporates bisimulation metric loss its objective function directly optimize the encoder. This time-step-we -wie direct optimization enables the learned encoder to extract intrinsic information from the original state space while discarding irrelevant details and preventing the gradients and errors from diverging. BS-MPC improves training stability, robustness against input noise, and computational efficiency by reducing training time. We evaluate BS-MPC on both continuous control and image-based tasks from the DeepMind Control Suite, demonstrating superior performance and robustness compared state-of-the fetthhe-rt art baseline methods.
INTRODUCTION
Reinforcement learning (RL) has become a central framework for solving complex sequential decision- making problems in diverse fields such as robotics, autonomous driving, and game play- ing. Among RI. methods, model-based reinforcement learning (MBRL) gets its attention thanks to its ability achieve higher sample efficiency and better generalization. Representation learning fur- ther enhances MBRL by encoding high-dimensional information into compact latent spaces, which can accelerate learning by focusing on essential aspects of the environment. However, achieving stable and robust representations remains challenge, especially in high-dimensional partially observable environments, where noise and irrelevant features can degrade performance.
One prominent MBRL method, Temporal Difference Model Predictive Control (TD-MPC)) (Hansen ctt al. 2022),, combines temporal difference learning with model predictive control to improve pol- icy performance by simulating future trajectories in the learned latent space. TD-MPC sets itself apart from other methods by leveraging the learned latent value function as a long-term reward estimate approximate cumulative rewards, allowing for the efficient computation of optimal ac- tions. Despite its successes, TD-MPC suffers from several limitations, including instability during training, vulnerability noise, and expensive computational costs, which are shown in Fig [ The first graph illustrates TD-MPC's performance degradation during training, demonstrating notable collapse after certain number of steps. The second set of results focuses on an image-based task, where the addition of background noise (adding a completely irrelevant image to the background) led TD-MPC's s failure to achieve a high reward in the noisy environment. The third picture shows that TD-MPC suffers from long calculation time. These problems are attributed to the encoder's training method and the objective function's structure.
To address these issues, introduce Bisimulation Metric for Model Predictive Control (BS-MPC), a new approach that leverages x -i-iimiulaaioon metric (on-pole con-policy bisimulation metric) (Zhang et al. 2021) to improve the stability and robustness of latent space representations. Bisimulation met- rics measure behavioral equivalence between states by comparing their immediate rewards and next state distributions, providing formal way to ensure that the learned latent representations retain

---

Published as a conference paper at ICLR 2025
meaningful and essential information from the original states. In BS-MPC, minimize the mean square error between the on-policy bisimulation metric and é1-distance latent space at each time step, directly optimizing the encoder to improve stability and noise resistance. Integration of the bisimulation metric gives BS-MPC a theoretical guarantee, ensuring that the difference in cumula- tive rewards between the original state space and the learned latent space can be upper-bounded over a trajectory. This value function difference bound validates the fidelity of the encoder projection. Additionally, the proposed method reduces training time by making the computation of the objec- tive function parallelizable, leading to less computational cost than TD-MPC. All these performance improvements are also summarized in Fig [
We implement BS-MPC using the Model Predictive Path Integral (MPPI) (Williams et al. 2016 2018) framework and evaluate its performance on variety of continuous control tasks from Deep- Mind Control Suite (Tassa et al. 2018). Our results show that BS-MPC outperforms existing model- free and model- based methods in terms of performance and robustness, making it promising new approach for model-based reinforcement learning.
humanoid run
pendulum swingup
finger spin
Episode Return
1000
1000
12
Time [h]
500
500
6
0 0
2M 4M
6M
8M
1M
2M
3M
0
1M 2M Step
3M 4M
Step
Step
BS-MPC (Ours)
TD-MPC
Figure 1: Three open problems of TD-MPC. (Left) TD-MPC initially performs well but collapses after 4 mil- lion steps, while BS-MPC steadily improves. (Middle) With added distraction in the input image, TD-MPC fails to gain rewards, whereas BS-MPC remains robust. (Right) BS MPC roduces training time by removing sequential computation in objective function.
2 RELATED WORK
Reinforcement Learning Reinforcement Learning (RL) (Sutton & Barto 2018 has two main
approaches: model-free methods (Silver al. 2014 Fujimoto et al. 2018 Haarnoja al.) 2018a 8a
Schulman et al. 2015) 2017;
Kalashnkov et al. 2021 Kalashnikov et al. 2018 Mnih et al. 2015
Hessel et al. 2018
Yarats al. 2021 2022 Laskin et al. 2020) and model-based methods (Sutton
2024 Luo et al. [2019] Janner et al. 2019 Chua et al. 2018 Schrit- function and policy, Wang & Ba 2020). While model-frec methods focus on learning the value methods aim to learn the underlying model of the environment, using this learned model to compute optimal actions. This paper focuses on the model-based ap- proach, specifically methods that combine planning and MBRL (Hansen et al., 2022 2024), which learns the underlying model in the latent space and applies sampling- based Model Predictive Con- trol (MPC) (Williams et al.) 2016 Kobilarov 2012) to solve the trajectory optimization problem. Several variants of TD-MPC have been proposed (Lancaster et al. 2024 Zhao et al. 2023 Chitnis et al. 2023 Feng al. 2023 Wan et al. 2024), but none fully address all the challenges outlined in Fig. I To the best of our knowledge, BS-MPC is the first approach to tackle all three open problems in TD-MPC, as discussed in Section
Harne et al.
2020 2021
twieser et al. 2019
2

---

Published as a conference paper at ICLR 2025
Representation Learning Learning models in the latent space is an efficient way to approximate internal models, especially for image-base based tasks. One approach to learning latent space projections is by training both the encoder and decoder to minimize the reconstruction loss (Lange & Ried- miller) 2010 Lange et al., 2012 Hafner et al., 2019 2024 Lee et al., 2020). However, this method often suffers model errors and instability and has difficulties in long-term predictions. An alterna- tive approach is to train only the encoder to obtain the latent representation. Bisimulation (Larsen & Skou 1989 is state abstraction technique defined in Markov Decision Processes (MDPs) that clusters states producing identical reward sequences for any given action sequence. |Ferns et al. (201T): Ferns & Precup (2014) defined a bisimulation metric that measures the similarity between Two states based on the Wasserstein distance between their empirically measured transition distri- butions. However, computing this metric can computationally expensive high-dimensional spaces. To address this, Castro (2020) proposed an on-policy bisimulation metric, which considers the distribution future states under the current policy, providing a scalable way to measure state similarity. Zhang et al. (2021) extend this idea to x² -bisimuultiion metric by minimizing MSE loss between bisimulation metric and & distntteeee in latent space to train the encoder. Following this ap- proach, the on-policy bisimulation metric train the encoder in model-based reinforcement learning architecture.
PRELIMINARIES
This section provides brief introduction to reinforcement learning and its associated notations, along with an explanation of bisimulation concepts.
3.1 REINFORCEMENT LEARNING
Reinforcement Learning (RL) aims to optimize agents that interact with a Markov Decision Process (MDP) defined by a tuple (S,A, A. P, R, Po. 7), where S represents the set of all possible states, A the set of possible actions, R is a reward function, Po is the initial state distribution, and y the discount factor. When action E A is executed at state E S, the next state is generated according to 8' P(-1s, a), and the agent receives stochastic reward with mean r(s, a) E IR.
The Q-function (s, for policy <-11s) represents the discounted long-term reward attained by executing given observation history and then following policy ı there- after. Q" satisfies the Bellman recurrence: Q=(s,a)
r(s, a) + ylE:g)n [Qh+1 (s), The value function V considers the expectation of the Q- function over the policy (*hh) = Enwx((ls) [Q=(s. Meanwhile, the Q-function of the opti- mal policy Q* satisfies: (*(s,a) = r(s,a) + (EEs-P()ssa)) [max,' ('' a')], and the optimal value function is (*) max, (" a). Finally, the expected cumulative reward is given by J (T) = Ear-ps (s)). The goal of RL is to optimize a policy T(. | s) that maximizes the cumulative reward 7² 8) argmax (T))
[F(T((ss
In large-scale or continuous environments, solving reinforcement learning (RL) problems can be challenging due to the prohibitively high computational cost. To address this issue, function approx- imation is often employed to estimate value functions and policies. With function approximation, present Q" as Q09, Tw. with 0 and v their parameters. With a replay buffer B, the policy evaluation and improvement steps at iteration & can be expressed as:
[Qoo -R(()) a) - 7.E oog (10) Qog) (8, (s,, 5]
02 argmin E (s,a,))))
-
(1)
Vk+1 + argmax Eg-B, 00g (s, (s,
where 0º are target parameters that are a slow-moving copy of og. Note that in this paper, we denote target parameters in this paper.
3.2 BISIMULATION METRIC
When working with fech-dimensionll state problems, it is often helpful to group "similar" states into the same set. Bisimulation is type of state abstraction that groups state 8; and sy if they
3

---

Published as a conference paper at ICLR 2025
"behaviorally equivalent" (Li et al. 2006). A more concise definition states that two states bisim- ilar they yield the same immediate rewards and have equivalent distributions over future bisimilar Larses(Larsen & Skou, 1989 Givan et al. 2003). Bisimulation metric quantifies the bisimilarity of two states 8j and 85- It is defined with y-th Wasserstein metric (PP,,,P,, which represents the distance between two probability distribution Pi and P2:
Definition 1. (Bisimulation metric (Ferns et al. 2011)). The following metric exists and is unique, [o, 1] and E (0, 1) for continuous MDPs:
given R:
d( (s 83) max(1 R(; a)
P(1ss,),))
In high-dimensional and continuous environments, analytically computing the max operation in 2q.2in is challenging. In response to this difficulty, Castro (2020) proposed new approach, known the on-policy bisimulation metric (or a-bisimulation).
Definition 2. (On-Policy bisimulation metric bisimulation metric uniquely exists
(Castro 2020 ) Given fixed policy æ, the following
d d(s 83) 1r
r2 + WW (+|s:), P" (+(ss))
where
r = Σ ((s))) (T((s)= = <a(ass)
(als)) Σ P(s'(s,a) s'ES
(4)
2021) extended the concept the on-policy bisimulation metric (referred to the n' rr-bisimulation metric) to learn comparable metric in the latent space Z. In their approach, the encoder is learned by minimizing the mean square error between the on-policy bisimulation metric and l distanee in the latent space.
Recently, (Zhang et al.
- J(Ø) (10(())) - ((1))1 -
WW2 (P(11((()))) a.): P(11(())); (((())))))
(5)
where the latent dynamics model P is modeled with Gaussian distribution. In Eq. 2- Wassertttii metric W2 is used because it has a convenient closed form for Gaussian distribution. Following this approach, we train our encoder similarly by including this MSE loss (Eq, 5) in our objective function.
4 BISIMULATION METRIC FOR MODEL PREDICTIVE CONTROL
We introduce Bisimulation Metric Model Predictive Control (BS-MPC), robust and efficient model-based reinforcement learning algorithm. This section provides a detailed explanation of BS-MPC algorithm. Furthermore, we present theoretical analysis that bounds the suboptimality of cumulative rewards in the learned latent space under our architecture. Finally, we highlight three key distinctions between BS-MPC and TD-MPC that contribute to their performance differences.
4.1 BISIMULATION METRIC FOR MODEL PREDICTIVE CONTROI.
We introduce BS-MPC, an improvement over TD-MPC that employs म" bissimulationn metrics to train the encoder. The training flow for BS-MPC is detailed in Appendix C
Components BS-MPC shares the same five core components as TD-MPC: encoder, latent dynam- ics, reward, state-action value and policy.
Encoder:
z high (8k)
Latent dynamics:
11- do~ ak)
Reward: F = = (2k, ak)
State-action value: Q = Qoo (#k, ax)
(6)
Policy: āg ~ Tp (2x)
When the input s is a state vector, the encoder is modeled as a multi-layer perceptron (MLP) and as convolutional neural network (CNN) when 3ę is image. Given the latent state 2 and action dg. we compute the next latent state 2 using the latent dynamics model dga (#k: dx), parameterized by 8d. Following other model-based reinforcement learning methods, model the latent dynamics with an MLP. BS-MPC estimates the reward fĄ and state-action value Qx based 2₁ and ds.
4

---

Published as a conference paper at ICLR 2025
modeling both RøR and Q64 with MLPs. Finally, we train a policy that outputs the estimated optimal action âs given 2y; the policy is also parameterized as an MLP.
At each time step kt the original observation sk is encoded into the latent state 2R- Using 2 and the action ag. we compute the rewards, state-action values, and the next latent state. As highlighted in prior work, these values are computed in the latent space rather than the original observation space, as the latent state 25 captures the essential information from the high-dimensional original state. Since the latent space typically has more compact dimensions, this approach is commonly used in image-based tasks where input images are high-dimensional. However, state-based tasks, despite being represented more compactly, also benefit from this structure by utilizing latent states learned through temporal consistency (Zhao et al. (2023).
Objective function We jointly train the encoder, latent dynamics model, reward model, and state- action value model. BS-MPC minimizes the following loss function:
H
0- = arg min [()) arg min ((s,,,;;')
Σ * (x)
(7)
LA=
where 0 = [ 00, 0d, 0*1 This objective function is identical to the one proposed in TD-MPC. However, BS-MPC has an additional bisimulation metric loss term every time step, as shown in Eq. 5
(x = 01 Rox (Zk, ax) rxll +02 11Q0a (Z 0x) - (rx + 7Q80 (#k+1, mga (=x+1))) 13
(A) reward loss
(B) state-action value loss
+ 03 |gg as) -
hag (S++ +C4 (1lhgn (sk) - (gg (6x) - Ins - Fxl - > >|12x11 1 - 2x+11 112)
(C) consistency loss
(D) bisimulation
(8)
where 6; permute (x)) and Ek+1 dg (2k: 0+)- C1, C2 C3, C4 are parameters that can change the weight of each loss. The last term is an expansion of Eq. [ under the assumption that the a Dirac delta distribution (i.e., a Gaussian
model outputs deterministic predictions, corresponding distribution with zero variance). As in TD-MPC, we use the same three loss components: (A) reward loss, (B) state-value action loss, and (C) consistency loss. Each training loss aims to update its corresponding parameters, i.e. reward parameter 0², state-action value parameter 08, and dynamics parameter 0d These losses also help to update the encoder parameter (f by using the derivative of the composition function. In addition to these losses, BS-MPC includes bisimulation metric loss in its objective function, which explicitly depends on the encoder parameters gh. This bisimulation metric loss (D) is designed to train the encoder to learn representation space where the & diiittne corresponds to the n -bisimulation metric.
For policy training, we use the following loss function to update the policy parameter sb.
H
16 = arg min Jn = arg min
Σ k=
x Quo (2k, To (Zx))
(9)
In Section. 4.3 we give details about the benefit of using Eq. ass the objective function for BS-MPC and the differences between our approach and TD-MPC.
Model Predictive Control with Learned Model Following TD-MPC, our method has closed- loop controller using the learned latent dynamics model, reward model, state-value function, and prior policy to compute the optimal action. Due to the high affinity between reinforcement learning and sampling-based planners, we design closed-loop controller using MPPI, type of sampling- based MPC, following the approach of TD-MPC. MPPI is a derivative- free method that samples a large number of trajectories, calculates the weight for each, and then generates the optimal trajectory by taking the weighted average of these trajectories. This framework enables us to solve the local trajectory optimization problem.
First, it encodes the current observed state s, into the latent space with the trained encoder 2, hig (8t). After that, we sample M action sets from Gaussian distribution ((, (4a,0") o²) based on the
5

---

Published as a conference paper at ICLR 2025
initial mean 2t and standard deviation o², and each set contains H length actions dd where M. Starting from the initial latent state 24, we use the learned latent dynamics 24+1 dos (2 at) and sample M trajectories. We then calculate the weight of each trajectory based on its cost and compute the weighted mean of the sampled trajectories to get the updated optimal trajectory. The parameter μa and 0g is updated by maximizing the following equations:
1 1+t1 ghtt arg max E (N.0)
? " ((++H) +
Σ S Rgn (En.
(10)
h=t
where ((*++) = Qaa (2+4H, Ty Th (Z)+R)) We continue this calculation until it reaches the given number of iterations. More details can be found in (Hansen al. 2022 Williams et al. 2016 2018).
4.2 THEORETICAL ANALYSIS
is important to measure the quality of the learned representation space. In this section, we show that BS-MPC upper-bounds expected cumulative reward by leveraging value function bounds de- rived from the on-policy bisimulation metric. This property, absent in TD-MPC, strongly differenti- ates BS-MPC.
We assume that the learned policy in BS-MPC continuously improves throughout training and even- tually converges the optimal policy '², which supports Theorem
Theorem 1. (Theorem 1 in (Zhang et al., 2021) Let's 's assume policy π in BS-MP continuously improves over time, converging to the optimal policy x². Under this assumption, the following bisimulation metric has a least fixed point d and that x -bbiimuulatiin metric.
d(s;, 85) = (1-c)lrā,
+ c,, -,,
(11)
where WG (P,, (inf (infyer(p,..))) Ssxs d(s,, 8,) dr' (s., 1/p all couplings of P; and Pj.
and r(P:, Pj) is the set of
Proof can be found in Zhang et al. 202T). Under this x* bisimuuuattio metric, we can divide the latent space into partitions based on some >> 0, where 1 < (1-c)e Let & represent the encoder that maps each original state from the state space S to corresponding e-cluster. With these notations, (Zhang et al. (021) shows the following value bound based on bisimulation metrics.
Theorem 2. (Theorem in (Zhang et al., 2021)) Consider an MDP M, which is formed by clus- tering states within e-neighborhood, along with encoder 6 that maps states from the original MDP M to these clusters. Under the same assumption in Theorem optimal value functions for the two MDPs are bounded by
2c - 2C
(V'(s) V (1 (1-7)(1-2
(12)
where L. = sup,, a ES [l.())) ((s,) )| di (s,, s5)1 is the learning error for encoder d. Note that this theorem assumes access to the true dynamics model P and reward function R.
Proof can also be found in (Zhang al. 2021). This result demonstrates that the optimal value function in the original state space and the optimal value function in the latent space, projected by the n" -iiimmmulatoo metric, is bounded from above. Leveraging Theorem [ and Theorem 2 we can bound the cumulative reward of a trajectory under the original MDP M and the latent MDP M.
Theorem 3. Consider a trajectory т = (so, ag, $1, Q1, $H -H 8H) in the original state space S, and its corresponding encoded trajectory Φ(T) = (zo, an 21, 01 ........ -1--, 0H-1, z
where ag ~ r. sx) and 25 ((sx), with & defined as in Theorem Under the same assumption in Theorem 1 and Theorem 2 the following expected cumulative rewards
H
S(T) = E,
> H (SH)
Σ
r R(sh an)
S(O(7)) = E,
7 [=V=(((sm)) +
A R(d(sa),
h=0

---

Published as a conference paper at ICLR 2025
can be bounded as follows:
2c(1 -) (1 (1-7)(1-))) -c) (1-7)(1- cc)
()+
((() - S(((T))
(13)
Proof can be found in Appendix A.1 This theorem states that if the cluster radius and the en- coder error & are sufficiently small, the learned representation space 2 does not change the original cumulative rewards over the same trajectory т. This suggests that the latent space retains essential information from the original space.
4.3 DIFFERENCE BETWEEN BS-MPCC AND TD-MPC
The main difference between BS-MPC and TD-MPC lies in its objective function and computation flow. Specifically, BS-MPC updates the encoder parameter by minimizing MSE loss between on- policy bisimulation metric and & disistteeee in the latent space at every time step k, which is shown in Eq. 7 These differences result in the following improvements.
Improved training stability In TD-MPC, the encoder is only updated indirectly through gradi- ents propagated from the latent dynamics loss, as the objective function lacks explicit encoder loss term and only consists of the first three terms in Eq. 8 This indirect update makes it challenging to effectively optimize the encoder parameters each training step, potentially leading to significant inconsistencies in the latent dynamics and destabilizing the learning process. Such model inconsis- tencies have also been reported in Zhao et al. (2023). In contrast, BS-MPC has an explicit encoder loss in its objective function thanks to the inclusion of bisimulation metric loss. This allows the gradients of the encoder parameters to directly computed from the objective function, ensur- ing continuous improvement of the encoder. As a result, the learned encoder effectively maps the original state s to the latent space 2, leading to reduced model discrepancies. The encoder update difference is shown in Fig. 2a and Fig. 2b
Theoretical support of the latent space The encoder in BS-MPC generates a latent representation Z where the Éi-distance corresponds to the bisimulation metric. This indicates that the encoder efficiently filters out irrelevant information from the original state s and preserves intrinsic details in the latent state z. Consequently, BS-MPC exhibits robust resilience noise. Additionally, this property guarantees that the cumulative reward difference between the learned representation space and the original space over trajectory is the encoder in TD-MPC lacks theoretical guarantees in its learned representation space, potentially leading to the projection of irrelevant details into the latent space. This absence theoretical validity in the encoder contributes to increased sensitivity to noise, as shown in our experimental results (Section. 5.3)
as discussed in Section 4.2 In contrast,
Ease of parallelization TD-MPC predicts the latent state bx+1 by applying the dynamics model to the previous predicted latent state Êx. introducing a sequential dependency that hinders parallel computation (see Lines 12 to 17 of Algorithm 2 in Hansen al. (2022). In contrast, BS-MPC generates the predicted latent state â by encoding the current state into the latent state 24 = has (5k) and using it as input to the dynamics model, which removes the sequential dependency and allows for parallel computation across time steps. Fig. 2a and Fig. (b show the calculation flow difference berween TD-MPC and BS-MPC. Consequently, BS-MPC achieves faster computational times compared to TD-MPC.
5 EXPERIMENTS
We evaluate BS-MPC across various continuous control tasks using the DeepMind Control Suite (DM Control (Tasaa et al. (0188). The inputs in these experiments include both high-dimensional state vectors and images, with some tasks set in sparse-reward environments. The objective of this section is to demonstrate that BS-MPC maintains its performance over time and remains robust to noise. Additionally, we aim to confirm that it outperforms TD-MPC in terms of computational efficiency. In this experiment, we focus specifically on comparing BS-MPC with TD-MPC. For fair comparison, BS-MPC uses the same model architecture as TD-MPC, with an identical
7

---

Published as a conference paper at ICLR 2025
TD-MPC
BS-MPC
2g-1
d1
d2
81
22
11
Qx-1
Fy-1
11
2:
2
Fg-1
Loss
Loss
(a) TD-MPC calculation flow
(b) BS-MPC calculation flow
Figure 2: Calculation flow comparison. The black line shows the forward calculation flow, and the red arrows represent the gradient of 0*. While TD-MPC needs sequential calculation in its forward compu- tational flow, BS-MPC can process the calculation parallel. Moreover, BS-MPC has explicit encoder loss its cost function, so its derivative directly updates the paramcters of the encoder. Note that TD-MPC only encodes the original observation at the initial time step and predicts latent states by using the latent dynamics model.
number of parameters and tuning parameters set to the same values. We also run BS-MPC and TD-MPC using the same random seeds. The only difference between BS-MPC and TD-MPC explicit encoder loss the objective function with an additional parameter c4. We adopt the same
environmental settings those used in the original TD-MPC paper (Hansen al. 2022 Detailed
experimental configurations are provided in Appendix
For the basclines, model-based RI. algorithm Dreamer-v3 (Hafner et al.] 2024), and the planning- based model modell-bsed RL approach TD-MPC [ansen al. (2022) and its successor TD-MPC2 Hansen et al. (2024). evaluating them on both state-based and image-based tasks. Note that we use a model with million parameters for TD-MPC2 because is their default model. In addition these algorithms, we also compare BS-MPC with DrQ-v2 Yarrat el al. 2022) and CURL (Laskin et al.) 2020) on image- based tasks. We publicly release the value of episode return at each time step and code for training BS-MPC agents.
the model-free RL algorithm SAC (Harrojj et al. 2018a 2019) the
5.1 RESULT ON STATE-BASED TASKS
We evaluate BS-MPC across 26 diverse continuous control tasks with state inputs, comparing its performance to other baseline methods. In this setting, agents have direct access to all internal states of the environment.
Fig. 3 shows the average performance of each algorithm across 10 tasks, along with the individual scores from 9 specific tasks. We ran 10 million time steps for the dog experiment and 8 million time steps for the humanoid experiment. For the other tasks, the experiments were run for million time steps. The results demonstrate that BS-MPC consistently outperforms existing model-based and model-free reinforcement learning methods, particularly in high-dimensional environments. On complex tasks involving dog and humanoid environments, BS-MPC significantly outperforms TD- MPC, SAC, and Dreamer-v3. In particular, BS-MPC achieves higher episode returns early in train- ing and maintains superior performance throughout, whereas the other methods either plateau or display instability. TD-MPC performs well in the early stages of training, achieving competitive results up to approximately million steps. However, in many tasks, its performance suddenly col- lapses after this point, leading to high variance and reduced episode returns. Both BS-MPC and TD-MPC2 resolve the issue of performance divergence observed in TD-MPC; however, TD-MPC2 requires many more parameters and employs more complex model architectures. Additionally, TD- MPC2 requires significantly more computation time than both TD-MPC and BS-MPC due to its reliance on discrete regression for optimizing the reward and value function models. It is important to note that BS-MPC and TD-MPC share the exact same model architecture, hyperparameters, and number of parameters. The only difference between them lies in the cost function: BS-MPC explic- itly minimizes the bisimulation metric loss at every time step to train the encoder, whereas TD-MPC only calculates the encoder loss at the initial time step. Appendix, B shows all the results from 26
8

---

Published as a conference paper at ICLR 2025
continuous control tasks, computation time comparison, and detailed analysis of the training failure of TD-MPC.
Average
dog trot
dog walk
finger spin
fish swim
1000
Episode Renurn
500
0
1M
2M
3M
4M
0
2M 4M 6M SMIOM
2M 4M 6M SMICM
1M
2M
3M
4M
0
1M
2M
3M
4M
Step
Step
Step
Step
Step
humanoid run
humanoid walk
quadruped run
walker stand
walker walk
Fpisede Retum
1000
4M
2M
4M Step
0M
2M Step
3M
4M
4M
1M
Step
Step
Step
BS MPC (Ours)
TD MPC
SAC
Dreamee v3
TD MPC2 (5M)
Figure Performance comparison on the average over 26 state-based tasks and DM Control tasks with state imput. At each evaluation step, the episode return is computed over 10 episodes. The results are averaged over 3 seeds, with shaded regions representing the standard deviation. Results for SAC and Dreamer-v3 are obtained from (Hansen their official code with the same architecture and hyperparameters for BS-MPC, We use the same seeds for evaluation.
(2024), and results for TD-MPC are reproduced using
5.2 RESULTS ON IMAGE-BASED TASKS
Next, we evaluate BS-MPC and other baseline methods on image-hased tasks from 10 DM Control environments. In these tasks, the encoders both BS-MPC and TD-MPC modeled by CNN to project high-dimensional image data into compact latent space. To ensure a fair comparison, we use the same model architecture, hyperparameters, and number of parameters for both BS-MPC and TD-MPC, and we use identical seeds for evaluation. We run million environmental steps for all tasks. Fig. 4 shows the results across 10 image-based tasks. BS-MPC demonstrates performance competitive with TD-MPC, DrQ-v2 and Dreamer-v3, consistently outperforming CURL and SAC. TD-MPC2 converges faster than BS-MPC certain tasks (e.g., quadruped-run and walker-run); however, BS-MPC achieves comparable or slightly better performance overall with fewer parame- ters than TD-MPC2.
cartpole balance
cartpole swingup
chectab run
cup catch
finger spin
1000
Epioode Retur
500
MUNN
0
IM
2M
3M
1M
2M
3M
0
1M
2M
3M
1M
2M
3M
0
1M
2M
3M
Step
Step
Step
pendulum swingup
reacher easy
walker run
walker stand
walker walk
1000
Epiiadee Retur
500
1M
3M
1M
Step
Step
Ntep
Step
Step
TD-MPC
CURL
Figure Performance comparison on DM Control image- based tasks. each evaluation step, the episode
return computed over 10 episodes. The results are averaged over 3 seeds, with shaded regions representing the standard deviation. Results for DrQ-v2 are obtained from their official results, and results for CURL, SAC and Dreamer-v3 obtained from Dreamer-v3 code (Hafner al. 2024

---

Published as a conference paper at ICLR 2025
5.3 RESUTLS ON IMAGE INPUT WITH DISTRACTIONS
Finally, we evaluate the robustness of the proposed method in the presence of distracting informa- tion. The goal of this experiment is to show that BS-MPC has better resilience against irrelevant data in the input. We benchmark BS-MPC on 5 DM Control tasks by introducing irrelevant information into the imput as noise. Following (Zhang al. 2018 2021), driving videos from the Kinetics dataset (Kay et al.) 2017) are used as background for the original images. In this experiment, the same parameters and architecture as in Section 5.2 are employed, and the performance BS-MPC is compared to that of TD-MPC.
Figure 3 shows the experimental results, which reveal that BS-MPC constantly outperforms TD- MPC in every environment. Since TD-MPC does not have an explicit objective function for its encoder, its encoder simply learns representation space to keep the latent dynamics consistent. Therefore, its encoder struggles filter out the noise during training. BS-MPC, however, learns its encoder by minimizing the bisimulation metric loss to retain bisimulation information in learned representation space. This architectural modification enhances performance and increases robustness to noise compared to TD-MPC.
cheetah run
pendulum swingup
walker run
walker stand
walker walk
Episode Return
1000
1000
1000
1000
1000
500
500
500
500
1M
3M
1M
2M
2M
1M
3M
1M 2M 3M
Step
Step
Step
Step
BS-MPC. (Oun)
MD-MPC
Figure 5: Performance comparison on 5 DM Control image-based based tasks with distracted information from Ki- neties dataset. At each evaluation step, the episode return is computed over 10 episodes. The results are averaged over seeds, with shaded regions representing standard deviation. (Top) Original Image. (Middle) Distracted Image. (Bottom) Performance results. BS-MPC constantly outperforms TD-MPC when the input is disturhed.
CONCLUSION
In this paper, we propose a novel model-based reinforcement learning method called Bisimulation Metric for Model Predictive Control (BS-MPC). While inheriting several properties from TD-MPC, our approach differentiates it from the previous method in three key areas: inclusion of explicit en- coder loss term, adaptation of bisimulation metric, and parallelizing the computational flow. These improvements stabilize the learning process and enhance the model's robustness to noise while re- ducing the training time. Experimental results on continuous control tasks from DM Control demon- strate that BS-MPC has superior stability and robustness, whereas TD-MPC and other baselines fail to achieve comparable performance.
Limitations. Despite the theoretical foundations and experimental results supporting BS-MPC, has one notable limitation: the need for extensive parameter tuning of сд across different environ- ments. In this paper, we employ a grid search to identify the optimal parameter values; however, future research should foeus on developing methods for automatic parameter adjustment.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This research was supported by TIER IV, Inc. through the Student Research Scholarship program.
REFERENCES
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov de- cision processes. In Proceedings of the Thirty- Fourthh AAAI Conference Artificial Intelligence (AAAI-20), 2020.
Rohan Chitnis, Yingehen Xu, Bobak Hashemi, Lucas Lehnert, Ürün Dogan, Zheqing Zhu, and
Olivier Delalleau. Iql-td-mpe: Implicit q-learning for hierarchical model predictive control. 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 9154 2023. URL
https: www.eeeiieeneollaaeeeeeee org/Corpus :ttp:/ccrpusD::55999.... 228899967
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 4759-4770, Red Hook, NY, USA, 2018. Curran Associates Inc.
Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Fine- tuning offline world models in the real world. In 7th Annual Conference on Robot Learning. 2023. URL https::
net/ forum? id- JkF UKFFeECCVXY
Norm Ferns and Doina Precup. Bisimulation metrics are optimal value functions. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI' 14, pp. 210-219, Arling- ton, Virginia, USA, 2014. AUAI Press. ISBN 9780974903910.
Norm Ferns, Prakash Panangaden, and Doina Precup. Bisimulation metrics for continuous markov decision processes. SIAM Journal on Computing, 40(6); 166-11666 2011. doi: 10.1137/
10080484X. Utt/attss://ai..ooo orw.:00...r..... 111.1..1117//100488801110
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, Stockholmssmissar. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591.. PMLR, 2018. URL
wwtpeeeddiiii.........mmm.. www..ppress. vt////00///// wwt.///ee/oollleeeeeoeoooo www.omomaii...... 1motol8a.htmm
Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence, 147(1):163 2201633-22222222 2003. ISSN 0004-3702 doi: 00tp0/0000.000000000000000000000 coom loom/science/an http:///rrriicee/a11/00000000000 complete Information.
URL httpp::////tt //w::///wwwww /wwww scienced rect. Planning with Uncertainty and
Tuomas Haamojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul ml www.smmlrr.rsss.. v80eeess.vvv// a ja 8b. html
2018a. httttps::/ppoceeee/ee
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, Abs/1812.05905, 2018b. URL http:://axiv..omm wwtexiiioorg/bss..
1812 2. 05905
Tuomas Haarnoja, Sehoon Ha, Aurick Zhou, Jie Tan, George Tucker, and Sergey Levine. Learning to walk via deep reinforcement learning. In Proceedings of Roboties: Science and Systems, FreiburgimBreisgaau Germany, June 2019. doi:
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha. Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning.
11

---

Published as a conference paper at ICLR 2025
volume 97 of Proceedings of Machine Learning Research, pp. 2555 2565. PMLR, 09-15 Jun 2019. URL htt.p::://prceeednggs. mlz http://......e.e////////aaaaaaa
htmi
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
URL https: htttpp:///peereeee
net/ wtt.o/wwwweee/eeeeeeeee id=S110CCADDS
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with
discrete world models. In International Conference on Learning Representations, 2021. URL
https://openrevil ew net. forum? d=t.:ummmiid000aoobbbb...
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models, 2024. URLbttpp htttttppp//aatee //axxiv org/ aaa//////////aa abs/abbs..3//./////// 22tp//301001144
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control, 2024.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 Proceedings of Machine Leurning Research, pp. 8387-8406. PMLR, 17-23 Jul 2022. htttts::/proceeeeeeeeeeeee
mir www.s/pess//vveee
html
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dab- ney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: combining im- provements in deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Con- ference Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 18/AAII 18/EAAI 18. AAAI Press, 2018. ISBN 978-1-57735-800-88
Michael Janner, Justin Fu, Marvin Zhang. and Sergey Levine. When to trust your model: Model- based policy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché- -uc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL TRL/tttpss/pprrceddnngg 5 http:///0.....itt............a..
files/
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, al. Scalable deep reinforce- ment learning for vision-based robotic manipulation. In Conference on robot learning, pp. 651- 673. PMLR, 2018
Dmitry Kalashnkov, Jake Varley, Yevgen Chebotar, Ben Swanson, Rico Jonschkowski. Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv, 2021.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and An- drew Zisserman. The kinetics human action video dataset. CoRR, abs/1705.0695. 2017. URL
httt:://axiii org/ httpg//bb/170.00000
Marin Kobilarov. Cross-entropy motion planning. Int. J. Rob. Res., 31(7):855-871, jun 2012. doi: 10. 10.11774077866411444444 URL
ISSN 0278-3649,
02 0278364912444543
Patrick Lancaster, Nicklas Hansen, Aravind Rajeswaran, and Vikash Kumar. Modem-v2: Visuo- motor world models for real-world robot manipulation. In International Conference on Robotics and Automation (ICRA), 2024.
Sascha Lange and Martin A. Riedmiller. Deep auto-encoder neural networks in reinforcement leamn ing. The 2010 International Joint Conference on Neural Nenworks (IJCNN) pp. 1-8, 2010. URL https: /taps://appi semanticsche http://coo......oooooooooooo0000
12

---

Published as a conference paper at ICLR 2025
Sascha Lange, Martin Riedmiller, and Ame Voigtländer. Autonomous reinforcement learning raw visual imput data in a real world application. In The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2012. doi: 10.1 10.11091ICNN22211222522.2.
K. G. Larsen and A. Skou. Bisimulation through probabilistic testing (preliminary report). In Proceedings of the 16th ACM SIGPLAN-SIGA Symposium on Principles of Programming Lan- guages, POPL '89, pp. 344-352, New York. NY, USA, 1989, Association for Computing Machin- cry. ISBN 12942.
doi: 10. 10.1145775277.75337
URL https:://6i..rr//10..
7527 75277. 75307
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learninn In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 5639- PMLR, 13-18 Jul 2020. URL hctps: http:sss//pcceeeingge .mlnnnss.rr.....
press/ vvl 19 laskin20a. html
Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic la- tent actor-critie: Deep reinforcement learning with latent variable model. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 741-752. Curran Associates, Inc., 2020. URL https::/proceedddnt F1le/000055552225520000138888883
neurips cct.ppsccccpeee cappaper.ppp..e..eepee halpaapaaaaaaaaalllllallallaeeee filess/paper/2220/
Lihong Li, Thomas J. Walsh, and Michael Littman. Towards unified theory of state abstrac- tion for mdps. In International Symposium Artificial Intelligence and Mathemutics, Al&Math 2006, Fort Lauderdale, Florida, USA, January 4-6, 2006, 2006. umass. edu/ http:///aaaaatt//rrrreeeeee/a///
ime. cs.
pdf
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In Interna- tional Conference on Learning Representations, 2019. URL httph:t//////.. /wtp://eeeeeeeeeeeeeeeee..
Forum?1d=BJJ11225
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski. Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nat., 518(7540):529 533, 2015. doi: 10. N0.103//AATUR114236 URL.
https: ooooorgggll. 1038 http//atuurr2222
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, L. Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Na- ture, 588:604 609, 2019. URL |nttps://apl www.stiescannnaee.mm.. Crg/CorpssI:: 208115222
John Schulman, Sergey Levine Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889- Lille, France, 07-09 Jul 2015. PMLR. URL lattps:// /osass/33///chllaann h
mmrr.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, /td//777.00647, 2017. URL btttt::/dbbp... -niitrierr de/ de/db/ wttpc////oaa/ssooooooo wwt.aaaaaooooooooooooaooooooooct 707 html #
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 387-395, Bejing, China, 22-24 Jun 2014. PMLR.
13

---

Published as a conference paper at ICLR 2025
Richard S. Sutton. Dyna, integrated architecture for learning, planning, and reacting. SIGART Bull., 163: jul 1991. ISSN 0163-5719. doi: 10.1145/1223441122577 URL https:
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy P. Lillierap, and Mar- 2018. URL https:
tin A. Riedmiller. Deepmind control suite. ArXiv, 6315299
//api. sew.aattiiccooollooooooooooo/ooe 6332p2222
Weikang Wan, Yufei Wang. Zackory Erickson, and David Held. Differentiable trajectory op- timization policy class for reinforcement and imitation learning, 2024. URL https:
//openreview. wttp:/oormmmcdd-iSSPPP......
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. In Interna- tional Conference on Learning Representations, 2020. URL hctpp:///ppenrevvver net.
Forum?1d=Hlexf66RR
Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. Ag- gressive driving with model predictive path integral control. In 2016 IEEE International Con- ference Robotics and Automation (ICRA), pp. 1433-144h 1440, 2016. 7487277.
doi: 10.1109/ICRA.20
Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. information-thhorett model predictive control: Theory and applications autonomous driving. IEEE Transactions on Robotics, 34(6): (03))1600-1622) 1622, 2018. doi: 10.1109/TRO.201 20.110999002001.28866999
Denis Yarats, llya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representa- tions, 2021. URL https::
net www.fforrmooo/ Satrummid--Y6 6. IvGa
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https: tttpp:://ppe.eee///// http:///oppeeeieeeeeee net www.:neenweoeeeeeeeeeee.. -____S_____ _ye_s
Amy Zhang, Yuxin Wu, and Joelle Pineau. Natural environment benchmarks for reinforcement learning. ArXiv, 02w/1811.06033 2018. URL. https::/ www.ntissccoooee...m.... org/
(orppusI::550 5333348D0
Amy Zhang, Rowan Thomas McAllister Roberto Calandra, Yarin Gal, and Sergey Levine. Learn ing invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021, hwllhttps://ppeereeeee...r wwwtse/owwwweeeeeomoumu fforum?
(d=-2FCWDKRREU
Yi Zhao, Wenshuai Zhao, Rinu Boney, Juho Kannala, and Joni Pajarinen. Simplified temporal con- sistency reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning. ICML'23. JMLR.org. 2023.
14

---

Published as a conference paper at ICLR 2025
A PROOF AND ANALYSIS
In this section, we provide proof of our statement and some analysis give theoretical difference between BS-MPC and TD-MPC.
A.1 PROOF OF THEOREM S
Our proof uses the following Lemma.
Lemma 1. Assume action a sampled from optimal policy a ~ xt (-(s). With the same assumption in Theorem 2 the difference between R(s, a) and R(o(s), a) under the -i-iimimulation metric has the following upper bound.
(1 (1-c))((s.a) R(((s), a)l s 2c
(14)
Proof. From Theorem 1 the fixed point d satisfies
((s) = (1 -c] a) R(o(s), a)] + cW cWp (Ms,), (1ss),p") (+1s))) Since the second term is always positive, can get
(15)
(1 - R(s.s)) R(o(s), S d(s, o(s))
(16)
Theorem 3. Consider trajectory t (sp: ao. $1, aI....... a in the original state space S, and its corresponding encoded trajectory x(T) (=o, 00; 21, "1.............................. -,, where ag =*(((s)) and 2 o(sk). with & defined in Theorem 2 Under the assumption that both the reward model and dynamics model have no approximation er-
ror, the cumulative rewards S(r) E, (V"V"())) E 7" R(sh, an) and S(((T) =
E, (*"V"(((HH) Em 7x R(6(s)). an) can be bounded as follows.
27 (c L) + (1-7)) c) S(T) - S((())) &
2x(1 y -
(17)
(1 (1-7) c)
Proof. Simply calculating the difference between S(r) and S(p(T))
[S(r) S(o(r)) E, " (
h (R(sh, an) H(o(s)).an))
h=
Er h (V* - V (Vden)))) +E,
A un) - -(((0)))))))
< Er | (V (sH) V* E,
A (R(sh an) ((((s)).
(Triangle inequality)
H
E, ( +E,
Σ h=
h R(sh, an) R(((ss), an)
(Jensen's inequality)
H-1
=, [V* V (o(sn)) + Σ
7
"E, [[[(sh, an) R(((5n), an)l)
h=
(c+
H-1 Σ
h
(From Theorem and Lemma
71 - c)
1 1-c c
hm
2 +))
2c(1 2 H-1
7) c)
(1 (1
(18)
15

---

Published as a conference paper at ICLR 2025
16

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 ALL STATE-BASED TASKS RESULT
acrobol swingup
curipole bulance
curtpole balance spurse
curipole swingup
Episods Retum
1000
500
3M
2M
4M
1M
3M
4M
2M
3M
Step
Step
Scep
Step
cartpole swingup sparse
cheetah run
catch
dog Trot
1000
Episode Return
500
1M
2M
3M
4M
1M
2M
3M
4M
0
IM
2M
3M
4M
2M 4M 6M
8M 10M
Step
Step
Step
Step
dog walk
finger spin
finger turn
finger lurn hard
1000
Episode Recurn
MAN
500
2M
4M
6M SM 10M
0
1M
2M
3M
4M
0
1M
2M
3M
4M
1M
2M
3M
4M
Step
Step
Step
Step
fish swim
hopper hop
hopper stand
humanood
Episade Return
1000
500
1M
2M
3M
4M
1M
2M
3M
4M
1M
2M
3M
4M
2M
4M Step
6M
8M
Step
Step
Scep
humanoid stand
humanoid walk
pendulum swingup
quadruped
Episode Return
1000
0
2M
4M
6M
SM
2M
4M
6M
SM
IM
2M
3M
4M
1M
2M
3M
4M
Step
Step
Step
Step
quadruped walk
reacher easy
rencher hard
walker runl
Episode Retan
1000
0
2M
1M
3M
Step
Step
Step
Step
walker stand
walker walk
Episode Return
1000
500
1M
3M
2M
Step
Step
BS-MPC (Ours)
TD-MPC
SAC
Dreamer- v/
TD-MPC2 (5M)
Figure State-based tasks result from DMControl Suite. Performance comparison on 26 DM Control tasks with state input. At each evaluation step, the episode return is computed over 10 episodes. The results are averaged over 3 seeds, with shaded regions representing the standard deviation.
17

---

Published as a conference paper at ICLR 2025
B.2 COMPUTATIONAL TIME
Table. iments.
ssows the computational time of BS-MPC and TD-MPC. We use RTX-4090 for our exper-
Table 1:
Computational time between BS-MPC and TD-MPC in state-based tasks. The table shows how many hours the training takes.
Cartpole-Swingup
Cheetah-run Finger-Spin
Walker-run
BS-MPC
2.0
4.0
8.3
8.3
TD-MPC
2.4
4.8
10.0
10.1
B.3 DETAILED ANALYSIS FOR THE FAILURE OF TD-MPC
In this section, analyze the reason why TD-MPC failed to achieve the same performance as BS-MPC in our experiments. First, we look at the losses both BS-MPC and TD-MPC in the Homanoodd-Walk environment. Fig 7 shows the average value of each loss over the batches. From this image, it can be observed that the consistency loss in TD-MPC gradually increases and diverges. In contrast, BS-MPC has much smaller values for every component, resulting in more stable perfor- mance. One possible cause is that TD-MPC fails to learn the encoder, leading to large errors, which in turn result in the failure train the latent dynamics model properly. BS-MPC, on the contrary, has explicit encoder loss in its objective function, thus enabling it to actively update the encoder. Fig N shows the learned Q values and gradient norm of the objective function. As shows, TD-MPC vulnerable to exploding gradients, which leads to the divergence of the loss. Moreover, the learned Q values drop significantly when the gradient norm explodes.
consistency loss
reward loss
value loss
encoder loss
total loss
10000
1.0
10
10000
8000
0.8
7500
5000
4000
0.4
4
2500
2
0
0.0
0
0
0
Step
Step
Step
Step
Step
BP-MPC (Ours)
1D-MPC
Figure Comparative analysis of loss components between BS-MPC TD-MPC across training steps in Humanoid-we walk environment. Each graph presents different loss types consistency loss, reward loss, value loss, encoder loss, and total loss plotted against training steps. Note that TD-MPC does not have encoder loss, and it only exists MP-M See Eq. g for more details.
grad norm
200
200000
100
100000
0
Step
Step
BS-MPC (Ours)
TD-MPC
Figure 8: Average values of the learned Q functions and gradient norm of the loss function between BS-MPC and TD-MPC across training steps Humanoid-walk environment.
18

---

Published as a conference paper at ICLR 2025
C BS-MPC TRAINING ALGORITHM FLOW
In this section, we describe the algorithm flow of BS-MPC and compare it with TD-MPC.
C.1
ALGORITHM FLOW
The training algorithm flow is described in Algorithm. !
Algorithm BS-MPC (Model training)
Require: [ A 0², 8d), the randomly initialized network parameters, Episode Length L, Number of parameter update K, Buffer B
1: while the training is not complete do
2: // Collect episode
3: for = O... do
4:
a4 BSMPC(hga (s:)) (Compute action with BS-MPC)
5:
($1+1.T1) ~ P(st, ae), ((s,, az) {Execute action against the environment] B+BU (st, a2 Tt, 1141) (Add to buffer)
6:
7:
end for
8:
// Update model parameters
9:
do 0 {Initialize Oo with current parameter)
10:
for k = -..KK do
11:
{8t:+ 01:4 TE++H) ~ B (Sample trajectory with horizon H from the buffer B} 2xt-H+1 = hap (+ (Encode all observations with online encoder)
12:
13:
FE = Rog (2ta+ H. Om (Estimated rewards]
14:
Qu++H = Qog (Z10H a.+h)) (Estimated state-action value}
15:
1++1++ 1+1 = dog (Ze:4 (1)+4) (Estimated next latent state]
16:
DR+1 = arg mino, C(02) {Update 0g by minimizing Eq.
Uk+1 arg ming' Ix (+t) {Update v by minimizing Eq.
18:
end for
8 0x (Update current parameter)
20:
end while
C.2 COMPARISON WITH TD-MPC
As discussed in Section BS-MPC facilitates parallel computation. Algorithm 2 outlines the calculation flow of TD-MPC. While TD-MPC shares many similarities with BS-MPC, the primary distinction lies in how computes the estimated values and the model cost L.
In BS-MPC, the observed state variables St:t+ over H steps are first projected into a sequence of latent states H. Subsequently, the rewards, state-action values, and predicted next states for these H latent states are computed collectively. Since all calculations are performed simultaneously across the steps, parallel computation effectively utilized, resulting in high computational effi- ciency. This computation process is detailed in Lines 12 to 15 of Algorithm
Conversely, in TD-MPC, only the initial state 8; is encoded into the latent state 2, (see Line 12 of Al- gorithm 2). and the subsequent latent states & computed sequentially using the latent dynam- ics. As a result, TD-MPC requires sequential computation when calculating rewards, state-action values, and the cost L, which limits its ability leverage parallel computation. This sequential computation process is described in Lines 14 to 19 of Algorithm 2
In summary, BS-MPC obtains the sequence of latent states 26:t-H by encoding the entire sequence of observed states $ta+ using the encoder hig. In contrast, TD-MPC encodes only the initial state into and derives the remaining latent states 2441:++ sequentially using the latent dynamics
19

---

Published as a conference paper at ICLR 2025
dgg. Therefore, while BS-MPC enjoys parallel computations to speed up its calculation, TD-MPC suffers from the bottleneck of the sequential computation in the cost calculation.
Algorithm TD-MPC (Model training)
Require: [0*, 0 v:: randomly initialized network parameters,
Episode Length L, Number of parameter update K, Buffer B
1: while the training is not complete do
2: // Collect episode
3: for = O... do
4:
Ot ~ TDMPC(hga (xt)) (Compute action with TD-MPC)
5:
(s)+ ~ P(st, 0z), R(s,,, az) (Execute action against the environment) B B (st, az, Tt, St+1) {Add to buffer]
6:
7:
end for
8:
// Update model parameters
9:
Bu {Initialize Og with current parameter)
10:
for k = 000k= 10...K do
M++1 1 01:4+ Tu++ 11) } B (Sample trajectory with horizon H from the buffer B)
12:
& = haz (8t) {Encode the initial observation with online encoder) C 0 {Initialize the cost}
13:
14:
for t...t + H do do
15:
f. Rog a;) (Estimated rewards]
: (2)(a (Estimated state-action value}
17:
+ døg (2,, a.) (Estimated next latent state)
18:
L + L + X (2(4111, fs, a.) {Add to the cost]
19:
end for
20:
0x+1 = arg mino, C(O%) {Update 0g by minimizing Eq,
21:
thk+1 arg min (=(x)) {Update uh by minimizing Eq.
22:
end for
0 0K41 {Update current parameter)
24:
end while
D IMPLEMENTATION DETAILS
Here we give details about the
and model architectures.
D.1 HYPERPARAMETERS
Shared Parameters: First, outline the parameters that are common to both TD-MPC and BS- MPC. They are described in Table.
BS-MPC specific parameters: Next, we list the parameter that is used for tuning the weight for bisimulation metric loss (c1). We change the value based on the environment and tune the weighting coefficient сд across 10 0.0001, 0.001, 0.01, 0.1, 0.5 with grid search. All of the numbers
listed Table. 3
D.2 MODEL ARCHITECTURE
In our experiments, both BS-MPC and TD-MPC utilize the same model architecture and the number of trainable parameters. We employ multi-layer perceptrons (MIPs) to represent the underlying environment models P and R, the state-action value function Q, and the policy æ. The architecture details are shown in Table. 4 More details can be found in our official code.
20

---

Published as a conference paper at ICLR 2025
Table 2: Hyperparameters used for TD-MPC and BS-MPC in the experiment.
Hyperparameter
Value
Discount factor (?)
0.99
Seed steps
5,000
Replay buffer size
1,000,000 (state-based tasks)
100,000 (image-based tasks)
Sampling technique
Uniform Sampling
Planning horizon (!t)
Initial parameters (4" o0)
(0, 2)
Population size
512
Elite fraction
64
MPPI Update Iterations
12 (Humanoid, Dog)
(otherwise)
Policy fraction
5%
Number of particles
Temperature (r) Latent dimension
0.5
100 (Humanoid, Dog)
50 (otherwise)
Learning rate
3e-4 (pixels)
1c-3 (otherwise)
Optimizer (0)
Adam (31 = 0.9, 82 = 0.999)
Temporal coefficient (A)
0.5
Reward loss coefficient (c1)
0.5
Value loss coefficient (c2)
0.1
Consistency loss coefficient (es)
0.5
Exploration schedule (c)
0.5 0.05 (25k steps)
Planning horizon schedule
1 5 (25k steps)
Batch size
512 (State-based tasks)
256 (Image-based tasks)
Momentum coefficient (C)
0.99
Steps per gradient update Target parameter 0 update frequency 2
Table 3: Bisimulation metric parameter used in the experiment.
Environment
Value (c4)
Acrobot
0.0001
Cartpole
Cheetah
0.001
Cup
0.5
Finger
0.001
Fish
0.001
Hopper
0.1
Humanoid
0.001
Pendulum
0.01
Quadruped
0.1
Reacher
0.01
Walker
0.001
Dog
10-8
21

---

Published as a conference paper at ICLR 2025
Table 4: Model Architecture used in the experiment.
Models
Number of Layers
Hidden Dim
Activation
Latent Model Dynamics P
3
512
ELU
Reward Model R
3
512
ELU
State-action value function Q
3
512
ELU LayerNorm
Policy π
3
512
ELU
22


Reviews:
[{"review_id": "ggQrsRBRSf", "reviewer": "Reviewer_w1co", "summary": "This paper presents a new method for model-based reinforcement learning (MBRL) called BS-MPC. The key innovation lies in incorporating a bisimulation metric loss into the objective function to improve encoder stability, robustness to noise, and computational efficiency. By using the bisimulation metric, BS-MPC aims to ensure behavioral equivalence in the latent space, maintaining key characteristics of the original state space. The method is benchmarked against the Temporal Difference Model Predictive Control (TD-MPC) and other model-free and model-based methods on various tasks, showing superior stability and resilience to noise.", "questions": "Could the authors expand on the sensitivity of BS-MPC to the parameter c4 and potential ways to reduce this dependency?\n\nHow does BS-MPC perform in scenarios with dynamic backgrounds that align with the movement instead of pure noise?\n\nAre there additional computational costs associated with bisimulation metric loss, especially in high-dimensional latent spaces?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper provides a new perspective by integrating the bisimulation metric to address known challenges in MBRL, particularly around stability and robustness to noise. The experimental results demonstrate how BS-MPC performs well in both state-based and image-based tasks, showing increased resilience to noise and achieving faster training times due to parallel computation. The theoretical analysis adds depth by bounding cumulative rewards in the learned latent space, suggesting that BS-MPC retains meaningful state information effectively.", "weaknesses": "While the theoretical foundations are thorough, certain explanations, particularly on encoder stability and noise resilience, could be made clearer to broaden accessibility. The parameters require extensive tuning, which may be impractical for real-world applications lacking automated parameter selection. Additionally, the approach to introducing perturbations, particularly with visual distractions, doesn’t seem entirely effective. It would be beneficial to test perturbations that are more representative of realistic environmental changes, which could better showcase BS-MPC’s resilience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GYc3LRf9Wm", "reviewer": "Reviewer_CXNw", "summary": "This paper proposes BS-MPC, a model-based reinforcement learning approach that introduces bisimulation metrics (loss) on top of TD-MPC. Compare to TD-MPC, BS-MPC has an explicit encoder loss term, the adaptation of bisimulation metric, and parallelizing the BS loss. The authors found that their approach can improve training stability, robustness against input noise, and computation efficiency, which is validated on a set of simulation environments.", "questions": "When you say BS-MPC improves computation efficiency, what does it mean? Is it compared to TD-MPC?\nIt is surprising to me because BS-MPC has one additional loss term compared to TD-MPC and why is BS-MPC faster to run? \n\nWith the above question, I'd like to know the latency overhead of BS loss term in the training.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. The overall presentation is good. The approach is sound and makes sense to the reviewer. The experimental results look promising, compared to TD-MPC.", "weaknesses": "However, the major weakness is its novelty. \n1. The whole framework is based on TD-MPC. The difference is the authors introduce the Bisimulation metric and its corresponding loss design, which are from the existing literature, as stated in the paper. \n2. It is also a common way to introduce additional regularization loss terms for the encoder of model-based RL. \n3. The theoretical analysis mainly borrows from the existing work and does not have any major significant result. It would be great if the authors could provide \"Under the BS loss training error, what's the performance gap between the final converged policy by their approach and the ideally optimal policy\", and \"Theoretically, how much performance gain could their approach improve, compared to TD-MPC.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "21oVCAFWxF", "reviewer": "Reviewer_hjrQ", "summary": "This paper considers model based reinforcement learning and proposes bisimulation metric to improve over temporal differential MPC method. The authors show theoretical analysis of the expected cumulative rewards in the latent space, and empirically demonstrate enhancement over TD-MPC and other baselines on several continuous control tasks.", "questions": "“We assume that the learned policy in BS-MPC continuously improves throughout training and eventually converges to the optimal policy π∗, which supports Theorem 1.”\nThis seems to be a very strong assumption. For example, by looking at the training curve, the return does not improve monotonically, and we have no information about if the learned policy is converging to the optimal policy. How do you explain such a strong assumption? Is it possible to remove it for the theoretical results?\n\nIn Fig. 4, why do all non-MPC based methods only have results till 10M steps?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written and well presented. The proposed bisimulation metric seems to work well on the experiments considered, compared to TD-MPC and other baselines. The supplementary sections are comprehensive.", "weaknesses": "The novelty of the paper seems ambiguous. It seems that both on-policy bisimulation and TD-MPC methods are well studied for model based RL, and the authors plug bisimulation into TD-MPC. \n\nThere are several typos in the paper.\n“In BS-MPC, the latent dynamics are modeled using an MLP. We also model the latent dynamics model with an MLP” I believe BS-MPC should be TD-MPC.\n“we sample M action sets from Gaussian distribution N (μ0, σ0) based on the initial meanμ0 and standard deviation σ0” Missing spacing between mean and \\mu_0", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N3v9RuT4mp", "reviewer": "Reviewer_seSt", "summary": "This paper is concerned with a new, model-based reinforcement learning method, which utilizes bi-simulation metric. This formulation helps with the stability of training and helps with the robustness of the controller. General idea is that they seek to find states that \"behave\" similarly, and intuition behind it is that one can use similar control input for similar states, which simplifies the controller and make it more interpretable. Authors learn an encoder which maps states of the environment to another domain, in which similar states are identified, and are mapped to the same representation (roughly speaking). Then, this representation is utilized to train a controller. Novelty of this work is to add the encoder loss directly into the training procedure.", "questions": "1- Based on your case studies, your method does not seem to change the episode return that much, except for a few cases like dog walk, dog trot and humanoid walk. In your Appendix, you provide a rough explanation of why that may be. Looking at Figure 7 and 8, it appears that loss, and consequently the gradient, explode (in TD-MPC), however, in RL, gradient clipping is used to tackle this issue. When you compared your method to TD-MPC, did you employ gradient clipping for it or not? It does not appear to be a fair comparison if you didn't, and perhaps that is why your method did not do significantly better in other case studies; as loss did not *explode*.  \n\n2- I suggest you revise the experiments' section, and run all case studies on TD-MPC2 rather than TD-MPC. I realize it is touched upon in appendix D, however since TD-MPC2 is the updated version, I suspect it would make for a more fair comparison. Moreover, adding a thorough comparison would certainly present your method better; between training time, sample complexity, number of parameters used, and hyper parameter tuning and different configurations; it will strengthen your case if you could show how it might fail. I would also like to know the rationale behind using TD-MPC in the main body and mentioning TD-MPC2 in the appendix.  \nTo the best of my knowledge, TD-MPC2 can have many parameters, since it can be used on different domains. Thus, it is not an apple to apple comparison, unless it is specifically mentioned in the paper.\n\n\n3- Is there any theoretical results on why your method requires less parameters, and converges faster? or is it mainly based off of experiments? Since Theorem 3 only offers an upper bound for expected cumulative rewards for the optimal policy. \n\n4- I suggest a more rigorous approach for robustness, such as comparing the Lipschitz constant of your controller to TD-MPC's.\n\n5- I believe if you can theoretically confirm in which case studies your method is going to perform better than TD-MPC, it will strengthen your paper significantly.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Paper is well written, all ideas are clearly explained. Overall, paper is mathematically rigorous. Authors do a good job in walking the reader through the preliminaries, highlighting distinctions, and presenting their work.\n\nFurthermore, paper presents more than 20 case studies, which helps immensely in comparing their performance to the state of the art methods.", "weaknesses": "Improvements and contributions seem incremental, and overall not that beneficial according to the case studies (Figure 6), one only sees improvements in few case studies (such as humanoid walk, dog walk and trot), and basically identical to TD-MPC in others (such as humanoid stand, pendulum, cheetah). \nThe only major contribution is adding the bi-simulation metric loss to the loss function, the other two contribution naturally follow from this addition. \n\nAs authors have mentioned, the hyper parameters play a huge role, and one wonders how much time is needed to tune these parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)