RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Authors: Yushuai Li
Hengyu Liu
Torben Bach Pedersen
Yuqiang He
Kim Guldstrand Larsen
Lu Chen
Christian S. Jensen
Jiachen Xu
TIANYI LI
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement learning
state perturbation
MuZero
deep learning
Primary area: reinforcement_learning->deep_rl

Abstract: The MuZero reinforcement learning method has achieved superhuman performance at games, and advances that enable MuZero to contend with complex actions now enable use of MuZero-class methods in real-world decision-making applications. However, some real-world applications are susceptible to state perturbations caused by malicious attacks and noisy sensors. To enhance the robustness of MuZero-class methods to state perturbations, we propose RobustZero, the first MuZero-class method that is $\underline{robust}$ to worst-case and random-case state perturbations, with $\underline{zero}$  prior knowledge of the environment’s dynamics. We present a training framework for RobustZero that features a self-supervised representation network, targeting the generation of a consistent initial hidden state, which is key to obtain consistent policies before and after state perturbations, and it features a unique loss function that facilitates robustness. We present an adaptive adjustment mechanism to enable model update, enhancing robustness to both worst-case and random-case state perturbations. Experiments on two classical control environments, three energy system environments, three transportation environments, and four Mujoco environments demonstrate that RobustZero can outperform state-of-the-art methods at defending against state perturbations.

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Yushuai Li!
Hengyu Liu
Torben Bach Pedersen Christian S. Jensen
Yuqiang He Kim Guldstrand Larsen
Lu Chen
Jiachen Xu Tianyi Li!
Abstract
transportation (Sun et al. 2023), and healthcare (Hao et al. 2022). In DRI., an agent interacts repeatedly with an environment: The agent receives a state update, or an ob- servation, along with a reward; in response, the agent takes an action that in turn yields a new state update and reward. The objective of DRL is achieve an agent that is able to optimize the cumulative reward. Model-free DRL has the strength of straightforwardly estimating the optimal policy or value that is used to evaluate the policy, without knowing the environment's 's dynamics. However, model-free DRL suffers from low sample efficiency and lacks sophisticated look-ahead capabilities. Model-based DRL is capable of addressing these weaknesses. However, most model-based DRL. methods, e-g., AlphaGo (Silver al., 2016) and AI- phaZero (Silver et al. 2017), require prior knowledge of the environment's dynamics, which restricts their applicabil- ity. To relax this requirement, MuZero (Schrittwieser et al., 2020) learns an abstract environment model in combina- tion with Monte Carlo tree search (MCTS). In this sense, MuZero has the strengths of model-free and model-based DRL and therefore has attracted broad attention (Hubert et al.
2022)
The MuZero reinforcement learning method has achieved superhuman performance at games, and advances that enable MuZero to contend with complex actions now enable use of MuZero-class methods in real-world decision- decision-making applica- tions. However, some real-world applications are susceptible to state perturbations caused by ma- licious attacks and noisy sensors. To enhance the robustness of MuZero-class methods to state perturbations, we propose RobustZero, the first MuZero-class method that is robust to worst-case and random-case state perturbations, with zero prior knowledge of the environment's dynamics. We present training framework for RobustZero that features self-supervised representation net- work, targeting the generation of consistent ini- tial hidden state, which is key to obtain consistent policies before and after state perturbations, and it features a unique loss function that facilitates robustness. We present an adaptive adjustment mechanism enable model update, enhancing robustness to both worst-case and random-case state perturbations. Experiments on two classical control environments, three energy system envi- ronments, three transportation environments, and four Mujoco environments demonstrate that Ro- bustZero can outperform state-of-th aoffthheart art methods at defending against state perturbations.
2021
Ye et al.
2021 Danihelka al.) 2022 Nuu
et al.
2024 Xuan et al.
2024)
The superhuman performance achieved by MuZero-class methods assumes real and unperturbed states. However, in real-world applications, the observed states may undergo imperceptible perturbations. Even small "reality gap" can compound errors in the predicted abstract states within MuZero-class methods, causing reduced rewards and po- tentially harmful decision-making]] To date, no studies have been reported that address this problem. To improve robustness, we address two challenges.
1. Introduction
Deep reinforcement learning (DRL) has achieved notable successes in numerous domains, c.g., energy (Wang et al.
Challenge I: How to achieve robust training for MuZero- class methods under state perturbations? The current ro- bust training proposals for DRL focus mainly designing strategies to: 1) minimize balanced nominal and adversarial losses (Pinto et al. 2017 Zhang et al., 2020a Oikarinen et al. 2021 Liang et al. 2022): 2) minimize the cross- entropy/KLL-divergemn policy or value loss between ac-
'Department of Computer Science, Aalborg University. Aal- borg, Demmark "School of Computer, Electronics and Information, Guangxi University, Nanning, China *College of Computer Sci- ence, Zhejiang University, Hangzhou, China. Correspondence to: Tianyi Li <tianyi @ as. dk>, Hengyu Liu cheli@es.aau. cak..u.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada, PMLR 267, 2025. Copyright 2025 by the author(s).
AAppenii A provides an example to illustrate the importance of considering state perturbations.

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
tions taken under non-perturbed states and perturbed states
learned models converge to flat minima and escape from steep minima, which further enhances the robustness un- seen worst-case and random-case state perturbations. In summary, our main contributions are as follows.
(Zhang et al.
2020a
Sun et al. 2022 Zhang et al., 2021
Liang et al.
2022 Zhou et al. 2024 Liu et al. 2024 Dong
et al. 2025). These strategies are suitable for DRL methods
that use neural networks (NNs) to approximate a policy or
We propose RobustZero, a novel robust reinforcement learning framework. We believe this is the first MuZero- class method to defend against state perturbations.
value function, such as deep Q-learning (Mnih et al. 2013).
deep deterministic policy gradient (DDPG) (Timothy et al.
2015). and proximal policy optimization (PPO) (Engstrom lt al. 2020). However, MuZero-class DRL methods differ in that they employ MCTS with learned model to derive policy instead of using NNs directly. This calls for a new robust training method to counteract state perturbations.
We propose a self-supervised representation network to facilitate the generation of consistent policies; and based on this, we construct loss function for robust training.
We propose a new adaptive adjustment mechanism that enables RobustZero to generate policies that are highly robust to both worst-case and random-case perturbations.
Challenge II: How to achieve high robustness for MuZero- class methods to worst-case and random-case state per- turbations? Worst-case state perturbations yield states that minimize cumulative rewards, while random-case state perturbations imply that perturbations occur randomly in states. Most existing robust DRL methods (Pinto et al.
Extensive experiments on two classical control environ- ments, three energy system environments, three trans- portation environments, and four Mujoco environments show that RobustZero is able to outperform the state-of- ahe methods in terms of robustness to worst-case and random-case state perturbations.
2017; Zhang et al. 2020a; Oikarinen et al. 2021; Russo
& Prouticre 2021
Liang et al.
2022
Zhang et al.
2021
Sun et al.) 2022 Zhou et al.,
2024; Zhihe & Xu
2023
Sun & Zheng, 2024: Dong et al. 2025) investigate defense
2. Related Work
strategies for worst-case state perturbations, but these meth- ods often suffer from reduced performance when dealing with random-cass state perturbations that are prevalent in real-world environments, where sensors produce noisy sam- ples (Zamora et al., 2017). Further, learning a worst-case perturbation policy is complex and challenging (Liu et al. 2024). whereas implementing a random- case perturbation policy is relatively straightforward. To address this issue, recent study (Liu et al. 2024) proposes adaptive defense mechanism that achieves high robustness to both worst-case and random-case state perturbations. However, this mecha- nism cannot be applied to MuZero--lass methods because its policy search (Liu et al., 2024) is gradient-based, while MuZero-class methods use gradient-free MCTS for policy search. A new adaptive mechanism is needed for MuZero- class methods.
2.1. DRL with MCTS
Many DRL. methods have been proposed, primarily us- ing temporal difference learning (Mnih et al.) 2015) and gradient- based policy search (Han et al. 2023). In contrast, AlphaGo (Silver al., 2016). a different DRL method, in- tegrates MCTS the policy improvement operator with NNs to approximate value and policy networks. Due to the strong look-ahead search capability of MCTS, AlphaGo achieves superhuman performance in numerous games. Sub- sequently, several DRL methods incorporating MCTS have emerged, including MuZero (Schrittwieser et al. 2020) that can learn abstract enviromment models without knowl- edge of underlying dynamics. Diverse MuZero-class meth- ods have been proposed, addressing aspects such as lim- ited action access (Danihelka et al. 2022). complex action spaces (Hubert et al. 2021). limited data (Ye et al. 2021). and time consumption (Xuan et al.) 2024). Of these, sam- pled MuZero (S-MuZero) (Hubert et al., 202T) supports complex action spaces, which is important in real-world decision-making We extend S-MuZero to achieve robust- ness to state perturbations, an aspect overlooked by existing MuZero-class methods.
To tackle these challenges, we propose RobustZero, novel robust training framework for MuZero-class methods that achieves high robustness to both worst-case and random- case state perturbations. First, by integrating contrastive learning, self-supervised representation network is devel- oped generate similar initial hidden states that are mapped from unperturbed states and perturbed states. This enables planning results made in the abstract hidden space with and without perturbations to be consistent. Second, following the self--uupervieed representation network, we propose an overall loss function that promotes robust training. Third, we design an adaptive adjustment mechanism that initially prioritizes learning from worst-case perturbations and then gradually extends to learn from random-cass perturbations on the fly. The adaptive adjustment mechanism enables
2.2. Robust Reinforcement Learning
DRL robustness may relate to multiple aspects, e.g., action
perturbations (Tessler et al. 2019 Bukharin et al.
2024
state perturbations (Pinto et al. 2017 Zhang et al.
2020a
Oikarinen et al.] 2021 Russo & Proutiere 2021
Zhang
et 2021 Sun ct al.
2022: Liang et al. 2022 Zhou ct al.
2024
Liu et al. 2024 Zhihe & Xu 2023 Sun ZZheg
2

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
2025), reward corruptions Zhang al. ). and environment dis- 2020 Huang et al.) (022). We target DRL robustness to state perturbations. A robust ad- versarial reinforcement learning method is proposed (Pinto) |l 2017) to jointly train an agent and an adversary, where
2024
Dong et al.
one notable work is EfficientZero (Ye al. 202 which significantly improves the sample efficiency of the MuZero method while maintaining superior performance. It achieves this by using contrastive representation learning to build a consistent environment model and by using the learned model to correct off-policy value targets. Different from these studies, we aim to leverage contrastive representation learning to improve the robustness of MuZero-class methods to state perturbations.
2020b Eysenbach & Levine
crepancies (Sinha et al.
ctt at the agent aims to accomplish the primary task objectives while learning to remain robust against disturbances intro- duced by the adversary. Zhang et al. (Zhang al.] (020a) offer a foundational theory that equates learning an optimal adversary to finding an optimal policy within a new Markov Decision Process (MDP). They propose a principled policy regularization method to defend against state perturbations. Following this work, the alternating training with learned adversaries (ATLA)-PPO (Zhang et al. (02T) and the Policy Adversarial (PA)-ATLA-PPO (Sun et al. 2022) are designed for launching worst-case perturbation policies under black- box and white-box attack scenarios, respectively. With similar idea of formulating the design of an optimal adver- sary as the solution to another MDP, a black-box attack strategy based on DDPG (BA-DDPG) is proposed (Russo & Proutiere 202T) to derive the worst-case perturbation policy. We consider black-box attacks, where the adversary has no knowledge of the policy and model of the agent. A recent study (Dong et al. 2025) proposes the use of variational opti- mization over worst-case adversary distributions, rather than a single adversary, and trains an agent to maximize the lower quantile of returns to mitigate over-optimism. As a different branch, several studies (Zhihe& & Xu 2023] Sun & Zheng 2024) introduce the diffusion models enhance robustness to state perturbations. Specially, a diffusion model-based predictor is proposed (Zhihe& & Xu 2023) for offline RL to recover the actual states against state perturbations. A belief- enriched pessimistic Q-learning method is proposed (Sun & Zheng 2024) by using diffusion model to purify observed states. We note that the studies covered here (Pinto et al..
3. Preliminaries
3.1. Markoy Decision Process (MDP)
Definition 3.1. An MDP is a 4-tupe M = (S, A. R. po), where S is a state space, A is an action space, P a deterministic transition function, R is a reward function, and po is the initial state distribution.
An MDP is used widely to describe and model an environ- ment in DRL. An agent observes a current state 8 E S. Driven by a policy, the agent takes an action a++1 E A and then receives an updated state 8t+1 and reward
T++1 = R(se, 0e+1). For a sequential decision-making task (e.g.. voltage control), the goal is to maximize the expected return Ei=o Yt,, where 7 is the discount factor at time t.
3.2. MuZero
MuZero learns an abstract MDP to represent an abstract environment model and employs MCTS for planning. Note that planning occurs in the abstract MDP based on abstract states instead of real states. To avoid the misuse of symbols, we use to represent the time step the real world. Planning is made at each time step We use k- = 0, ..., K to represent the index of unrolled steps in the abstract MDP. Next, we summarize the components of the abstract MDP: 1) a repre- sentation network Ho that maps real state to an abstract and initial state sy denoted as $ == Ho(o:); 2) a dynamics network Go that maps a previous abstract state s and a candidate action at to an immediate reward r and new abstract state st denoted as rf == Go aaf) and 3) prediction network Fo that maps sy to a policy pt and value function off denoted as pf. of := Fo())), where 0 denotes the NN parameters of the representation, dynamics, and prediction networks. By using the abstract MDP, MCTS obtains a search policy n; at each time step to determine the real action a++1. Then, the real environment receives a++1 and further provides a new state 0141 and a real re- ward Me+1. By employing value equivalence, the training objective of MuZero is enable achieving P zs Te+k- of RJ Ze+t, and r R Pe+k, where Z++h is the sampled return. The sequence of real actions from the sampled tra- jectory is used to compute r In this way, the planning results made in the abstract MDP are equivalent to those in the real environment.
2017 Zhang et al. 2020aa Oikarinen et al.) [021] Russo &
Proutiere 2021
Liang al.
2022
Zhang al. 2021 Sun
et
al.
2022
Zhou et al.
2024
Zhihe & Xu, 2023 Sun &
Zheng 2024 Dong et al.
2025 target worst-case state per-
turbations. Recently, the most related work (Liu et al. (2024) proposes an new method, called PROTECTED, that outper- forms the state-of-the-art methods under both worst-case and random-case state perturbations. However, as discussed above, this method employs gradient-based policy search and is inapplicable in our setting.
2.3. Contrastive Representation Learning
The contrastive representation learning has been used to improve the sample efficiency (Schwarzer et al. 2021
Ye et al. )02T) and enhance the representation ability of states Nasiriany et al., 2019), rewards (Kang et al. 2023),
and value functions (Eysenbach et al. 2022). Among them,
3

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
3.3. S-MuZero
real environment, without relying on the actual dynamics of the environment. Unlike MuZero, the additional and unique objective of RobustZero is to generate consistent planning results both without and with state perturbations restricted by a budget constraint. This aims to enable satisfactory planning results even when the agent receives perturbed states, resulting in enhanced robustness of the method.
The traditional MuZero method is only suitable for low- dimensional and discrete action spaces. By combing the sample strategy and MCTS, S-MuZero (Hubert et al.) (021) enables planning in complex action spaces, including con- tinuous and high-dimensional discrete spaces. For ease of distinction, we name the policy evaluation and im- provement in S-MuZero as sampled MCTS, denoted as s-MCTS(s) IC), where K is the number of sampled ac- tions. We employ sampled MCTS to obtain the search policy nH + s-MCTS(s)) 0, K), where Te is also called the sample-based improved policy (Hubert et al.) (021))
5. RobustZero
The RobustZero framework is shown in Fig. It encom- passes three parts: parallel data collection under state per- turbations, self-supervised representation network and loss function, and adaptive adjustment during training.
4. Problem Statement
5.1. Parallel Data Collection under State Perturbations
4.1. State Perturbations
To increase the efficiency of data collection, RobustZero operates in two parallel environments, Environment 1 and Environment 2, under worst-case and random-case state perturbations, respectively. Thus, two replay buffers, worst- case buffer Dwe and random-cant cass buffer PRC. are used to store different data. The details of the parallel data collection are provided in Appendix [B.1
An adversary or attacker aims to launch state perturbations to reduce the accumulated reward gained by an agent. We consider the black-box attack and assume that the adver- sary cannot change the MDP, The adversary can then only observe and perturb states before they reach the agent. To obtain general robust-defense performance, we consider both worst-case and random-case state perturbations. Next, each perturbed state is subject to budget constraint (Oikari- 2024 Zhang al. 2020a 2021 Liu et al. 2024 Sun et al.
5.2. Self-supervised Representation Network and Loss Function
nen et al. 202T Liang et al., 2022
Zhou et al.
2022). which
To obtain similar planning results with and without state perturbations, we propose a self-supervised representation network that replaces MuZero's 's original representation net- work. Then, we define a new loss function-encompassee a worst-case loss term, a random-case loss term, and a decay term-to guide the training of the learned model.
limits the power of the adversary.
Definition 4.1. The budget constraint (o.) denotes Ip norm ball centered at o with radius
For clarity, we use superscripts "*" and "-"""" to represent the relevant symbols and variables under worst-case and random-case state perturbations, respectively. For instance, E and & denote data at time t from a trajectory generated under worst-case and random perturbations, respectively, and or and og denote the perturbed states after worst-case and random-case state perturbations, respectively. Next, we formalize the worst-case and random-case perturbation policies.
a) Model of self-supervised representation network: Re- calling the structure of MuZero-like methods, the represen- tation network serves the interface between the real-world states and the abstract hidden states. Taking states before and after perturbations, i.e., Or and oz. as input to the rep- resentation network, we can get the corresponding roots of the search trees, i.e., of <- Ho( and By +- Ho( If the encoded root nodes are the same, the final policies generated by the sampled MCTS will also remain the same. In this sense, the agent is capable of defending against state perturbations well.
Definition 4.2. A worst-case perturbation policy is defined by a function Aworst that maps OF to oF == Aworst (o;) E (oz). with the objective of finding an opti- mal policy Worst that minimizes Ei=o 7z7..
With this inspiration, we include a self-supervised repre- sentation network that leverages contrastive learning in the framework. The network contains two branches. The first branch contains representation network and a projector network without gradients. The second branch contains the same representation network, the same projector network, and an additional predictor network. We use á to represent the NN parameters of the projector and the predictor. Next, we define the notions of projector and predictor. Definition 5.1. A projector P1 transforms an initial hidden
Definition 4.3. A random perturbation policy is defined by a function Avandom that maps o; to random state o; := Arandom (oz) E (or))).
4.2. Objective of RobustZero
Similar to MuZero, RobustZero aims to learn an abstract MDP that, by employing MCTS, produces a planned policy, reward, and value. These planning results are capable of aligning with those of the policy, reward, and value in the
4

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Parallel data collection under state perturbations
Self-supervised representation and function
H
P
F2,
Launch worst-case perturbations
i
Find search policy x
0,
Representation
Projector
Predictor
Contrastive loss
Adversary 1
Agent
O;
4441
H4 Representation
M Projector
Stop-grad
0,
O
Environment
/+1
Worst-case buffer: Dy
1
Decay term
Random-case loss term
Worst-case loss term
Launch random-case perturbations
Find search policy x,
Loss function
Adversary
Agent
wl
0
Worst-case
Random-case
Flat minima
Adaptive
Environment
a c-
Rundom-cuse buffer:
w2,
Steep minima
Adaptive adjustment during training
Figure Overview the RobustZero framework.
state to a d-dimensional vector.
perturbation by using the encoded root node before pertur- bation to improve the representation network. The goal to achieve maximum similarity between the two branches. This enables the improved representation network to pull 8p (sh) and of (s) close to each other, resulting in enhanced robustness to state perturbations.
Definition 5.2. A predictor P2g transforms the output of the projector to another d-dimensional vector.
The key functionality of the projector is to map high- dimensional features from the encoder to a lower- dimensional space, making the learned features more suit- able for the subsequent contrastive loss computation. Next, the key functionality of the predictor is to transform pro- jected features stabilize optimization and reduce col- lapsing solutions. Specifically, if we omit the predictor and directly enforce similarity between the outputs of two branches (c.g., using mean squared error cosine similar- ity). the model may exploit a shortcut by collapsing to constant output vector. In such a scenario, since all inputs produce the same output, the loss is minimized regardless of input variation, resulting in model collapse. When collapse occurs, all states regardless of whether they are perturbed not-are mapped to the same very similar initial hidden state, effectively losing the ability distinguish between different inputs. To avoid model collapse, one effective strategy is to introduce an asymmetric network architecture, where a predictor added to one branch, while the other branch remains without and has its gradient flow stopped. This asymmetry prevents both branches from trivially con- verging the same constant output. The branch with the predictor learns align its output with the target branch, which acts as stable reference point. Because the target branch does not receive gradients, it cannot adjust itself to match the predictor's potentially trivial solution, thus breaking the symmetry that often leads to collapse. With this model, we can supervise the encoded root node after
b) Contrastive loss: To achieve the objective of the self- supervised representation network, define worst-case and random-case contrastive loss functions:
P2s(P1
sg(PLo(s7)
Ener EB-Dm
[P2
| Pl (s?(()
(2.((1)(3))
sg(PLo(57))
Be. - EB-DAC
(P 1sg P1
where || ||2 is the Iz-norm; sg(-) )) is the stop-gratd -rpdggra opera-
tion; and B is the batch.
c) Overall loss function: has three weighted terms: worst-case loss term, ii) random-case loss term, and iii) decay term. The first of these is composed of the original loss function and the contrastive loss function:
The loss function of RobustZero
K
K KL'
È ED
Σ
T' (TELK pb) + c1 Σ k=
(
km
+ 02
7 (Zites o
+ tcatan
(3)
5

---

The settings of l", KLP, and 1" are the same as their coun- terparts in S-MuZero, with the objectives of minimizing the errors between the predicted policy 2 and the real search policy Ti+ between the predicted value हे and the target and the RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
where cl, c2, and c3 are loss coefficients; I" and 1 are the Cross-Entropy loss functions for reward and value, re- spectively; and KLP is the KL-divergence loss function for policy.
where A is a hyperparameter that affects the rate of change of wly-
Since for E [-1,1] we have Eor + E [0, 2]. The smaller Lar is, the more similar 50 and of are, indicating that RobustZero is more robust to worst-case state perturba- tions. According to Eq. wly increases as Lar decreases. This feature allows us to first assign high weight to the worst-case loss term, enabling updates to 0 to strengthen the model's robustness to worst-case perturbations. Then, as the similarity between sp and sf increases, we gradually adjust wly allow RobustZero to accommodate random-case state perturbations. When Car -1, we have wly 1. In this case, the worst-case and random-case loss terms have equal weight, which occurs only if the trained RobustZero model can handle worst-case state perturbations fully.
value T+++ and between the predicted reward ? real reward Pi+ke
Similarly, the random-case loss term is defined as:
K
C = B
(Pinki # +
KL
(+
k=
+ 02
Σ ( ot) of)
+ cashar
(4)
Next, we design w2,r. Note that while training guided by the combination of the worst-case and random-case loss terms can enhance the robustness of RobustZero, the trained model may experience compromised performance in the presence of unseen worst-case or random-case state perturbations. To address this issue, we dynamically adjust the decay term to guide the learned model towards flat minima and away from steep minima, thereby improving generalization. To this end, we employ the gradient information (Ghiasi et al.] 2023 to determine the dynamic setting of w2e. More specifically, the gradient update for parameter 0 from time step t' to t' is:
As regularizer, the decay term is added to the main loss function to avoid over-fitting:
Edecay = |0012
(5)
With these components, we define RobustZero's loss func- tion as:
w2y 2 Laieaay
(6) where wle and 22y are weights of the loss function, and I is the time index in the training phase. The performance of RobustZero is affected by the settings of the two weights. We thus consider shortly how to adjust wly and w2, dy- namically to enhance RobustZero's ' robustness to both worst- case and random-case state perturbations.
L= 1=
O0 -.-
- -s(V + w/e V Lo.., + 02 (8)
where is the stepsize and V Co, and L are gradi-
ents related to 0 at t' from Eqs. 3 and 4 respectively.
5.3. daptiiv Adjustment during Training
Similarly, we let V Lop and Lap, denote the gradients related to ő at - from Eqs. and respectively.
We proceed to explain how to train RobustZero. We recall that the worst-case and random-case perturbation policies re- flect the worst and average scenarios, respectively. Although both types of perturbations are important, we hope to first increase the weight of the worst-case loss term to improve the robustness of RobustZero to worst-case perturbations, and then gradually accommodate the average-case during the training phase. The main reasons are: 1) the worst- case perturbation policy is purposeful and smart, while the random-car mane perturbation policy is random. Thus, first im- proving the ability of the model to cope with the worst-case perturbation is beneficial for the stability of the overall train- ing process; and 2) worst-case state perturbations are more dangerous than random-case state perturbations. This makes it preferable to weight the worst-case highly. To achieve this, we design wle as:
In Eq. 8 items Včo,_, wl, VL and 022 quan- tify the actual strength of an update of 0. Further, A is used to capture the ratio between the magnitudes of 02 and V Vo + wl. VL.
12 -112
A2
(9)
V wl, V V
In order to maintain the actual effect of the main loss (i.e., E + w1 L) and the decay unchanged, we keep x as a constant. This enables us to calculate w:2y, as:
Aa VallVCo, + wlr VLo .112
w2p = sg
(10)
10
wle le 2- 5g
1 =+>l(entt))
Eq. 10 shows that 22p is updated adaptively according to the changes to V Vo,_,, VLa,,,, wly, and 0p-1- Then,
6

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
the weighted decay term in Eq. can be re-written as:
15-dimensional (11 continuous actions and 4 discrete ac- tions), and 32-dimensional (22 continuous actions and 10 discrete actions), respectively. The three transportation en- vironments support the testing of autonomous driving tasks. Therein, an autonomous driving car interacts with other ve- hicles to navigate different scenarios: i) Highway-Drine fast, avoid collisions, and stay in the right-most lane; ii) Intersection -rross safely, follow traffic rules, and keep a steady speed; and iii) Racetrack Finkshh quickly while stay- ing track and driving smoothly. The action space of an autonomous driving car is two-dimensional. Furthermore, we compare with five baselines:
u22
sg(A2) V Vo,,, + Vo, 112)110
2 Ldecay
2
(11)
Eq. indicates that the weighted decay term increases with the gradient of the main loss function, and vice versa. The incorporation of gradient information brings two major benefits. First, when the gradient of the main loss function approaches zero, the weight decay term also approaches zero. This prevents over-optimization of the weighted decay term in flat minima, allowing stronger emphasis on the main loss function, which is beneficial for obtaining a model that is more robust to state perturbations. Second, a large weighted decay term is applied when the gradient of the main loss function is large. This prevents the model from settling into steep local minima, thereby reducing the risk of early overfitting during training. In this sense, adaptive adjustment of w2p enables the learned model to converge to flat minima and escape from steep minima.
ATLA-PPO Zhang et al. 2021 can obtain worst-case perturbation policies under black-box box attacks and can defend against such attacks. ATLA-PPO is trained under worst-case state perturbations.
PROTECTED (Liu et al., 2024) the state-of-the-an and most related model-free DRL method for handling worst-case and random-case state perturbations. PRO- TECTED is trained under worst-case and random-case state perturbations.
Next, to achieve more stable training, we use smooth update rules for both w and w:2gr, i.e.:
S-MuZero (Hubert et al. 2021) is the most related MuZero-class method supporting complex action spaces. S- S-MuZero is unable to defend against state perturbations and is trained under no state perturbation.
wle < + + 0.9w1e w2 + 0.9w2
(12)
(13)
S S-MuZero-wer worst and --MuZero-randonn are baselines, representing S-MuZero trained under worst-case and random-case state perturbations, respectively.
The pseudocode of the training process to update the model of RobustZero is provided in Appendix B.2
Following Zhang et al. (202T) and Liu et al. (024), we adopt ATLA-PPO obtain Aworst that reflects the strongest impact on reducing the accumulated reward. The Uniform noise is used to execute Arandom that reflects the average impacts of all possible perturbations. Details of the experi- mental settings are provided in Appendix C.I
6. Experiments
6.1. Experimental Setup and Baselines
We study RobustZero on: 1) two classical control environ- ments, including CartPole" Pendulun 2) three energy environments in power distribution systems with hybrid action spaces (Fan et al. 2022), including IEEE 34-bus, IEEE b23-bus, and IEBB 8500-node systems; 3) three trans- portation environments including Highway with discrete action space, Intersection with discrete action space, and Racetrack with continuous action space; and 4) four Mu- joco environments with continuous action spaces, including Hopper, Walker2d, HalfCheetah, and Ant, following a setup similar to that Liu al. (2024). Therein, the three energy environments support the testing of voltage control tasks with the objective of minimizing the total cost of voltage violations, control errors, and power losses, while meeting both networked and device constraints. The action spaces of TEEE b4s, IEEE 123-bus, and IEEE 8500-node are 10- dimensional (8 continuous actions and discrete actions),
6.2. Comparison Study
We conduct the main experiments to assess robustness of RobustZero. We evaluate the performance of the methods under three scenarios: i) no perturbations, ii) worst-case state perturbations, and iii) random-case state perturbations. The average episodic rewards + the standard deviation over 50 episodes for the three scenarios (natural, worst-case, and random-case are reported in Table [ In each column, the highest reward is highlighted in bold. Due to the space limitation, Table only reports the results on Pendulum, IEEEE 34- IEBE 8500-node, and Racetrack. The results and analyses for the remaining environments are provided in Appendix C.2 Overall, RobustZero outperforms all five baselines defending against state perturbations across the four environments. The details are as follows.
@gymassium.arraaa www..maw.wggweeiioomnnnnssaiiiia control/cart pole/
gymnasium. www.mmaaaaaaaaaaaaaaaa www.aa/.woggeiiooommesssssssasie control/pendulum/
RobustZero and S-MuZero achieve similar natural re- wards that are higher than those of the other baselines.
github.com cociii.cm//aaae

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table 1. Main experimental results Pendulum, IRFE 34-bus, IEFE 8500-node, and Racetrack.
Pendulum
IEEE 34-bus
Method
Natural
Worst-case Reward
Random-case Reward
Natural
Worst-case
Random-case
Reward
Reward
Reward
Reward
ATLA-PPO
-90.26 + 64.32
.94.24 + 66.42
992.66 69.42
-1.36 1 0.12
-14.07 1 0.33
-3,56 + 0.17
PROTECTED
-89.46 + 59.83
-91.% + 51.24
-91.32 1: 55,44
-10.43 = 0.09
-12.89 + 0.32
-12.26 +: 0.07
S-MuZero
-82.58 + 57.05
-250.02 + 102.56
-166.89 + 62.53
-8.56 = 0.19
-25.70 + 6.41
-19.50 + 2.95
S-MuZero-worst
-251.49 + 2.89
-89.98 48.36
-170.57 + 68.96
-15.31 0.47
-15.07 0.39
-15.79 + 2.59
S- S-Muuerr-random
-88,91 = 56.90
-168.70 + 60.55
-130.29 + 99.12
-10,96 0 0.49
-15.76 2.19
-14.68 0.39
RobustZero
-83.48 1 58.96
-87.92 1 53.24
84.68
55.74
-9.57 1 0.43
-12.16 + 1.19
-11.43 + 0.37
IEEE 8500-node
Racetrack
Method
Natural Reward
Worst-case Reward
Random-case Reward
Natural Reward
Worst-case Reward
Random-case
Reward
ATLA-PPO
-277 =
-1412 = 12
-1351 =
409.13 0.31
386.69 + 0.16
392.87 = 0.01
PROTECTED
-1264 1
-1376 + 19
-12% + 19
478.40 + 3.00
429.36 + 5.14
459.10 6.03
S-MuZero
-1135 +
-1863 +: 25
-1686 + 15
567.80 3.16
352.20 + 6.93
370.40 3,94
S-MuZero-worst
-1402 + 26
-1457 + 21
-1521 + 10
375.9 + 6.32
410.00 + 6,94
398.70 + 6.32
S-MuZZerorrandom
-1294 + 20
-1503 + 18
-1368 +
415.60 + 4.64
390.50
+ 7.64
410.82 + 6.06
RobustZero
-1158 + 30
-1374 + 35
-1217 = 27
520.60 = 1.32
477.00 + 1.26
494.00 2.42
The major reasons are: i) MuZero-class methods use an abstract environment model to make plan ahead, which enables better solutions than model-free DRI., e.g., ATLA-PPO and PROTECTED; and ii) S-MuZero- woostt and MuuZero-randon cannot obtain comparable natu- ral rewards because they are trained under worst-case state perturbations only and random-cass state perturba- tions only, respectively. Additionly, S-MuZero obtains slightly higher natural rewards than RobustZero. This is because S-MuZero is trained without state perturba- tions. It thus obtains the best natural rewards. However, its worst-case and random-case rewards decrease notably. In comparison, RobustZero can still obtain comparable natural rewards but much better in both worst-case and random-case rewards.
RobustZero achieves a higher random-case reward than all baselines across all the four environments, increasing this reward by 7.5 18% 101 .43% on Pendulum, by 7.26% 70.60% on IEEE 34-bus, by 6.49%-38.5 8500-node, and by 50%--3.377% on Racetrack. Sim- ilarly, S-MuZZero-random achieves a high random-case reward, but also reduced natural and worst-case rewards due to the absence of a defense strategy.
on IEEE
We make additional comparisons: 1) Evaluation of all meth- ods under perturbation policies that were not encountered during training, to assess the generalization capability of RobustZero (see Appendix C.5): 2) Time and sampling ef- ficiency analysis that focuses on the training and sampling time, and sampling efficiency (see Appendix C.6).
RobustZero achieves a higher worst-case reward than all baselines across the four environments. RobustZero in- creases the worst-case reward by 1..4%-184.379 on Pen-
6.3. A Ablation Study
We study the effects of the suff-supe representation network, the adaptive adjustment mechanism, and the paral- lel use of worst-case and random-case buffers on improving the robustness of RobustZero. We use RobustZero/cl to de- note RobustZero without the self-supervised representation network, use RobustZero/wl to denote RobustZero trained with w1 = + use RobustZero/u2 to denote RobustZero trained with w2 set 5c-6, 5e-8, 5e-8, and le-5 for Pen- dulum, IEEEE 34-bus, IEEE 8500-node. and Racetrack, respectively use RobustZero-Dwe to denote RobustZero employing only the worst-case buffer, and use RobustZero- DRCc to denote RobustZero employing only the random-case buffer. Similarly, we report results for Pendulum, IEEEE 34-bus, IEEE 8500-node, and Racetrack, as shown Table The counterparts for the remaining environments are
dulum, by 6.00%-111 1109--11..355 1.35% on IEEE 34-bus, by 0. 0.15% 35.59% on IEEE 8500-node, and by 11.10%-35.44 43%% on Racetrack. Further, S-MuZero-wors obtains higher worst-case reward than S S-MuZero and S-MuZero-random Notably, its worst-case rewards appear higher than the corresponding random-case rewards. This is because S- MuZero-worst does not incorporate any robustness mech- anism. It is trained solely on perturbed states, without any information about non-perturbed states. This causes S- MuZero-worst to be over-trained under the worst-case per- turbation policy, making it highly specialized and adapted to that particular perturbation pattern, As result, its per- formance under worst-case state perturbations unusually appears higher than its performance under random-case state perturbations as observed in Table [ Although S-MuZero- worst achieves relatively high worst-case re- wards, its natural reward and random-case reward are low, due to the lack of defense strategy.
5
ww = indicates that worst-case and random-case perturba- tions are treated equally.
'The method for selecting these values of w2 is provided in Appendix C
8

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table Ablation study on Pendulum, IEEE 34-bus, IREE 8500-node, and Racetrack.
Pendulum Worst-case Reward
IEEE 34-bus
Method
Natural Reward
Random-case
Natural
Worst-case
Random-case
Reward
Reward
Reward
Reward
RobustZerofcl
-85.43 1 54.00
-91.15
58.15
-87.24 + 55.00
-12.03 1 0.30
-15.18 = 1.15
-14.03 + 0.32
RobustZero/w
-83,99 + 54.64
-89.41 +: 49,42
-85.19 + 55.19
-10.12 + 0.42
-13,41
1.30
-12.20 +: 0.26
RobustZero/w2
-84.67 56.36
-90.23 + 57.00
-86.84 + 55.24
-11.57
0.36 0.36
-14,63
1.10
-13.49 + 0.24
RobustZero- Dwc
-89.36 + 56.10
-90.63 58.02
-90.88 = 54.69
-14.16 0.50
-14.89
0.71
-14.76 + 0.40
RobustZero-Dee
-87.67 + 55.81
-92.26 57.66
-89.09 = 56.02
-13.60
0.61
-14,96
1.06
-14.31 = 0.52
RobustZero
83.48 1 58.96
-87.92 53.24
-84.88 1 55.74
99.57
1 0.43
-22116 1.19
-11.43 + 0.37
IEEE 8500-node
Racefrack
Method
Natural Reward
Worst-case Reward
Random-case Reward
Natural Reward
Worst-case Reward
Random-case
Reward
RobustZero/el
1320 = 30
-1598 = 36
-1479 + 25
422.60 = 1.36
360.33 = 1.35
378.65 + 2.30
RobustZero/wl
-1200 + 28
-1488 + 32
-1369 28
490.11 + 1.32
400.00 = 1.24
440.89 2.31
RobustZero/u2
-1288
-1533 + 34
-1403 26
448.70 +: 1.28
385.23 = 1.34
414.55
2.28
RobustZero-Dwy
-1416 + 27
-1609 + 35
-1567 + 25
380.26 + 1.35
359.77 = 1.11
368.75 2.34
RobustZeno-Dre
-1380 + 28
-1641 + 37
-1520 + 24
392.30 + 1.31
355.74 = 1.20
370.02 + 2.52
RobustZero
-1158 + 30
-1374 + 35
-1217 + 27
520.60 + 1.32
477.00
1.26
494.00 + 2.42
provided in Appendix C.4 Compared to RobustZero, we derive the following observations from Table
of RobustZero, we conduct extensive experiments in Ap-
pendixes C.7 and C.8
RobustZero/cl decreases the natural reward by 2.34% 25.71% % the worst-case reward by 3.67%-32.38%% and the random-case reward by 3.02%-30.44%% Thus, the self-supervised representation network clearly enhances the robustness of RobustZero, while ensuring high natu- ral reward.
7. Conclusion and Future Work
We propose RobustZero, the first MuZero-class method designed ensure robustness state perturbations. It fea- tures novel robust training framework and training method to defend against both worst-case and random-case state perturbations. The framework includes a self-supervised representation network that enables the generation of con- sistent policies before and after state perturbations. Further, unique loss function enables robust training. In addition, an adaptive adjustment mechanism enables updates of Ro- bustZero to offer policy with high robustness to worst-case and random-cass state perturbations. Extensive experiments eight environments offer evidence that RobustZero ad- vances the state-of-the-art methods at defending against worst-case and andom-case state perturbations. In future re- search, is interest to develop multi-agent RobustZero and apply it in Internet of Things scenarios.
RobustZero/wi decreases the natural reward by 0.61 % 6. 6.22%, the worst-case reward by 1.69%-19.25%, and the random- reward by 0.60%-12.49% This verifies that using a dynamic w1 to first enhance the defense capability to worst-case state perturbation and then to gradually adapt to random-case state perturbation beneficial.
RobustZero/w2 decreases the natural reward by 1.43%- 20.909 90%, the worst-case reward by the random-case reward by 2.55% This be- cause an adaptive w2 enables the learned model con- verge flat minima with enhanced robustness to unseen states. A fixed w2 does not have this capability.
and
RobustZero-Dwy decreases the natural reward by 7.04%- 47. 47.96%, the worst-case reward by 3.08%-32.58%. and the random-case reward by 7.32%-33.97% Further, RobusiZero-DRc decreases the natural reward by 5.02%- 42.1 1 96, the worst-case reward by 14.%%34..99 and the random-case reward by 5.21% 1211%-335119 5.%. RobustZero- both reduce RobustZero's 's per-
A Acknowledgments
This work was supported in part by the Digital Energy Hub grant from the Association of Danish Industry and in part by Villum Investigator Grant S4OS under Grant 37819.
Dwc and RobustZero--Da formance notably. This is because the dynamic adjust- ment of wl and w2 requires the interaction of two buffers. Using separate buffers makes w1 and w22 ineffective.
Impact Statement
This paper advances the field of RL by enhancing the ro- bustness of MuZero-class methods state perturbations. To the best of our knowledge, this gives the first successful attempt in such field. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
We further evaluate the performance of all variants of Ro- bustZero against unforeseen perturbation strategies (see Ap- pendix C.5). In addition, to assess the impacts of hyper- parameter settings and attack radiuses on the performance

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
References
Hubert, T., Schrittwieser, J., Antonoglou, I,, Barekatain, M., Schmitt, S., and Silver, D. Learning and planning in complex action spaces. In International Conference Machine Learning, pp. 1476-4488 2021.
Bukharin, A., Li, Y., Yu, Y., Zhang, Q. Chen, Z., Zuo, S., Zhang, C., Zhang, S., and Zhao, T. Robust multi- agent reinforeement learning via adversarial regulariza- tion: Theoretical foundation and stable algorithms. Ad- vances in Neural Information Processing Systems, 36: 68121-68133, 2024.
Kang, Y., Shi, D., Liu, J., He, L., and Wang, D. Beyond reward: Offline preference-guided policy optimization. In International Conference on Machine Learning, pp. 2023.
Danihelka, I., Guez, A.. Schrittwieser, J., and Silver, D. Policy improvement by planning with Gumbel. In Inter- national Conference on Learning Representations, 2022.
Liang, Y., Sun, Y., Zheng, R., and Huang, F. Efficient ad- versarial training without attacking: Worst-case-awar robust reinforcement learning. Advances in Neural Infor- mation Processing Systems, 35:22547-22561 2022.
Dong, J., Hsu, H.-I.., Gao, Q. Tarokh, V., and Pajic, M. Variational adversarial training towards policies with im- proved robustness. In International Conference on Artifi- cial Intelligence and Statistics, 2025.
Liu, X., Deng, C., Sun, Y., Liang, Y., and Huang, Beyond worst-case attacks: Robust RL with adaptive defense via non-dominated policies. In International Conference Learning Representations, 2024.
Engstrom, I.., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L.,, and Madry, A. Implementation matters in deep policy gradients: A case study on PPO and TRPO. arXiv preprint (0Xi::000.11222) 2020.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540);
Eysenbach, B. and Levine, S. Maximum entropy RL (prov- ably) solves some robust RL problems. arXiv preprint arXiv:2103.062577 2021.
529-533, 2015.
Nasiriany, S., Pong. V., Lin, S., and Levine, S. Planning with goal-conditioned policies. Advances in Neural Infor- mation Processing Systems, 32:1-12, 2019.
Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. Contrastive learning goal-conditioned reinforce- ment learning. Advances in Neural Information Process- ing Systems, 35:35603-35600, 2022.
Niu, Y., Pu, Y., Yang, Z., Li, X., Zhou, T., Ren, J., Hu, S., Li, H., and Liu, LightZero: unified benchmark for monte carlo search in general sequential decision scenarios. Advances in Neural Information Processing Systems, 36, 2024.
Fan, T.-H., Lee, X. Y., and Wang, Y. PowerGym: A rein- forcement learning environment for volt-var control in power distribution systems. In Learning for Dynamics and Control Conference, pp. 21-33, 2022.
Oikarinen, T., Zhang, W., Megretski, A., Daniel, L., and Weng, T.-W. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Pro- cessing Systems, 01226166221661 2021.
Ghiasi, M. A., Shafahi, A., and Ardekani, R. Improving ro- bustness with adaptive weight decay. Advances in Neural Information Processing Systems, 36:79067-79080, 2023.
Han, X., Zhao, X., Zhang, L.., and Wang, W. Mitigating action hysteresis in traffic signal control with traffic pre- dictive reinforcement learning. In ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, pp. 673-684, 2023.
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. Ro- bust adversarial reinforcement learning. In International conference machine learming, pp. 2817-2826, 2017.
Russo, A. and Proutiere, A. Towards optimal attacks on reinforcement learning policies. In American Control Conference, pp. 4561-4567, 2021.
Hao, Q. Huang, W., Xu, F., Tang, K., and Li, Y. Reinforce- ment learning enhances the experts: Large-scale covid-19 19 vaccine allocation with multi-factor contact network. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4694-4694, 2022.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering Atari, Go, Chess and Shogi by planning with a learned model. Nature, 588 (7839):604 609, 2020.
Huang. P., Xu, M., Fang, F. and Zhao, D. Ro- bust reinforcement learning as a Stackelberg game via caapdiively-rguuariiie adversarial training. arXiv preprint www.xx:2202..9511222 2022.
Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A. C., and Bachman, Data-efficiens reinforcement learn- ing with self- -redderede representations. In International Conference on Learning Representations, 2021.
10

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L.., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I,, Panneershelvam, V., Lanctot, M., et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):4844455 489, 2016.
Conference on Knowledge Discovery and Data Mining, pp. 1899-1909, 2022.
Xuan, C., Niu, Y., Pu, Y., Hu, S., and Yang, J. ReZero: Boosting MCTS- algorithms by just-in-time and speedy reanalyze. arXiv preprint arXiv:2406 16364, 2024.
Silver, D., Schrittwieser, J.. Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L.,, Lai, M., Bolton, A., et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354 359, 2017.
Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Y. Mas- tering Atari games with limited data. Advances in Neural Information Processing Systems, 34:25476-25488, 2021.
Sinha, A., O'Kelly, M., Zheng, H., Mangharam, R., Duchi, J., and Tedrake, R. FormulaZero: Distributionally robust online adaptation via offline population synthesis. In International Conference on Machine Learning, pp. 8992- 9004, 2020.
Zamora, W., Calafate, C. T.,, Cano, J.-C., and Manzoni, P. Noise-sensing using smartphones: Determining the right time to sample. In International Conference on Advances in Mobile Computing & Mullimedia, pp. 196-200, 2017.
Sun, Q. Zhang, L., Yu, H., Zhang, W., Mei, Y., and Xiong, H. Hierarchical reinforcement learning for dynamic tonomous vehicle navigation at intelligent intersections. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4852-4861, 2023.
Zhang, H.,, Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33: 2020a.
Sun, X. and Zheng, Z. Belief-enriched pessimistic q- learning against adversarial state perturbations. In Inter- national Conference on Learning Representations, 2024.
Zhang, H., Chen, H., Boning, D., and Hsieh, C.-J. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101 08445,, 2021.
Sun, Y., Zheng, R., Liang, Y., and Huang, F. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In International Conference on Learn ing Representations, 2022.
Zhang, X., Ma. Y., Singla, A., and Zhu, X. Adaptive reward- poisoning attacks against reinforcement learning. In In- ternational Conference on Machine Learning, pp. 11225- 11234, 2020b.
Tessler, C., Efroni, Y., and Mannor, S. Action robust rein- forcement learning and applications continuous control. In International Conference on Machine Learning, pp. 2019.
Zhihe, Y. and Xu, Y. Dmbp: Diffusion model-based predic- tor for robust offline reinforcement learning against state observation perturbations. In International Conference on Learning Representations, 2023.
Timothy, P. L., Jonathan, J. H., Alexander, P., Nicolas, H., Tom, E., Yuval, T., David, S., and Daan, W. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02977 009.009999997,.. 2015.
Zhou, Z., Liu, G., and Zhou, M. A robust mean-field actor- critic reinforcement learning against adversarial pertur- bations on agent states. IEEE Transactions on Neural Networks and Learning Systems, 13300-113770-1381,, 2024.
Wang, M., Feng, M., Zhou, W., and Li, H. Stabilizing voltage in power distribution networks via multi-agent re- inforcement learning with transformer. In ACM SIGKDD

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
A. An Example in Energy System Subject to State Perturbations
The following example show why is important to enhance the robustness of MuZero-class methods to state perturbations. Example We consider an environment encompassing 6-bus power system, where the voltage (the state) of each bus must be in the range [0.50, 1.00] p.u. (per unit). At time t. the real voltages of the six buses are (0.95 0.55 0.78 0.86 0.87 p.0.)p.u. However, due to malicious attacks and noisy sensors, the agent receives perturbed voltage readings: (0.96, 0.57, 0.80, 0.85, 0.89, 0.97) p.u. These perturbations affect the agent's actions, leading different rewards and subsequent voltages. In particular, without perturbations, the reward is 0.89 p.u. and the resulting voltages are [0.94, 0.60, 0.75, 0.85, 0.85, 0.96] p.u. With perturbations, the reward drops to 0.22 p.u., and the next set of voltages is [0.90, 0.68, 0.77, 0.92, 0.80, 1.03] p.u., which violates the range constraint.
B. Details of RobustZero
B.1. Details of Parallel Data Collection
We consider parallel data collection to gain experience when states are suffering from state perturbations. Algorithm I provides the pseudocode for each round of data collection. Functions WC(.) and RC(-) define the data collection under worst-case and random-case state perturbations, respectively. Both Environment and Environment support multiple and parallel task execution to collect data. As a result, multiple WC(.) and RC(-) calls can be executed in parallel. For simplicity, we ignore episodes here. Additionally, WC(-) and RC(.) use the same NN parameters
We take function WC(.) as example illustrate data collection. Adversary observes the real state or from Environment I and launches worst-case perturbation policy (or) obtain the perturbed state oz. Next, RobustZero agent receives of that is further encoded the initial hidden state (i.e., the root node) & by the using the representation network
Algorithm 1 Parallel Data Collection
Input: NN parameters 0; number of sampled actions K:; end time of episode T and T Output: U1 oz, कz. 0441 P7411,1411000 9441) and Uggigf(0g of. Rg. difi Pi41 o141) function WC(or, Aworstt No. S-CCSS)
for = to 7 do Adversary do:
observe of. oz Awora Agent of RobustZero do:
receive 07. 20 + (oz)
7H + s-MCTS($ K.
01+1 7े / sample Environment 1 do:
action */
receive 0;41. return 044 and 1741
end for
RETURN U U1gig (or of Ti Tif1 Fit1. 9+1)
end function function RC(o;, Arandom Ho, s-MCTS)
for i 1 to T do
Adversary 2 do:
observe og, O +- Arandom (o:)
Agent of RobustZero do:
receive 0z. 80 <- (ō;)
#: S-MCTS(3) 0, K))
01+1 - # Environment 2 do: receive 0.41. generate 0: and fi54
end for
RETURN Uigist(o; Op, Rt aet1s Ai41s 0
end function
12

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Algorithm 2 Training process of RobustZero
Input: Number of iterations Nater, number of update per iteration Nu. batch size B, hyperparameters A and 22, and
step-size
Output: and
Initialize NNs of 0 and Φ, replay buffers Dwc and DRC, and wl and w2 0
for i = 1 to Niter do
Dwc - W-(-), DRC + RC(-) /x See Wo(+) and RC(-) from Algorithm 1 +/ for j to N₂ do
'' + (i 1) +j
Sample a random mini-batch of B samples from Dwc and DRC, respectively Compute wLe according to Eq. aand Eq.
wly 0.1w1e-1 0.9w1
L' - 1--C+0 wly C i V
VLo.. Sor backward((") /* Calculate gradients +/
Compute w2y according to Eq. 10
w2x + 0. 0.9w2
Oy + By <(V Co, + wly Vor., + 022
Or t Φ1 - s( V Lov + wly 1
end for end for RETURN 0 and の
Ha(ā)) The sampled MCTS (Hubert et al. al. 2021) is used to obtain the search policy क.. The action شيز is determined by sampling an action from Te with the same setting as in the literature (Hubert et al. 2021). Then, Environment 1 receives the action "z41 and generates the immediate real reward 7741 and the next state 0f+1- Finally, the data
Uicisf(or, oü, Ty. 07+1: P7-1: (42) is stored in the worst-case buffer Dwc- A similar procedure is applied to function RC((). The key differences are that the perturbed state is determined by using the random perturbation policy (rrandom(op) and the data Ulsist(0p oū, # di+1 Mit1, 2+1) are stored in the random-case buffer DRc-
B.2. Pseudocode of the Training Process
The implementation of training the RobustZero is provided in Algorithm 2
C. Details of Experiments
C.1. Details of Experimental Setup
We conduct experiments on an 8-core Intel Xeon E5-2640 v4 @ 2.40GHz CPU, with each node equipped with a GeForce RTX 3090 GPU, 2.40GHz processor, and 24GB RAM. Following existing studies (Zhang et al. 2021 Liu et al. 2024), we train 21 agents for each method with the same hyperparameters. The agent with median performance is reported for the purpose of reproducibility. Regrading attack radius, we set c = 0.20 for two classical control environments (i.e., CartPole and Pendulum),
0.10 for the three energy environments (i.e., IEEE 34-bus, IEEE 123-bus, and IEEE 8500- 6 <= 0.15 for the three transportation environments (i.e., Highway, Racetrack, and Intersection), and e = 0.075 for Hopper, e = 0.05 for Walker2d, = 0.15 for Halfcheetah, and = 0.15 for Ant, respectively. Moreover, we use Lx norm for the budget constraint.
We follow the original configurations for ATLA--PO (Zhang et al. 2021) and PROTECTED (Liu al. 2024). The configurations for S-MuZero, S-Muzerooworst, and S-Muzero-random are available on open science platforms alongside our project. The random perturbation is implemented by using Uniform noise that follows an uniform distribution UI (((-e,e) e).
C.2. Major Comparison Studies on The Remaining Eight Environments
Following the results in Section 6.2 we further provide the corresponding results the remaining eight environments. The average nature, worst-case, and random rewards for all methods on CartPole, IEEE 123-bus, Highway, Intersection, Hopper, Walker2d, HalfCheetah, and Ant are reported in Table We observe that RobustZero consistently outperforms
13

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table Main experimental results CanPooc, IEEE 123-bus, Highway, Intersection, Hopper, Walker2d, HalfCheetah, and Ant.
CartPole
IEEE 123-bus
Method
Natural
Worst-case
Random-case
Natural
Worst-case Reward
Random-case
Reward
Reward
Reward
Reward
Reward
ATLA-PPO
500.00 0.00
414.38 12.13
420.26 144.08
-3.84
0.06
-15.24 1 0.24
-14.54 1 0.04
PROTECTED
500.00 + 0.00
430.22 + 155,66
435.47 i 162.66
-12.29 +: 0.06
-14.15 + 0.27
-13,42 + 0,03
S-MuZero
500.00 0.00
200.14 + 18.01
359.98 + 86.31
-9.77
-28.84 5.84
-19.63 2.75
S- S-MuuZerowworst
239.14 14 6.24
428.64 69.73
412.62 + 16.42
-16.67
4.92
-15.78
3.99
-18.61
3.77
S-Muuerr-randon
444. 14 75.91
284.52
78.70
448.24 + 87.75
-11.59
2.88
-23.02 3.19
-17.16
4.12
RobustZero
500.00 0.00
460.56
56.33
490.06 1 5.15
-10.48 1.19
-1322 + 1.43
-1233 + 2.00
Highway
Intersection
Method
Natural
Worst-case
Kandom-case Reward
Natural Reward
Worst-case Reward
Random-case
Reward
Reward
Reward
ATLA-PPO
18.56 = 0.50
18.68 + 0.17
18.73 + 0.08
2.32 = 0.03
1.78 0.29
1.87 I 0.05
PROTECTED
20.26 + 0.80
19.11 0.14
19.48 + 0.03
3.18 - 0.07
2.05 0.30
2.30 - 0.10
S-MuZero
25.83 3.16
15.91
4.93
17.47 + 2.94
3.63 0.30
1.56 0.32
1.68 0.03
S-MuZero-worst
13.64 + 5.36
18.62 + 3.59
17.89 + 1.95
1.52 - 0.18
1.98 0.50
1.84 + 0.80
S-MuZZrroranddm
14.47 + 3.64
17.49
+ 4.64
18.19 + 2.36
2.07 = 0.68
1.92 0.13
2.13 + 0.21
RobustZero
23.24 = 1.32
20.95
1.26
22.09 + 1.42
3,43 = 0.26
2.87 1.48
3.15 0.34
Hopper
Walker2d
Method
Natural Reward
Worst-case
Random-cast case
Natural Reward
Worst-case Reward
Random caso
Reward
Reward
Reward
ATLA-PPO
3311 310
1736 360
3167 + 180
3831 200
3659 210
3898
PROTECTED
3629 + 300
2533 251
3538 + 310
6311 + 112
6010 + 321
6142 + 150
S-MuZero
3672 + 316
1452 + 390
2454 + 284
6430 + 280
890 + 180
2615 + 200
S S-MuZero-worst
1675 + 332
1810 + 194
1756 + 132
1315 + 120
1508 + 110
1435 + 130
S-MuZero-random
2678 264
1732 264
2589 + 306
2627 + 180
1158 + 200
2640 120
RobustZero
3660 - 132
2577 126
3553 + 242
6384 + 230
6096 + 350
6322 320
HalfCheetah
Ant Worst-case Reward
Method
Natural Reward
Worst-case Reward
Random-case Reward
Natural Reward
Random-case
Reward
ATLA-PPO
6106 1 210
5114 i 230
6112 1: 150
5335 1: 150
3780 1: 120
5352 1: 270
PROTECTED
7044 = 168
5186 + 175
6258 + 209
5736 = 280
4582 + 190
5586 + 190
S-MuZero
7185 360
2524 272
3245 + 184
5830 = 200
510 250
3730 150
S-MuiZcroworrs
2714 302
2830 - 149
2786 + 129
1206 + 260
1423 210
1330 100
S-MuZZro-rrnndo
3486 1 226
2692 246
3575 + 286
3894 1 200
805 - 180
3811 4 150
RobustZero
7100 + 232
5395 + 162
6394 + 192
5782 + 130
4686 +: 250
5692 +: 220
all baselines defending against state perturbations across all environments. Specially, RobustZero and S-Muzero still obtain similar nature rewards, where both of them are higher than those obtained by using other baselines. The exception on CartPole, where four methods are able obtain the optimal solution. Moreover, although S- Muuuero achieves the best natural rewards, it obtains low worst-case and random-cas rewards. This implies that s- M-MuZero is very sensitive to state perturbations due to the lack of a defense strategy. This is the motivation for developing RobustZero. As expected, RobustZero significantly enhances the robustness to state perturbations with highest worst-case and random-case rewards compared to all baselines, while maintaining comparable natural rewards compared to S-MuZero. As observed Table on CartPole, by Hopper, by 1.4%-584.9%% Intersection, by 1.7%-77.5%
RobustZero increases: 1) the worst-case reward by 7. 7.05% 10666130.1222 12%
on IEEE 123-bus,
by 13.%%-3..6999 Highway, by 30.00%%88..999
Walker2d, by 4.0%-113, 7% HalfCheetah, and by 2.3%-818. 818.8% Ant; and 2) random-case reward by 12.54%-36.1499 on CartPole, by 8.93%-59.3. 59.33% on IEEE 123-bus, by 13.40% 26.5% on Highway, by 36.96%-87..55 -7.50%% on Intersection, and by 102.3% on Hopper, by 2.9%-340.6% on Walker2d, by 2.2%-129.5% on HalfCheetah, and by 1.9%-328.00 on Ant.
C.3. Selection of w2 in Ablation Study
We set deterministic values for w2 in different environments based on the average of the corresponding dynamic w2, selecting the best from five points near this average. The results of natural, worst-case, and random-case rewards by using different values of w2 on the eight environments are provided Fig. 2 As observed, RobustZero/w2 obtains the best natural, worst-case, and random-cass rewards when it is set 5e-6, 5e-6, 5e-8. 5e-8, 5e-8, 1e-5, le-5, 1e-5, 5e-6. 5e-6, 1e-6, and le-6 for CartPole, Pendulum, IEEE 34-bus, IEEE 123-bus, IEEE 8o00-node, Highway, Intersection, Racetrack, and Hopper, Walker2d, HalfCheetah, and Ant, respectively.
14

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
reward
Woost-case reward
Random
500
-55.0
-12
-12
Reward
450
-87.5
-14
14
400
-90.0
-92.5
16
50-5
le-5
Se-6
Ie-6
5e-7
5e-5
1e:5
Se-6
Ie-6
5e-7
Ie-6
5e-7
1e-7
Se-8
le-8
1e-6
Se-7
1e-7
5e-8
1e-8
W2
W2
W2
W2
(8) CartPole
(b) Pendulum
(c) JEEE 34-bus
(d) LEEE 123-bus
-1400
20
Reward
2,0-
-1600
18-
-1800
16
1,5
1o-7
5o-8
1o-8
1o-4
50-5
1o-5
5c-6
1e-6
5o-5
1o-5
50-6
16-6
1o-5
So-6
16-6
W2
W2
W2
W2
(c) (EEE 8500-node
(i) Highway
Iotersection
(h) Raoctrack
6400
7000
3500
5500
Roward
6200
3000
5000
6000
2500
4500
50-5
16-5
5c-6
Io-6
Se-?
5o-5
16-5
50-6
5e-7
16.5
56-6
50-7
16-7
5o-7
16-7
W2
W2
Wa
W2
Hopper
Walker2d
(k) HallCheetah
(I) Ant
Figure 2. The selection of w2.
C.4. Major Ablation Study on The Remaining Eight Environments
Following the results in Section 6.3 we further provide the corresponding ablation study the remaining eight environ- ments. Specially, the average nature, worst-case, and random rewards for RobustZero/cl, RobustZero/w:l, RobustZero/ww, RobustZero-Dwe: and RobustZero-DRc on CartPole, IEEE 123-bus, Highway, Intersection, Hopper, Walker2d, HalfCheetah, Regrading RobustZero/wl, we maintain wl = Regarding RobustZero/u2, the settings of Of note, RobustZero/cl, RobustZero/w 1, and RobustZero/w2 can obtain the optimal natural reward on CartPole. To better analyze the effects of the sulf-ssper representation network and the adaptive adjustment mechanism, do not account for the corresponding natural rewards CartPole in following statistics. Next, compared to RobustZero, we have the following statistic results: 1) RobustZero/cl decreases the natural reward by 2.65%-61. 79%, the worst-case reward by 4.0% 61. 24%, and the random-cass reward by 1.2%-51, 51.44%; 2) RobustZero/w decreases the natural reward by 0.7%-18..... 18.28% the worst-case reward by 1.2%%42.7995 and the random case reward by 0.8%-13.66 3) RobustZero/w2 decreases the natural reward by 99%-36.65% the worst-case reward by 1. 15...%.5.4% and the random-case reward by 2.6%-33.44%% 4) RobustZero decreases the natural reward by 7.8%%67.32%% the worst-case reward by 82%, and the random-case casoo reward by 9.6%-61.54%; and 5) RobustZero- 3.6%-49.1394 19,1%% the worst-case reward by 1.7%%77..94% and the random-case reward by 6.3%-57.5%. These results consistent to those provided in Section 6.3 This further verifies that the absence of any of the self-supervised representation network, the adaptive adjustment mechanism, and the parallel use of worst-case and random-case buffers will diminish the robustness of RobustZero.
and Ant are reported in Table 4
w2 is provided in Appendix
decreases the natural reward by
C.5. Performance against Unforeseen Perturbation Policies
In order to assess the generalization capability to unforeseen perturbations policies, we evaluate the performance of RobustZero, all baselines, and all variants of RobustZero to handle two additional perturbation policies (including BA- DDPG (Russo& & Proutiere| (02T) and random perturbation based on Gaussian noise (RP-G)) that are not included during training. BA-DDPG a kind of worst-case perturbation policy, while RP-G is a kind of random-case perturbation policy that follows Gaussian distribution (O, e/3). We adhere the original neural network architectures and parameters to implement BA-DDPG across the eight environments. The results are listed in Table and T and We have the following
15

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table 4. Ablation study on CartPole, IEEE 123-bus, Highway, Intersection, Hopper, Walker2d, HaliCheetthh and Ant.
CartPole
IEEE 123-bus
Method
Natural Reward
Worst-case Reward
Random-case
Natural
Worst-case
Random-case
Reward
Reward
Reward
Reward
RobustZerofcl
500.00 = 0.00
374.15 + 12.04
460.24 = 100.12
-12.12 1 1.00
-15.96
1.16
-15.00
2.12
RobustZero/wl
500,00 = 0.00
396.30 + 50.42
480.12 + 80.56
-11.04 + 1.21
-14.11 = 1.20
-13.38 +: 2.06
RobustZero/w2
500.00 = 0.00
388.23 + 30.42
470.94 + 28.46
-11.77 + 1.06
-15.63
1.18
-14.69
2.10
RobustZero-
452.00 7.00
369.70 + 76.00
412.00 + 20.00
-14.01 + 1.02
-16.48
2.10
-15.85
2.00
RobustZero- Drc
463.70 763.70- = 60.00
360.56 + 62.12
435. 70 = 80.12
-13.87 + 1.01
-16.53
= 1.30
-15.49
1.92
RobustZero
500.00 1 0.00
460.56 1 56.33
490.06 5.15
-10.48 + 1.19
11.25
1.43
-12.32 2.00
Highway Worst-case
Intersection
Method
Natural Reward
Random-case Reward
Natural Reward
Worst-case Reward
Random-case
Reward
Reward
RobustZcro/cl
20.03 = 1,40
17.97 + 1.25
19.23 = 2.09
2.12 = 0.36
1,78 = 1.45
2,08
0.30
RobustZerofut
22.12 + 1.42
19.42 + 1.30
21.05 + 2.07
2.90 + 0.32
2.01 + 1.40
2.81
0.31
RobustZero/u22
21.07 + 1.36
18.69 + 1.25
20.38 +: 2.10
2.51 +: 0.28
1.85
1.34
2.36
0.33
RohustZero-Dwe
18.92 + 1.88
18.84 + 2.39
18.13 + 1.91
2.05 + 0.35
1.70 + 1.11
1.95
0.34
RobustZero-DRe
19.36 = 1.31
18.31 + 1.09
18.90 + 1.52
2.30 + 0.31
1.65 + 1.20
2.00 + 0.52
RobustZero
23.24 = 1.32
20.95 + 1.26
22.09 - 1.42
3.43 + 0.26
2,87 = 1.48
3.15 0.34
Hopper
Walker2d
Method
Natural Reward
Worst-case Reward
Random-can case
Natural Reward
Worst-case Reward
Random-can case
Reward
Reward
RobustZero/cl
3561 + 133
2478 + 137
3511 + 227
5377 + 162
4998 + 328
5007 + 295
RobustZero/ul
3634 = 128
2546 + 119
3524 + 227
5986 + 224
5592 = 379
5829 + 232
RobustZero/w2
3592 = 126
2538 + 133
3463 + 241
5157 + 178
4991 = 304
5252 +: 214
RobustZerr- Dwc
3396 = 122
2501 + 123
3241 + 234
4297 + 244
4738 = 214
5046 + 343
RobustZero-DD
3534 = 121
2462 + 140
3343 + 269
4436 + 345
4886 317
4891 + 433
RobustZero
3660 = 132
2577 + 126
3553+t 242
6384 + 230
6096 = 350
6322 + 320
HalfCheetah
Ant Worst-case Reward
Method
Natural Reward
Worst-case Reward
Random-case Reward
Natural Reward
Random-case Reward
RobustZero/el
6080 + 219
4746 1 164
5165 1 173
4743 1 144
3852 = 251
4513 1: 204
RobustZero/ul
6891 = 209
5006 + 138
5628 = 198
5467 = 131
3945 = 256
4992 + 198
RobustZero/w2
6486 = 224
4620 + 150
5510 = 187
4959 + 116
3789 = 271
4938 202
RobustZero- Dwe
5636 = 217
4551 + 158
4815 - 178
4803 - 130
3699 = 215
4392 + 221
RobustZero-Deg
6201 + 230
4507 + 167
5431 + 177
4351 + 126
3580 = 249
4411 + 214
RobustZero
7100 + 232
5395 + 162
6394 + 192
5782 + 130
4686
250
5692 +: 220
observations: 1) RobustZero achieves a higher worst-case reward than all baselines and RobustZero variants across all environments. Specially, RobustZero increases the worst-case reward by 7.8%-119.85%, on CartPole, by 2.32%-76.31 on Pendulum. by 2.78%-101. 901% on IEEE bus, by 6.33%-99 0116999.3195 1% on IEEE bus. by 2.04%-34.064% on IEEE 8500, Intersection, by
Racetrack, by 2.5%-80.0% Walker2d, by 3.9%-1 14.0% on HalfCheetah, and by 2.2%-8877.5% on Ant; and 2) RobustZero achieves higher random-case reward than all baselines and RobustZero variants across all environments, increasing the random-cass case reward by 1.64%-33.7 33.78% on CartPole, by 0.88% 188%-1001 100.88% on Pendulum, by 6.31 231%-6...22 67.02% on IEEE 8500, by 4.39% 23.49% on Highway, on Racetrack, by 2.5%-80.0% on Hopper, by 1.3%-583.2 on Walker2d, by 3.9%-1 14.0% on HalfCheetah, and by 2.2%-817.5 on Ant. Thus, RobustZero still outperforms baselines and RobustZero variants to unforeseen perturbations policies.
by 3.33%-29.3 29.35% Highway, by 36.65% 86.42% on Hopper, by 1.3%-583.3%%
on IEEE 34-bus, by 7.17%-57.009 on IEEE 123-bus, by 3.9%-32.99 22.92%
by 10.88%-8..8 (166-88.4499 48% on Intersection, by 8.93%-35. 109%--5.6999 5..
C.6. Time and Sampling Efficiency Analysis
We measure the average training time per iteration (TT), sampling time per step (ST), testing time per step (TeT), sampling time per episode (STE), and the number of samples per episode (NSE). The results provided in Table 7 We report the relationship between the number of environment samples and the natural, worst-case, and random-case rewards for RobustZero and the two robust model-free baselines: ATLA-PPO and PROTECTED. The results are presented in Table and Table 9 Of note, the number of samples per episode in RobustZero differs from those used in the two baselines. Therefore, while the numbers of samples reported in Table and Table are are similar across methods, they are not exactly the same. As observed from Table 7 ATLA-PPO and PROTECTED exhibit similar TT, ST, TeT, and STE across all
16

---

RobustZero:
Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table Performance comparison of different methods against unforeseen perturbation policies CartPole, Pendulum, IEEE 34-bus, IEEB 123-bus, IEEE node, Highway, Intersection, and Racetrack.
CartPole
Pendulum
Method
Worst-case Reward
Random case Reward
Worst-case Reward
Random-case Reward
(BA-DDPG)
(RP-G)
(BA-DDPG)
(RP-G)
ATLA-PPO
430.26 + 12.78
429.00 = 100.00
.93.96 61.49
.91.00 70.00
PROTECTED
441.39 72.94
440.10 + 90.00
-90.38 32.28
-90.10 55.00
S-MuZero
216.43 + 17.32
370.00 + 66.00
-150.32 + 89.46
-161.38 + 55.00
S-MuZcro-worst
431.26 + 66.89
417.11 + 20.00
-88.48 + 42.48
-168.24 + 62.00
S-MuZero-random
322.64 1:644:30 38.49
460.78 + 90.00
-134.79 + 48.51
-110.98 = 98.00
RobustZero/cl
185.76.199..
472.12 1 90.16
-89.61 59.49
-86.12 58.89
RobustZero/w1
411.43 + 44.87
487.00 + 88.18
-87.24 = 58.11
-84.49 53.62
RobustZero/w2
405.39 + 29.37
475.30 + 30.00
-88.49 + 46.32
-85.29 57.68
RobustZero-
383.24 + 63.98
419.48 = 19.32
-0.122 51.36
-89.0 56.00
RobusiZero-DRc
379.42 1 45.94
451.30 + 80.00
91.59 58.29
-88.06 55.75
RobustZero
475.82 39.28
495.00 6.00
-85.26 36.24
-83.75 55.15
IEEE 34-bus
IEEE 123-bus
Method
Worst-case Reward
Random-case Reward
Worst-case Reward
Random-case Reward
(BA-DDPG)
(RP-G)
(BA-DDPG)
(RP-G)
ATLA-PPO
113.23 0.14
13.15 = 0.04
-14.02 = 0.10
14.33 = 0,01
PROTECTED
-12.19 + 0.25
-12.20 + 0.12
-13.94 + 0.13
-13.26 + 0.04
S-MuZero
-23.84 4.53
-18.79 2.00
-26.13 + 4.93
-19.28 + 3.42
S-MuZerroworst
-14.92
-15.39 1.90
-15.66 + 2.60
-18.51 + 2.61
S-Muuero-random
1.83
-14.12 - 0.30
-2.433 + 2.18
-16.62 + 3.49
RobustZero/cl
+1.844
-13.81 0.59
-15.86 0.95
-14.89 2.39
RobustZero/v1
+13.14+1.11
-11.96 0.45
-13.96 + 1.95
-13.16 2.40
RohustZero/w2
-14.02
-13.23 0.68
-15.33 + 1.68
-14.23 2.38
RobustZero-DD Dwc
-14.43 0.96
14.56 0.51
-15.87 + 1.07
-15.68 + 2.51
RobusiZero-Drc
14.75 0.87
14.03 + 0.60
-16.24 1 1.78
-1.20 1 2.50
RobustZero
-1.86 0.81
-1.25 0.30
-13.11 1 1.74
-1.28 1 2.96
IEEE 8500-node
Highway
Method
Worst-case Reward
Random-case Reward
Worst-case Reward
Random-case Reward (RP-G)
(BA-DDPG)
(RP-G)
(BA-DDPG)
ATLA-PPO
-1392
-1288 7
18.79 0.21
18.87 0.05
PROTECTED
-1351 1 14
-1253 1
19.48 1 0.18
19.94 0.06
S S-MuZero
-1775 + 23
-1603 + 15
16,56 + 3.98
17.92 = 2.80
S-MuZero-wond worst
-1452
$1507
18.68 = 2.73
18.02 1.72
MuZero-random
-1473
5
17.94 4.45
18.64 2.00
RobustZero/cl
-1502 21
-1434
18.83 + 1.02
19.62 1.32
RobustZero/v1
-1445 38
1334
20.73 + 1.56
21.20 1.36
RobustZero/w2
-185 + 29
-1389 25
18.74 + 0.92
20.59 1.24
RobustZero
-1566 41
-1551
18.96 1.85
18.53 1.02
RohustZZero-DRe
-1612
-492 +
18.54 + 1.43
19.08 1.60
RobustZero
-1324 + 26
-206 + 27
21.42 + 1.35
22.13 2.10
Intersection
Racetrack
Method
Worst-case Reward (BA-DDPG)
Random-case Reward
Worst-case Reward
Random-case Reward
(RP-G)
(BA-DDPG)
(RP-G)
ATLA-PPO
1.81 +: 0.36
1.95 : 0.06
392.62 + 0.11
400.11 = 0.03
PROTECTED
2.11 0.31
2.36 0.14
433.17 4.71
463.95 6.00
S-MuZero
1.62 0.32
1.73 + 0.12
365.82 6.54
376.12 + 4.00
S-MuuZerowworst
2.04 0.58
1.92 + 0.44
418.88 + 5.94
401.49 + 6.38
S-MuZero-random
2.06 1 0.14
2.26 1 0.32
403.42 1 9.16
426.10 1 6.00
RobustZeroici
1.88 1.32
0.36
368.84 1.02
389.57 2.12
RobustZero/wl
2.21 - 1.96
2.94 + 0.49
421.38 1.15
450.75 - 2.06
RobustZero/u2
2.04 1.32
2.46 1 0.27
393.46 1.03
429,30 + 2.14
RobusiZero-Dwe
1.93 4: 1.36
2.11 + 0.31
371.96 + 1.54
372.48 4: 1.65
RobusiZero-DRc
1.84 +: 1.26
2.18 + 0.60
362.39 1.29
381.90 1. 2.00
RobustZero
3.02 0.95
3.26 0.38
481.20 1.00
505.40
environments, while S-MuZero, S-MuZero-wors S-MuZerr-random. and RobustZero show similar TT, ST, TeT, and STE within their group. ATLA-PPO and PROTECTED are model-free DRL methods, which generally suffer from low sampling efficiency, as indicated by their significantly higher NSE, meaning they require a larger number of interactions
17

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table Performance comparison of different methods against unforeseen perturbation policies Hopper, Walker2d, HalfCheetah, and
Ant.
Hopper
Walker2d
Method
Worst-case Reward
Random-case Reward
Worst-case Reward
Random-case Reward
(BA-DDPG)
(RP-G)
(BA-DDPG)
(RP-G)
ATLA-PPO
1744
3187 =
3695 - 746
746
3928 276
PROTECTED
2546 + 51
3567 + 43
6055 + 632
6176 = 328
S S-MuZero
1462 9
2473 = 105
898 = 831
2635 367
S-Muzerooworst
1826 1
1770 + 64
1517 +1: 549
1444 249
S-MuZero-random
1742 +: 50
2607 31
1169 912
2658 299
RobustZero/cl
2497 7777
3541 = 181
5039 = 193
5051 = 188
RobusiZerr//e1
2568 + 153
3543 + 180
5633 318
5880 = 176
RobustZero/uc2
2553 + 117
3457 + 169
5034 + 171
5304 187
RobustZero-Dww
2524 +: 167
3267 + 209
4768 + 394
5090 145
RobustZero-DRcc
2478 + 144
3374 + 204
4926 + 265
4937 238
RobustZero
2632 + 104
3593 + 192
6136 + 264
6369 299
HalfCheetah
Ant
Method
Worst-case Reward (BA-DDPG)
Random-cast Reward
Worst-case Reward
Random-case Reward
(RP-G)
(BA-DDPG)
(RP-G)
ATTA--POO
5145 + 256
6170 + 227
3815 125
5384 603
PROTECTED
5229 + 251
6293 + 269
4616 109
5630
S-MuZero
2538 +: 257
3266 + 257
514 = 150
3760 + 431
S-MuZero-worst
2844 +: 297
2803 +: 226
1436 + 286
1341 + 699
S-MuZorrorandoo
2706 + 222
3599 + 144
812 = 217
3846 672
RobustZero/cl
4788 1 221
5200 1 186
3884 215
4547
RobustZero/wl
5043 1 329
5679 1 236
3983 1 271
5034 239
RobustZero/u2
w2
4651 + 206
5550 + 147
3821 + 240
4973 = 206
RobustZero-Dwe
4583 + 228
4852 + 160
3731 = 380
4433 156
RobustZero-Dae
4537 202
5464 + 297
3608 288
4448 233
RobustZero
5432 + 154
6404 + 189
4716 232
5712 219
with environments. In contrast, RobustZero is model-based DRL methods, characterized by higher sampling efficiency. From Table 8 and Table by using similar samples, RobustZero achieves higher rewards compared to ATLA-PPO and PROTECTED, which further demonstrates its superior sample efficiency. The higher sample efficiency of RobustZero makes it more suitable for real-worid applications. Further, MuZero-class methods trade time efficiency for accuracy of decision-making (high rewards), by leveraging the learned models and MCTS to plan ahead. This explains why TT, ST, TeT, and STE for ATLA-PPO and PROTECTED are lower than those for S-MuZero, S-MuZero-wons RobustZero. However, it important to note that TT and STE for all methods remain within a comparable range, indicating that the time efficiency of S-MuZero, S-MuZero- worst, S-MuZero-randon and RobustZero does not significantly diminish when compared with ATLA-PPO and PROTECTED. On the contrary, S-MuZero, S-MuZero-wond RobustZero methods offer greatly improved sampling efficiency, requiring fewer samples compared to ATLA-PPO and PROTECTED. Additionally, TT, ST. TeT, and STE of RobustZero are slightly higher than those of S-MuZero, S-ar,o,SSM S-MuZero- worst, and L-Muuerr-rrndoo due to the integration of self-supervised representation learning and an adaptive adjustment mechanism, which requires additional computational resources. Nevertheless, RobustZero still maintains comparable TT, ST, TeT, and STE, while demonstrating much stronger robustness to worst-case and random-case perturbations.
S-MuuZrrorraddm. and
S-MuZero-rande -anddom,,,,,m, and
C.7. Hyperparameter Analysis
We study how the two hyperparameters A and Ag affect the adaptive adjustment process.
Impact of A Fig. 3 presents the average natural, worst-case, and random-case rewards of RobustZero with \ ranging from 0.25 1.5. For sake of distinction, we reuse B1-worst-case reward and B2-worst-case reward to represent the worst-case rewards obtained under the perturbation policies by using ATLA-PPO and BA-DDPG, respectively, Similarly, we reuse U-random-case case reward and G-random-can reward to represent the random-case rewards obtained under the perturbation policies by using Uniform noise and RP-G, respectively. We observe that the natural reward shows irregular changes, the worst-case reward increases as A₁ increases, and the random-case reward decreases as λ increases. This is because A controls the rate of change of w1, (see Eq. Z which does not impact the natural reward directly. Moreover, for the same Egr. an increase in A₁ reduces the weight of the random-case loss term, i.e., wlp. This, in turn, decreases the ability to defend against such state perturbations. Thus, the random-case reward decreases,
18

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table 7. Performance metrics across all environments.
CartPole
Pendulum
Method
TT
ST
Tel
NSE
STE
NSE
ATLA-PPO
0,4051
0.0036
0.0034
7.3728
2048.0
0.3881
0.0027
0.0025
5.5296
2048.0
PROTECTED
0.3693
0.0028
0.0025
7.7344
2048.0
0.3382
0.0023
0.0020
4.7104
2048.0
S-MuZero
0.4748
0.0357
0.0321
7.4599
208.4
0.4322
0.0914
0.0823
12.9971
142.2
S-Muzero-worst
0.4754
0.0396
0.0356
8,3912
211.9
0.4470
0.1032
0.0929
15.7277
152.4
S-MuZeno-random
0.4764
0.0423
0,0381
8,9634
211.9
0.4526
0.1043
0.0939
16,2812
156.1
RobusiZero
0.4972
0.0464
0,0418
9.9992
215.5
0,4551
0.1
0.0992
17.6320
160.0
IEEE 34-bus
IEEE 123-bus
Method
TT
ST
TeT
NSE
STE
NSE
ATLA-PPO
0.3823
0.0079
0.0071
16.7524
2048.0
0.3911
0.0131
0.0118
26,8363
2048.0
PROTECTED
0.3432
0.0077
0.0067
15.7696
2048.0
0.3683
0.0127
0.0115
26.0096
2048.0
S-MuZero
0.5746
0.2031
0.1828
15.5778
76.7
0.5848
0.7226
0.6503
54.7008
75.7
S-Muzeeroworst
0.5773
0.2112
0.1901
16,2835
77.1
0.5869
0.7466
0.6719
57.1896
76.6
S-Muzeno-randdm
0,5792
0,2182
0.1964
16.8232
77.1
0.5893
0.7492
0.6743
57.3887
76.6
RobusiZero
0.5854
0.2236
0.2012
17.3514
77.6
0.5951
0.7582
0.6824
58,8363
77.6
IREE 2o0e
Highway
Method
TT
ST
TeT
STE
NSE
TT
ST
TeT
STE
NSE
ATLA-PPO
0.5012
0.02T0
0.0191
43.0080
2048.0
0.42088
0.0092
0.0085
18.8416
2048.0
PROTECTED
0.4845
0.0201
0.0181
41.1648
2048.0
0.4116
0.0089
0.0080
18.2272
2048.0
S-MuZero
0.5983
1.0021
0.9019
76.6606
76.5
0.5725
0.3057
0.2751
23.6612
77.4
S-Muzero-worst
0.6027
1.0355
0,9302
79.6334
77.1
0.5736
0.3120
0.2808
24.2112
77.6
S- S-Muzeno-random
0.6011
1.0342
0,9308
78.2158
75.6
0.5758
0.3135
0.2822
24.4217
77.9
RobustZero
0.6120
1.0718
0.9646
81.7577
76.3
0.5814
0.3218
0.2896
25.1326
78.1
Intersection TeT
Racetrack
Method
TT
ST
STE
NSE
TT
ST
TeT
STE
NSE
ATLA-PPO
0,4604
0.0115
0.0108
23.5520
2048.0
0.4104
0.0080
0.0074
16.3840
2048.0
PROTECTED
0.4415
0.0112
0.0101
22.9376
2048.0
0.4015
0.0079
0.0071
16.1792
2048.0
S-MuZero
0.5814
0.3848
0.3463
30.0145
78.0
0.5623
0.2856
0.2570
21.9916
77.0
S-MuZero-worst
0.5933
0.3910
0.3519
30.5918
78.2
0.5644
0.2903
0,2613
22.6455
78.0
S-MutZerorrnnon
0,5925
0.3922
0.3530
30.4982
77.8
0,5632
0.2915
0.2624
22.5532
77.4
RobustZero
0.6038
0.4015
0.3614
31.1163
77.5
0.5809
0.3038
0.2734
24.0248
79.1
Hopper
Walker2d
Method
TT
ST
TeT
STE
NSE
TT
ST
TeT
STE
NSE
ATLA-PPO
0.3010
0.0014
0.0011
2.8769
2048.0
0.5098
0.0022
0.0019
4,4901
2048.0
PROTECTED
0.3225
0.0013
0.0009
2.7684
2048.0
0.4795
0.0018
0.0016
3.6420
2048.0
S-MuZero
0.3949
0.0738
0.0685
11.1581
151.3
0.5471
0.1126
0.1034
17.1104
152.0
S-Muzero-worst
0.4066
0.0767
0.0742
11.6543
152.0
0.5560
0.1195
0.1086
18.2534
152.8
S- S-MuZero-random
0.4022
0.0751
0.0733
11.4850
151.8
0.5584
0.1213
0.1105
18.4677
152.3
RobustZero
0.4168
0.0816
0.0798
12.3549
151.5
0.5678
0.1325
0.1210
20.2796
153.1
HalfCheetah
Ant
Method
TT
ST
TeT
STE
NSE
TT
ST
TeT
STE
NSE
ATLA-PPO
0.5106
0:0023
0.0021
4.5825
2048.0
0.5489
0.0029
0.0027
5.9043
2048.0
PROTECTED
0.4821
0.0019
0.0017
3.8054
2048.0
0.5374
0.0028
0.0026
5.6511
2048.0
S-MuZero
0.5517
0.1146
0.1082
17.5410
153.1
0.5712
0.1741
0.1609
26,7350
153.6
S-Muzero-worst
0.5608
0.1260
0.1168
19.3452
153.5
0.5810
0.1764
0.1668
27.1345
153.8
S- S-MuZZro-randon 0.5647
0.1287
0.1175
19.7628
153.6
0.5847
0.1798
0.1679
27.6278
153.7
RobustZero 0.5782
0.1376
0.1232
21.1976
154.0
0.5917
0.1850
0.1706
28.4967
154.1
while the worst-case reward increases.
Impact of Az. Fig. reports rewards obtained by RobustZero when varying X2. We observe that all rewards first increase and then decrease as X2 decreases. To understand why, note that X2 is used to control the ratio of the magnitudes between w2r0p-1 and V Lo, + w1g V Lo,,,, (see Eq. 9). On the one hand, a too large À means that w2..6p-1 maintains a high value during training. This will greatly reduce the impact of the worst-case and random-case loss terms, resulting in reduced natural, worst-case, and random-cas case rewards. On the other hand, if X is too small, w2y remains small. This leads reduced adjustment functionality of w2r,, which also results in reduced natural, worst-case, and random-case rewards. Based these analyses, the settings of both large and small Xg are not beneficial for achieving high rewards.
19

---

RobustZero:
Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table The relationship between environment samples and rewards CartPole, Pendulum, IEEE 34-bus, TEEE 123-bus, IREF 8500
node, Highway, Intersection, and Racetrack.
Pendulum
Cartpole
Method
Natural
Worst-case
Random-case
Natural
Worst-case
Random-case
Samples
Samples
Reward
Reward
Reward
Reward
Reward
Reward
73,776
-10.48
-16.61
-10..49
69,632
474.60
438.09
471.84
RobustZero
147,552
-89.98
111.00
-89.03
40,261
496.09
456.32
487.27
221,328
-83.99
-88.15
-84.98
210,891
500.00
459.33
489.56
73,728
-122799
167.442
-100990
69.632
384.66
325.31
390.76
ATLA-PPO
147,456
-92.02
-95.18
-95.64
40,960
474.60
356.09
371.84
221,184
-92.18
-95.34
-94.62
212,992
477.94
369.32
375.46
73,728
-119.59
-177.80
-00.336
69,632
457.19
376.10
380.54
PROTECTED
147,456
-94.33
-04.32
1000.79
40,960
462.52
428.36
418.76
221,184
-93.86
-99.32
-96.52
212,992
465.86
421.69
422.09
IEEE 34-bus
IEEE 123-bus
Method
Natural
Worst-case
Random-case Reward
Natural
Worst-case
Random-case
Samples
Samples
Reward
Reward
Reward
Reward
Reward
177.383
-2I.29
29.44
-21.38
198,042
-25.29
-20.89
-24.74
RobustZero
354,765
-13.31
-18.83
-17.79
396,084
-13.50
-16.39
-14.72
532,148
-9.78
-12.88
-11.76
594,128
-10.99
-13.85
-12.95
178,176
-25.47
-26.08
-23.92
198,656
-36300
-22.56
-25,44
ATLA-PPO
354,304
-21.29
-24.44
-28.38
395,264
-25.29
-20.89
-22.74
532,480
-17.16
-18.31
-15.56
594,128
-21.32
-18.16
-18.44
178,176
-21.19
20.551
-30.33
198,656
-34.85
-26.81
-28.62
PROTECTED
354,304
-13.80
-16.29
-24.99
395,264
-29.93
-22.21
-27.11
532,480
-12.62
-15.88
-18.99
594,128
-19.93
-15.21
-17.11
IEEE 8500-node
Racetrack
Method
Natural
Worst-case
Random-case
Natural
Worst-case
Random-case
Samples
Samples
Reward
Reward
Reward
Reward
Reward
Reward
271.413
-1300
-167
-1421
138,064
498.65
445.35
470.21
RobustZero
542,827
-1255
-1475
-1315
276,129
507.36
463.20
479.19
814,240
-1175
-1386
-1225
414,193
519.30
473.22
485.30
272.384
1454
-1769
-1506
137,216
364.74
326.57
348.37
ATLA-PPO
542,720
-1350
-1667
-1421
276,480
385.72
349.71
363.36
815,104
-1358
-1574
-1417
413,696
381.32
361.84
371.84
272,384
-1763
-1888
-1344
137,216
374.17
338.74
349,86
PROTECTED
542,720
-1457
-1671
-1427
276,480
411.04
354.74
359.71
815,104
-1358
-1574
-1417
413,696
417.32
361.28
371.28
Highway
Intersection
Method
Natural
Worst-case
Random-cass
Natural
Worst-casc
Random-case
Samples
Samples
Reward
Reward
Reward
Reward
Reward
Reward
71,594
15.42
15.38
13.77
127,725
2.47
2.35
2.41
RobustZero
143,188
21.98
17.15
17.02
255,449
2.86
2.44
2.66
214,782
23.10
20.33
21.95
383,174
3.21
2.75
3,01
71,680
13.31
12.98
6.86
126,976
1.62
0.96
1.02
ATLA-PPO
143,360
15.66
14.54
10.86
256,000
1.97
1.28
1.61
215,040
16.23
15.98
13.33
382,976
2.03
1.33
1.63
71,680
11.19
14.37
15.07
126,976
1.11
0.86
0.93
PROTECTED
143,360
13.02
15.69
16.75
256,000
1.65
1.30
1.36
215,040
13.63
17.38
18.13
382,976
2.18
1.38
1.62
C.8. Impact of The Attack Radius
We evaluate the performance of RobustZero and all baselines under varying values of attack radius c. RobustZero does not know the attack budget during training. Figs. 5 illustrate the changes in natural rewards, B1-worst-case rewards, B2-worst-case rewards, U-random-case rewards, and G-random-case rewards as the increase
across all environments, The key observations are as follows: 1) The natural rewards of S-MuZero remain unaffected by changes in c, as it is trained without state perturbations. Nevertheless, its B B1-worst-case rewards, B2-worst-case rewards, U-random-case rewards, and G-random-case rewards decrease dramatically. For all other methods, natural rewards decrease increases. Note that RobustZero achieves higher natural rewards than ATLA-PPO, PROTECTED, S-MuZero-worst and S-Muzerr-random across all e settings and environments. The exception is on CartPole, where RobustZero, ATLA-PPO, PROTECTED,
20

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Table 9. The relationship between environment samples and rewards Hopper, Walker2d, HalfCheetah, and Ant.
Hopper
Walker2d
Method
Natural Reward
Worst-case
Random-case
Natural
Worst-case
Random-case
Samples
Samples
Reward
Reward
Reward
Reward
Reward
333,521
2937
2066
2848
661,089
5150
4957
5112
RobustZero
667.042
3418
2424
3293
1,322,179
5902
S717
5912
1,000,564
3610
2554
3517
1,983,269
6308
6075
6287
331,776
2164
1136
2079
661,504
2556
2393
2507
ATLA-PPO
665,600
2802
1497
2696
1,323,008
3344
3180
3294
999,424
2945
1544
2824
1,982,464
3483
3281
3465
331,776
2389
1658
2303
661,504
4158
3972
4070
PROTECTED
665,600
3175
2198
3050
1,323,008
5562
5210
5315
999,424
3280
2267
3182
1,982,464
5738
5413
5540
HalfCheetah
Ant
Method
Natural
Worst-case
Random-case Reward
Natural Reward
Worst-case
Random-case
Samples
Samples
Reward
Reward
Reward
Reward
710,936
5711
4357
5099
800,568
4670
3789
4591
RobustZero
1,421,872
6591
5024
5883
1,601,137
5345
4390
5325
2,132,808
7060
5345
6320
2,401,706
5721
4636
5615
710,656
4030
3346
3987
800,768
3486
2438
3473
ATLA-PPO
1,421,312
5338
4407
5310
1,601,536
4628
3246
4620
2,131,968
5561
4616
5500
2,402,304
4802
3365
4786
710,656
4675
3375
4090
800,768
3799
3010
3696
PROTECTED
1,421,312
6090
4428
5379
1,601,536
5033
3932
4857
2,131,968
6377
4641
5652
2,402,304
5200
4107
5073
Natural reward
Hl-worst-ccse reward
82-worst-aase reward
U-mandott-co case reward
G-random-case reward
500
-84
-10
-11
Reward
480
-86
-11-
12
-12-
-13
888
0.25
0.5
1.25
1.5
0.25
0.5
1.25
1.5
0.25
0.5
1.25
1.5
0.25
0.5
1.25
A1
À1
A1
(a) CartPole
Pendulum
(c) LEBE 34-bus
(4) JEEE 123-bus
3.4
520
-1200
Roward
22
500
3.0-
480
0.25
0.5
1.25
1.5
0.25
0.5
1.25
1.5
0.25
0.5
1.25
0.25
0.5
1.25
A
A
A
A
(c) JEEE 8500-node
(1) Highway
(g) Intersection
Racetrack
6400
Roward
5500
3000
6200
6000
0.25
0.5
1.25
1.5
0.25
0.5
1.25
1.5
0.25
0.5
1.25
1.5
0.25
0.5
1.25
A1
A1
A1
(i) Hopper
Walker2d
(k) HallCheetah
(1) Ant
Figure 3. Performance of RobustZero with different A₁ across all environments.
and S-MuZero consistently attain optimal natural rewards, regardless of e; and 2) B1-worst-case rewards, B2-worst-case rewards, U-random-case rewards, and G-random-case rewards for all methods decline as increases. This indicates that larger attack radius weaken the defense capabilities of all methods to state perturbations. This is because increases, attacks become stronger. This further increases the difficulty defend against such attacks. Despite this, RobustZero consistently outperforms all baselines, which achieves the highest worst-case and random-cass rewards across all values and environments. These results imply that RobustZero is consistently the best option across all settings.
21

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Natural reward
case reward
B2 O2-worst-coe reward
U-random case reward
G-random case reward
-84
-10
-11
Reward
-86-
88
-13
460
-12-
90
-14
16-5
So-6
1o-6
16-6
5o-7
10-7
So-8
10-8
Ag
A2
A
A2
(a) CartPole
(b) Pendulom
(c) IEEE 34-bus
(d) IEEE 123-bus
23
5.0
11200
Reward
500
-1300
3.0
480
-1400
2,5
460
50-7
1o-7
1044
56-5
16-5
1o-4
5o-5
16-5
50-6
16-4
5o-5
1e-5
50-6
Àp
A
A2
IEEE 8500-node
(1) Highway
Intersection
(b) Racetrack
6400
3500
Roward
6200
3000
5000
6000
2500
4500
10.5
30-6
16.6
So/7
1e-7
16.5
36-6
1e-6
Se.7
1o-7
36.6
16.0
5e-7
le:7
5e.8
36-6
16.6
Se-7
1o-7
5e-8
A2
A2
A2
Hopper
Walker2d
(k) HalfChectah
Figure 4. Performance of RobustZero with different X across all environments.
RebusiZao
ATLA-PPO
PROTECTED
S-MuZeso
S-MuZcro-wust
S-Mizenn-nadom
-10
-00
15
0.10
005
0,10
610
0.15
0.8
(s) CariPole
Pendulum
IEBE 34-bus
(4) IEBE 123-bus
Roward
200
11500
8.6
6
618
8.20
0.25
005
618
835
0.28
0,25
008
618
0,5
8.25
0.06
8.18
8.5
(e) IEEE 8500-node
(i) Highway
(g) Intersection
(h) Racetrack
6000
3000
60000
Roward
4000
1000
2000
2000
0.035
0.075
0125
2012
0,25
omp
Hopper
Walker2d
HalfCheetah
AD)An
Figure 5. Natural rewards of RobustZero and baselines with different values of across all environments.
22

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
Robaiiaao
ATLA-PP
PROTECTED
S-MuZaro
S-MuZznswuni
5:MiZememmoo
Reward
-100
-10
-10
-200
20
-30
0,10
005
0,88
0.25
016
0.6
0.00
(a] CartPole
(b) Pendulum
(c) IEEE 34-bus
(6) IEBE 123-bus
20
400
Roward
3000
200
-2000
005
0
015
0,5
0,10
0,35
0,5
0.30
0,05
4,30
IEEE 8500-node
Highway
(g) Intersection
Racetmic
Roward
2000
4000
4000-
2000
2000
2000
0,225
0.88
0.075
0.25
00125
0,05
0.075
0.88
0.39
8.68
0,30
Hopper
Walker2a
(k) HalSCheetah
An.Ant
Figure 6. B Bl-worst-case rewards of RobustZero and baselines with different values of across all environments.
RobisiZero
APLA
PROTECTED
Muzero
MuZero wont
S M.Mieerr madom
400-
Reward
10
10
200
-100
20
-20-
0.05
0.10
0.15
0.35
0.85
0.18
0.15
0.5
0.05
0.18
0.25
0.05
0.6
[8] CariPole
(b) Pendulum
(0) IEEE 34-hus
(d) IEBE 123-bus
5000
Reward
1000
10
200
-1500
0
0,15
0,10
0.15
0,8
8.2
008
0.15
(c) IEEE 8500-node
(0) Highway
(g) Intersection
(hi Racetrack
6000
6000
Rewsrd
2000-
40000
1000
2000
2000
0.02
038
0125
8012
008
0.8
Hopper
Walkerza
HalECheetah
Am
Figure B2-worst-case rewards of RobustZero and haselines with different values of across all environments.
23

---

RobustZero: Enhancing MuZero Reinforcement Learning Robustness to State Perturbations
RobusiZero
ATLA
PROTECTED
MuZero
MuZeoro
MiZen nadom
Reward
10
-100
-100
200
20
0.55
0.30
008
0
0.25
005
0.10
0.5
910
3
(4) CariPole
Pandulum
IERE 34-bus
(4) IRBR 123-bus
400
Roward
1000
200
-5500
0,18
6A5
0.5
0,10
0.23
0.89
8.10
0.88
0.19
(c] IEEE 8500-node
(i) Highoay
(g) Intersection
(hi Racetrack
6000
2000
Rowaed
3000
4000
4000
2000
2000
0.03
0:08
0.075
0.
0123
0.02d
0.025
0.08
oms
01
008
0.10
0099
0.10
Hopper
Walker2a
HalaCheetah
Ant
Figure& 8. U-random-case rewards of RohustZero and baselines with different values of across all environments.
RobisiZero
APLA
PROTECTED
MuZero
Muzero wost
S MiZeer nadom
Reward
400
10
10
-109
-00
005
0.10
0.15
0.25
0.85
0.18
0.15
0.35
0.65
8.18
0.15
6.55
0.05
0
0.55
[8] CariPole
(b) Pendulum
(6) IEEE 34-hos
(d) IEBE 123-bus
20
Reward
500
400
10
200
-1500
018
0
0,15
1315
0.25
008
(c) IEEE 8500-node
(0) Highway
(g) Intersection
(hi Racetrack
6000
3000
Rewsed
2000
6000
4000
2000
2000
0.22
038
0.95
0125
80125
0.25
0.8
0.5
010
Hopper
Walkezza
HalECheetah
Am
Figure 9. G-random-case rewards RobustZero and baselines with different values of e across all environments.
24


Reviews:
[{"review_id": "dJ7NN1qXJR", "reviewer": "Reviewer_yCqT", "summary": "The paper introduces RobustZero, an enhanced MuZero framework designed to be robust against both random-case and worst-case adversarial perturbations. RobustZero dynamically balances data generation between these perturbations and incorporates them directly into online training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and our responses are detailed below.\n\n$\\textbf{Response to Method and Evaluation Criteria and Q1}$: We would like to address your comments as follows: \n1) We have studied RobustZero and all baselines on five Mujoco environments, including Hopper, Walker2d, HalfCheetah, Ant and Humanoid. The results are provided in Table S-1 (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Tables/Table%20S-1.png). RobustZero consistently outperforms all baselines at defending against state perturbations. This provides evidence that RobustZero also performs well in general and challenging Mujoco environments. \n2) Following ATLA-PPO and PROTECTED, we adopt an $l_{p}$-norm perturbation model to generate small, semantically invariant perturbations. However, in board games, the states are discrete and highly structured, and even small $l_{p}$-norm perturbations can result in invalid or illegal states that violate game rules. Therefore, this perturbation model is not suitable for board games, and we do not evaluate all methods in such environments.\n 3) We will add the following explanations for the energy and transportation tasks. The three transportation environments support the testing of autonomous driving tasks. Therein, an autonomous driving car interacts with other vehicles to navigate different scenarios: i) Highway$-$Drive fast, avoid collisions, and stay in the right-most lane; ii) Intersection$-$Cross safely, follow traffic rules, and keep a steady speed; and iii) Racetrack$-$Finish quickly while staying on track and driving smoothly. The action space of an autonomous driving car is two-dimensional. The three energy environments support the testing of voltage control tasks with the objective of minimizing the total cost of voltage violations, control errors, and power losses, while meeting both networked and device constraints. The action spaces of IEEE 34-bus, IEEE 123-bus and IEEE 8500-node are 10-dimensional (8 continuous and 2 discrete), 15-dimensional (11 continuous and 4 discrete), and 32-dimensional (22 continuous and 10 discrete), respectively. Thus, these energy environments are complex and high-dimensional. \n\n$\\textbf{Response to Experimental Designs or Analyses}$: We have added these training curves (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Figures/Figure%20S-1.png).\n\n$\\textbf{Response to Q2}$:  Our method involves four parameters: $w1$, $w2$, $\\lambda_1$, and $\\lambda_2$. We clarify their design and selection: 1) One of our contributions is the development of an adaptive mechanism to adjust $w1$  and $w2$, eliminating the need for ad-hoc adjustments. Note that in Appendix C.3, we show the selection of $w2$ across different environments. This is used in ablation studies, where we intentionally fix $w2$ and find best value to isolate its effect and demonstrate the advantage of the adaptive mechanism; 2) $\\lambda_1$ adjusts the trade-off between robustness to worst-case perturbations and random perturbations. A larger $\\lambda_1$ emphasizes worst-case robustness, while a smaller value favors random-case robustness. When both are considered equally important, setting $\\lambda_1=1$ (as done in our experiments) is a reasonable and effective default. Thus, $\\lambda_1$ does not require tuning unless specific emphasis is desired; and 3) Among the four parameters, $\\lambda_2$ is the only one that requires manual tuning. To keep the selection process simple, we employ a standard grid search. Appendix C.7 analyzes the effect of different $\\lambda_2$ values, which is not the selection process. In summary, our method does not rely on ad-hoc adjustments. The adaptive mechanism removes the need to tune $w1$ and $w2$. $\\lambda_1$ can be fixed to a default value, e.g., 1. Only $\\lambda_2$  requires simple tuning via grid search. Therefore, the hyperparameter selection is not complex. \n\n$\\textbf{Response to Essential References Not Discussed}$: We will add the following statements. The contrastive representation learning has been used to improve the sample efficiency [A, B] and enhance the representation ability of states [C], rewards [D], and value functions [E]. Among them, one notable work is EfficientZero [B], which significantly improves the sample efficiency of the MuZero method while maintaining superior performance. It achieves this by using contrastive representation learning to build a consistent environment model and using the learned model to correct off-policy value targets. Different from these studies, we aim to leverage contrastive representation learning to improve the robustness of MuZero-class methods to state perturbations.\n\n[A] Data-efficient reinforcement learning with self-predictive representations\n\n[B] Mastering Atari games with limited data \n\n[C] Planning with goal-conditioned policies\n\n[D] Beyond reward: Offline preference-guided policy optimization\n\n[E] Contrastive learning as goal-conditioned reinforcement learning", "claims_and_evidence": "The claims presented in the paper are supported by extensive empirical evidence.", "methods_and_evaluation_criteria": "I suggest employing more general and challenging benchmarks (e.g., board games, MuJoCo) that are standard in the RL community. Pendulum and CartPole appear overly simplistic. Additionally, detailed explanations of the transportation tasks would help clarify their complexity.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Consider including training curves (e.g., reward vs. training steps plots) to illustrate learning dynamics and convergence.", "relation_to_broader_scientific_literature": "The paper proposes a robust framework addressing sensory noise and adversarial perturbations. Demonstrating effectiveness on more complex tasks could significantly strengthen the proposed method’s relevance. Additionally, the current design involves numerous hyperparameters and ad-hoc choices. Reducing these or providing theoretical justifications would enhance the rigor of the method.", "essential_references_not_discussed": "Given that the paper integrates contrastive representation learning with MuZero, it should further discuss representation learning approaches within the Related Work section, notably EfficientZero (https://arxiv.org/pdf/2111.00210).", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "[Q1] How does RobustZero perform on more complex tasks, such as board games (e.g. Go, Chess, Shogi) and high-dimensional continuous control environments (e.g., MuJoCo Humanoid)?\n\n[Q2] The current method involves numerous hyperparameters and seemingly ad-hoc adjustments. Can you reduce the complexity or provide theoretical justification for these choices?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H79MdPbD9k", "reviewer": "Reviewer_BDsu", "summary": "The authors propose RobustZero, the first MuZero-class method designed to ensure robustness against state perturbations, including both worst-case and random-case scenarios. The proposed method introduces a training framework that includes a self-supervised representation network, which facilitates the generation of consistent policies both before and after state perturbations. The framework also incorporates a unique loss function that enhances the robustness of the training process. Furthermore, the authors present an adaptive adjustment mechanism that allows for model updates, ensuring high robustness to perturbations. Extensive experiments conducted across eight environments provide strong evidence that RobustZero outperforms existing state-of-the-art methods in defending against state perturbations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive and valuable comments.\n\n$\\textbf{Response to Claims and Evidence:}$ Following your comments, we have analyzed the relationship between the number of environment samples and the natural, worst-case, and random-case rewards for RobustZero and the two robust model-free baselines: ATLA-PPO and PROTECTED. The results are presented in Table S-2 (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Tables/Table%20S-2.png). Please note that the number of samples per episode in RobustZero differs from those used in the two baselines (refer to Table 6 in Appendix C.6). Therefore, while the numbers of samples reported in Table S-2 are similar across methods, they are not exactly the same. From Table S-2, by using similar samples, RobustZero achieves higher rewards compared to ATLA-PPO and PROTECTED, demonstrating its superior sample efficiency.\n\n$\\textbf{Response to Method and Evaluation Criteria:}$ Following your suggestion, we have studied RobustZero and all baselines on five Mujoco environments, including Hopper, Walker2d, HalfCheetah, Ant and Humanoid. The results are provided in Table S-1 (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Tables/Table%20S-1.png). The results show consistently that RobustZero outperforms all baselines at defending against state perturbations. This provides evidence that RobustZero also performs well in general and challenging Mujoco environments. \n\n$\\textbf{Response to Experimental Designs or Analyses:}$ To ensure a fair comparison, we follow prior works, i.e., ATLA-PPO and PROTECTED by using ATLA-PPO to obtain a worst-case perturbation policy that is applied to all methods. Regarding robust RL methods, e.g., RobustZero, ATLA-PPO and PROTECTED, the agent is trained by using non-perturbed states and perturbed states information. This enables obtaining consistent policies before and after state perturbations. Regarding S-MuZero-worst, it is an extended baseline, representing S-MuZero trained under the worst-case perturbation policy. Importantly, S-MuZero-worst does not incorporate any robustness mechanism. It is trained solely on perturbed states, without any information about non-perturbed states. This causes S-MuZero-worst to be over-trained under the worst-case perturbation policy, making it highly specialized and adapted to that particular perturbation pattern.  As a result, its performance under the worst-case perturbation policy unusually appears higher than its performance under random or no perturbations as observed in Table 1. This explains why its worst-case reward may appear higher than its random-case reward. Note that S-MuZero-worst is deliberately designed to showcase the effect of over-training under worst-case perturbations, regardless of its natural and other performance. Except for S-MuZero-worst, the worst-case rewards are lower than the corresponding natural and random-case rewards. \n\n$\\textbf{Response to Essential References Not Discussed}:$ We would like to address your comments as follows: 1) Following your suggestion, we will add the two references. Specifically, a robust adversarial reinforcement learning method is proposed [1] to jointly train an agent and an adversary, where the agent aims to accomplish the primary task objectives while learning to remain robust against disturbances introduced by the adversary. A recent study [2] proposes the use of variational optimization over worst-case adversary distributions, rather than a single adversary, and trains an agent to maximize the lower quantile of returns to mitigate over-optimism; and 2) In our paper, a worst-case state perturbation refers to an adversarial modification of the agent’s observed state that is carefully crafted to minimize its expected return. While, a random-case state perturbation refers to a stochastic disturbance applied to the agent’s observed state. Formal definitions of both perturbation policies are provided in Definitions 4.2 and 4.3 of our paper (see pages 3-4). Additionally, we would like to note that for reference [2], only the abstract is currently available, and the full version has not yet been released online. As a result, we are unable to determine the specific settings used in that work regarding worst-case and random-case perturbations.\n\n$\\textbf{Response to Other Comments or Suggestions:}$ We will change “row” to “column”. \n\n$\\textbf{Response to Questions for Authors:}$ S-MuZero can solve the tasks with continuous action spaces. We will add this statement in Section 3.3.", "claims_and_evidence": "The motivation of this work starts from the limitations of both model-free and model-based methods. Therefore, MuZero was proposed to integrate the strengths from both methods. However, it is not guaranteed that RobustZero will inherit same efficiency benefits compared with robust model-free method. In short, learning curve or other metrics should be included to show that the proposed method is more sample efficient.", "methods_and_evaluation_criteria": "The benchmark is not sufficient. Authors only pick cartpole and pendulum from MuJoco while there are more complicated environments in Mujoco. Only evaluating on more complicated environments can convince that pure model-based methods cannot have the access to prior knowledge of the environment’s dynamics", "theoretical_claims": "No theory", "experimental_designs_or_analyses": "How do you train S-MuZero-worst under worst case? Do you assume that you know the worst case scenario for different policies and is the worst case here fixed during training? This also connects to my another question that how can worst-case reward higher than random-case reward in Racetrack for S-Muzero-worst in table 1? I thought worst case means that the performance should always be the worst.", "relation_to_broader_scientific_literature": "Introduce the robustness concept to a blend of model-free and model-based method", "essential_references_not_discussed": "I will suggest the authors to include the following papers, which are also an important branch of robust RL, especially that it is still not clear what the definition of worst case and random case in this paper's setting. It should be clearly discuss and identify the difference with [2]'s worst/random settings.\n\n[1] Lerrel Pinto et al. \"Robust Adversarial Reinforcement Learning\", ICML, 2017\n\n[2] Juncheng Dung et al. \"Variational Adversarial Training Towards Policies with Improved Robustness\", AISTATS, 2025", "other_strengths_and_weaknesses": "Strengths:\n* This paper makes a valuable contribution to the field with promising results.\n* Comprehensive analysis and experiments", "other_comments_or_suggestions": "* In line 363, I thought it is in each column instead of row.", "questions_for_authors": "* Can S-MuZero solve the tasks with continuous action space? If yes, the explicit statement should be mentioned in 3.3 section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jCcsnKy39q", "reviewer": "Reviewer_v7JN", "summary": "This work proposed a robust version of Muzero framework called RobustZero to gain robustness when facing state perturbations. Muzero features a self-supervised representation network to generate a consistent initial hidden state and a unique loss function to gain robustness. In the experiment setting, Muzero shows superior performance under both worst and random state perturbations.\n\n## update after rebuttal\n\nThe rebuttal has adequately addressed most of my concerns, and I am now leaning toward accepting this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments and recognition of our contributions. \n\n$\\textbf{Response to Experimental Designs or Analyses:}$  Please refer to the responses to Q1-Q3. \n\n$\\textbf{Response to Essential References Not Discussed:}$ We will add the two references as follows.  The recent studies [1-2] introduce the diffusion models to enhance robustness to state perturbations. Specifically, a diffusion model-based predictor is proposed [1] for offline RL to recover the actual states against state perturbations. A belief-enriched pessimistic Q-learning method is proposed [2] by using diffusion model to purify observed states. \n\n$\\textbf{Response to Q1:}$ Following your suggestion, we have studied RobustZero and all baselines on five Mujoco environments, including Hopper, Walker2d, HalfCheetah, Ant and Humanoid. The results are provided in Table S-1 (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Tables/Table%20S-1.png). The results show consistently that RobustZero outperforms all baselines at defending against state perturbations. This provides evidence that RobustZero  performs well in general and challenging Mujoco environments. \n\n$\\textbf{Response to Q2:}$ We fully agree with the reviewer that PA-AD (referred to as PA-ATLA-PPO in our paper) is currently considered the strongest attack method. However, as mentioned in Section 2.2, it is a white-box attack, requiring access to the internal parameters of the victim model. In contrast, our work focuses on black-box attack settings, where such access is not available. Therefore, we do not include PA-ATLA-PPO as a baseline in our experiments.\n\n$\\textbf{Response to Q3:}$  Since SA-PPO is a white-box attack strategy, we do not use it as a baseline.\n\n$\\textbf{Response to Q4 and W1:}$  Our method involves four parameters: $w1$, $w2$, $\\lambda_1$, and $\\lambda_2$. We clarify their design and selection: 1) One of our contributions is the development of an adaptive mechanism to adjust $w1$  and $w2$, eliminating the need for ad-hoc adjustments. Note that in Appendix C.3, we show the selection of $w2$ across different environments. This is used in ablation studies, where we intentionally fix $w2$ and find best value to isolate its effect and demonstrate the advantage of the adaptive mechanism; 2) $\\lambda_1$ adjusts the trade-off between robustness to worst-case perturbations and random perturbations. A larger $\\lambda_1$ emphasizes worst-case robustness, while a smaller value favors random-case robustness. When both are considered equally important, setting $\\lambda_1=1$ (as done in our experiments) is a reasonable and effective default. Thus, $\\lambda_1$ does not require tuning unless specific emphasis is desired; and 3) Among the four parameters, $\\lambda_2$ is the only one that requires manual tuning. To keep the selection process simple, we employ a standard grid search. Appendix C.7 analyzes the effect of different $\\lambda_2$ values, which is not the selection process.  In summary, we only need to search for the best setting of $\\lambda_2$  for a new environment. Therefore, the hyperparameter selection is not complex.\n\n$\\textbf{Response to Q5:}$ RobustZero does not know the attack budget $\\epsilon$ during training. We will clarify this statement in the final paper.\n\n\n$\\textbf{Response to Q6:}$ Literature [3] is the first study to design a defense strategy against state perturbations specifically for offline RL. The strength of offline RL lies in its ability to train agents solely from a fixed dataset, without requiring any interaction with the environment. However, this limits its exploration capability, making it more challenging to learn optimal policies. That said, [3] demonstrates strong robustness to state perturbations when compared to other offline RL methods that lack defense strategies. In contrast, RobustZero and all baseline methods evaluated in our work are online RL methods, where the agent learns by actively interacting with the environment. Generally, online RL methods are capable of achieving higher rewards than offline RL methods due to their adaptive exploration. Given the fundamental differences in training protocols and assumptions between offline and online RL, we do not include [3] as a baseline, as it would not constitute a fair comparison under the same experimental setup.\n\n$\\textbf{Response to Q7:}$ In Appendix C.6, we provided comparison analysis for the training time and sampling time. Since the sampling time closely approximates the testing time, we initially omitted testing time results in the submitted paper. Now, we have  added the testing time per step (TeT) and updated Table 6 accordingly (refer to https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Tables/Table%206.png). As shown in the update Table 6, the testing time is similar to the sampling time, the previous analysis of sampling time remains applicable. For a comprehensive comparison, please refer to Appendix C.6.", "claims_and_evidence": "I believe the claims are well supported.", "methods_and_evaluation_criteria": "Yes, the proposed methods gain the best robustness under both worst-case and random perturbation under various environments.", "theoretical_claims": "Yes, I checked the formation of the loss functions proposed by the authors.", "experimental_designs_or_analyses": "The evaluation of the proposed method is similar to previous related work by providing rewards under no attack and rewards under worst and random perturbations. However, I do have some concerns about the experiments. 1. The environments do not include Mojuco, which is a commonly used environment in previous related work. 2. The worst attack is achieved by using ATLA-PPO, why not use PA-AD[1] attack that is currently considered the strongest attack? 3. It would be better to include some classical baselines that are compared in most related work, such as SA-PPO[2] in the experiments.\n\n[1] Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022\n[2] Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations NeurIps 2020", "relation_to_broader_scientific_literature": "This work is related to robust RL, trustworthy AI and AI safety in general.", "essential_references_not_discussed": "I believe there are recent works in this field using diffusion models to gain robustness under state perturbation attacks that should be cited.\n\n[1] DMBP: Diffusion model based predictor for robust offline reinforcement learning against state observation perturbations. ICLR 2024\n[2] Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations. ICLR 2024", "other_strengths_and_weaknesses": "Strengths:\n1. Comprehensive experiment results and alation studies reported by the authors.\n2. The first work to consider state perturbation in Muzero algorithm.\n\nWeaknesses:\n1. There are multiple hyperparameters introduced in the Robustzero, and it is shown in the Appendix that the optimal parameters are different across different environments. Does Robustzero need to search for the best parameters for a new environment?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I will summarize all my questions here.\n1. The environments do not include Mojuco, which is a commonly used environment in previous related work. \n2. The worst attack is achieved by using ATLA-PPO, why not use the PA-AD[1] attack that is currently considered the strongest attack? \n3. It would be better to include some classical baselines that are compared in most related work, such as SA-PPO[2] in the experiments.\n4. There are multiple hyperparameters introduced in the Robustzero, and it is shown in the Appendix that the optimal parameters are different across different environments. Does Robustzero need to search for the best parameters for a new environment?\n5. Does Robustzero know the attack budget $\\epsilon$ during training? If it knows, how is the performance with uncertain attack budget $\\epsilon$?\n6. Diffusion model based methods also achieved strong robustness, as shown in [3]. Could the authors include it as a baseline? I believe the setting of [3] is similar to the environments used in the paper.\n7. Could the author provide training time and testing time comparisons?\n\n[1] Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. ICLR 2022\n\n[2] Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations NeurIps 2020\n\n[3] DMBP: Diffusion model based predictor for robust offline reinforcement learning against state observation perturbations. ICLR 2024", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XaXFDltrJG", "reviewer": "Reviewer_ZVib", "summary": "The paper adapts the MuZero algorithm (RobustZero) with a state-robustness loss and two adaptive hyper-parameter adjustment methods. The method is designed to deal with both worst-case and random perturbations of the state, if those are available during training. The authors evaluate their approach against the S-MuZero baseline and the ATLA-PPO and PROTECTED algorithms on 4 reasonable benchmark tasks. While RobustZero does not perform in all tasks significantly better than the baselines, it shows significant improvement in the IEEE tasks, in particular under \"random perturbation\", and generally in the perturbed RaceTrack environment. The authors also perform a large number of ablations, which show that their method with adjustment between worst-case and random-case perturbation outperforms robustifying only one of these cases, and how large the influence of the proposed changes to S-MuZero are.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive and valuable comments. \n\n$\\textbf{Response to Methods and Evaluation Criteria:}$ 1) Regarding the projector and predictor networks, please refer to the response to Q1; 2) Regarding the bold entries, we have revised all tables (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Tables/Tables%201-5.png); 3) We have added the learning curves (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Figures/Figure%20S-1.png).\n\n$\\textbf{Response to Experimental Designs or Analyses:}$ We have revised all tables according to your suggestion. \n\n$\\textbf{Response to Other Comments or Suggestions:}$ We would like to address your comments as follows:\n\n1)  MuZero-class methods learn abstract environment models, i.e., learned models, and employ MCTS for planning. At each time step, a real state is mapped to an abstract initial state by using the representation network. Then, the dynamics network and prediction network work in conjunction with MCTS to predict future evolution of these abstract states. This framework inherently involves a prediction process.\nThe compound errors mean the accumulation of inaccuracies in these predicted abstract states. To clarify this point, we will revise the statement to ‘Even a small “reality gap” can compound errors in the predicted abstract states within MuZero-class methods, causing reduced rewards and potentially harmful decision-making’. \n\n2) We will supplement the following missing information: i) $\\rho_0$ is the initial state distribution; ii) $\\mathcal{P}$ is a deterministic transition function; and 3) the state space is continuous.\n\n3) We will use “state” uniformly.\n\n4) We will add the following statement. The sequence of real actions from the sampled trajectory is used to compute $r_t^k$. \n\n5) We will revise the statement to “the goal is to maximize the expected return.”\n\n6) We have carefully double-checked the MuZero algorithm, and make sure that the reward  $r_t^k$ is the output of the dynamics network $\\mathcal{G}$. \n\n7) We will replace “environment model” with “MDP”.\n\n8) We will revise the statement to “$\\widetilde{t}$ and $\\hat{t}$ denote data at time t from a trajectory generated under worst-case and random perturbations, respectively.”\n\n9) We have revised Fig. 1 by adding the mathematical denotations of the networks (see https://anonymous.4open.science/r/RobustZero-SupportMaterials-8512/Figures/Figure%201.png).\n\n10) We have defined $\\mathcal{D}\\_{\\textit{WC}}$  and $\\mathcal{D}\\_{\\textit{RC}}$ in Fig. 1 (see the link above). \n\n11) We will revise the expressions: i) From $ \\mathcal{D}\\_{\\textit{WC}}\\sim\\mathcal{B}$ to $ \\mathcal{B}\\sim\\mathcal{D}\\_{\\textit{WC}}$; ii) From $ \\mathcal{D}\\_{\\textit{RC}}\\sim\\mathcal{B}$ to $ \\mathcal{B}\\sim\\mathcal{D}\\_{\\textit{RC}}$; and iii) From “B is the batch size” to  “B is the batch”.\n\n12)  We will add the missing symbol for Eq. 10. \n\n$\\textbf{Response to Q1:}$  Model collapse refers to a degenerate solution in which the encoder produces identical or nearly identical outputs for all inputs, resulting in trivial and non-informative representations. If we omit the predictor and directly enforce similarity between the outputs of two branches (e.g., using mean squared error or cosine similarity), the model may exploit a shortcut: “I might as well just output a constant vector, since all inputs yield the same output, the loss will be minimized.” This leads to model collapse. When collapse occurs, all states—regardless of whether they are perturbed or not—are mapped to the same or very similar initial hidden state, effectively losing the ability to distinguish between different inputs.  To avoid model collapse, one effective strategy is to introduce an asymmetric network architecture, where a predictor is added to one branch, while the other branch remains without it and has its gradient flow stopped. This asymmetry prevents both branches from trivially converging to the same constant output. The branch with the predictor learns to align its output with the target branch, which acts as a stable reference point. Because the target branch does not receive gradients, it cannot adjust itself to match the predictor’s potentially trivial solution, thus breaking the symmetry that often leads to collapse. This design encourages the network to learn meaningful representations, resulting in stabilization of optimization and a reduction in collapsing solutions.\n\n$\\textbf{Response to Q2:}$ The reasons are as follows. First, as shown in Eqs. 3-4, both the worst-case and random-case loss terms already incorporate information from perturbed and unperturbed states due to the use of a contrastive loss function. This contrastive loss encourages the learned policy to remain consistent before and after state perturbations. Therefore, an additional MuZero loss term specifically targeting unperturbed states is redundant. Second, an additional loss term will increase the computational cost.", "claims_and_evidence": "- \"The presented method is the first that achieves robustness against state perturbation with MuZero.\" I am not an expert in robust RL, but I have not seen any relevant MuZero papers claiming to do that, so I am inclined to believe this claim.\n- \"RobustZero performs well in both worst-case and random-case state perturbation.\" While no strictly true for all environments, RobustZero seems to outperform the baselines in both perturbed cases on Racetrack and in the randomly-perturbed case in the two IEEE environments. Somewhat surprisingly, even baselines that specifically optimize for that type of perturbation are outperformed here, which is pretty impressive.\n\nIn summary, I believe the authors back their claims up sufficiently.", "methods_and_evaluation_criteria": "The method seems to be mostly adapted from Liu et al. (2024), and its implementation is not terribly surprising. However, it is also not trivial and passes my personal threshold for sufficient novelty in an ICML paper. Which is of course subjective. \n\nI did not fully understand why RobustZero requires projector and predictor networks. I have seen similar arrangements when transition models are learned, and the encoding of $s_{t+1}$ can therefore be predicted from a projection of the encoding of $s_t$, but here this approach seems meaningless. It is also not clear why the projection does not simply learn a constant output (which would be guaranteed to be the same under perturbation). The authors claim that \"the key functionality of the predictor is to transform projected features to stabilize optimization and reduce collapsing solutions\", but I do not see why that is. Do the authors have a better explanation for this? \n\nEvaluation seems to be sufficient, with two suggestions for the authors:\n- Please do not make only the *best* entry in a column bold, this can easily deceive a casual reader. Make **all** entries bold that are not significantly worse (overlapping standard deviations) than then best entry. In your case the Pendulum task is practically useless, and some columns in the IEEE benchmarks should have multiple bold entries.\n- I am missing learning curves (performance in all three reward cases over environment steps). These are technically not necessary, but as a reinforcement learner I always want to see them to make sure the results are actually stable and the final results are not cherry picked. Please add some to the appendix.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "There is are a decent amount of ablations, demonstrating the usefulness of the proposed changes, which is welcome. Making **all** significantly better entries in bold will also enhance this Table.", "relation_to_broader_scientific_literature": "I am not an expert in robust RL, but the MuZero literature is sufficiently covered.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The paper is mostly well written, but sometimes, in particular in the formal parts, it is a bit hard to understand.", "other_comments_or_suggestions": "- l.36R: Your method does not address \"Even a small reality gap can compound errors in the learned models\", as you only consider robust state encoding, which does not compound (only future predictions do this). Please clarify here which error you are addressing in the paper.\n- l.135L: the MDP is missing an initial-state distribution. You should also define $\\mathcal P$ more precisely: is it stochastic or deterministic? Also, please mention that your state space must be continuous (to allow $\\epsilon$-balls).\n- l.142L+157L+115R: Please denote $o_t$ *either* a state *or* as an observation, not both. I recommend \"state\" to differentiate it from the observations in POMDP.\n- l.117R: when you define the goal of MuZero, please clarify which action sequence has been used to compute, e.g., $p_t^k$.\n- l.144Lf: the goal is to maximize the *expected* return.\n- l.161L+163L: the reward $r_t^k$ is usually the output of $\\mathcal F$, not $\\mathcal G$ in MuZero.\n- l.141R: \"environment model\" -> \"MDP\"\n- l.154R: The sentence \"$\\tilde t$ and $\\hat t$ denote time under worst-case and random-case state perturbations\" makes no sense. Time is not perturbed. You mean to say that $\\tilde t$ denotes data at time $t$ from a trajectory that is drawn under worst-case perturbation.\n- Please add the mathematical denotations of the networks in Figure 1.\n- Define $\\mathcal D_{WC}$ and $\\mathcal D_{RC}$ somewhere, e.g., in Figure 1.\n- In any case, it must be $\\mathcal B \\sim \\mathcal D_{WC}$, not the other way around, and $\\mathcal B$ is the batch and not the batch size (l.249L).\n- eq.10 misses a $||_2$", "questions_for_authors": "- Can the authors explain why \"the predictor is to transform projected features\" leads to \"stabiliz[ation of] optimization and reduc[tion] collapsing solutions\"?\n- Why do the authors not also use a third MuZero loss that optimizes for the unperturbed states? RobustZero already uses them and a third learning signal might improve performance further.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes RobustZero, a MuZero-class method that is robust to worst-case and random-case state perturbations, with zero prior knowledge of the environment’s dynamics. Although reviewers show some concerns, particularly for the difficulty of the benchmarks and the performance of the proposed algorithm, all reviewers finally voted for strong/weak accepts. Thus, we are happy to accept this paper. However, please make sure to address the reviewer concerns in the final version.
Decision: Accept (poster)