DMBP: Diffusion model-based predictor for robust offline reinforcement learning against state observation perturbations
Authors: Zhihe YANG
Yunjian Xu
Venue: ICLR 2024 poster (2024)
Keywords: Robust Reinforcement Learning
Offline Reinforcement Learning
Diffusion Models
Primary area: reinforcement learning
TL;DR: For state-based reinforcement learning tasks with state observation perturbations, we propose a new framework that recovers the actual states with offline-trained conditional diffusion models.

Abstract: Offline reinforcement learning (RL), which aims to fully explore offline datasets for training without interaction with environments, has attracted growing recent attention. A major challenge for the real-world application of offline RL stems from the robustness against state observation perturbations, e.g., as a result of sensor errors or adversarial attacks. Unlike online robust RL, agents cannot be adversarially trained in the offline setting. In this work, we propose Diffusion Model-Based Predictor (DMBP) in a new framework that recovers the actual states with conditional diffusion models for state-based RL tasks. To mitigate the error accumulation issue in model-based estimation resulting from the classical training of conventional diffusion models, we propose a non-Markovian training objective to minimize the sum entropy of denoised states in RL trajectory. Experiments on standard benchmark problems demonstrate that DMBP can significantly enhance the robustness of existing offline RL algorithms against different scales of ran- dom noises and adversarial attacks on state observations. Further, the proposed framework can effectively deal with incomplete state observations with random combinations of multiple unobserved dimensions in the test. Our implementation is available at https://github.com/zhyang2226/DMBP.

Published as a conference paper at ICLR 2024
DMBP: DIFFUSION MODEL PREDICTOR FOR RO- BUST OFFLINE REINTTRCEMENT LEARNING AGAINST STATE OBSERVATION PERTURBATIONS
Zhihe Yang!. Yunjian Xu
The Chinese University of Hong Kong, Hong Kong SAR, China
2 The Chinese University of Hong Kong, Shenzhen Research Institute (SZRI), Guangdong, China zhyange ink, cuhk. edu. hk, yixuêmae. cuhk wuu.k.ed...ee. edu.
ABSTRACT
Offline reinforcement learning (RL), which aims to fully explore offline datasets for training without interaction with environments, has attracted growing recent attention. A major challenge for the real-world application of offline RL stems from the robustness against state observation perturbations, e.g., as a result of sensor errors adversarial attacks. Unlike online robust RL, agents cannot be adversarially trained in the offline setting. In this work, we propose Diffusion Model-Bar Bodel-Based Predictor (DMBP) in a new framework that recovers the actual states with conditional diffusion models for state- based RL tasks. To mitigate the error accumulation issue in model-basee estimation resulting from the classical training of conventional diffusion models, we propose a non-Markovian training objective to minimize the sum entropy of denoised states in RL trajectory. Experiments on standard benchmark problems demonstrate that DMBP can significantly enhance the robustness of existing offline RL algorithms against different scales of ran- dom noises and adversarial attacks state observations. Further, the proposed framework can effectively deal with incomplete state observations with random combinations of multiple unobserved dimensions in the test. Our implementation is available at htttps://githhb.coo htw.thhb.ccooo/.... wtw.o///wwwggg2////////
INTRODUCTION
Reinforcement learning (RL) has been proven to be a powerful tool for high-dimensional decision- making problems under uncertainty (Mnih al. 2015 Silver al. 2017 Schritwieser et al. 2020).
However, its trial-and-errov learning manner requires frequent interactions with the environment, which can be expensive and/or dangerous in a variety of real-world applications (Levine et al. 2020). A widely adopted solution is to build up simulator for policy training, which is costly and may fail due to the discrepancy between the simulator and reality. As a promising alternative that has received growing attention, offline RL fully explores offline datasets and requires no interaction with the environments in the training process.
A major challenge of offline training is on the robustness against perturbation state observations, which may result from sensor errors, adversarial attacks, and mismatches between statistic datasets and the real environment. For example, GPS signal errors can lead to inaccurate positioning of autonomous vehicles, and position sensor errors can lead to erroncous estimation of robot arm postures. The robustness of the trained policy against state perturbations is vital for preventing agents from catastrophic movements. In online settings, various adversarial training methods have been proposed robustly handle the mismatch between observed and actual states (Zhang al. 2020 2021
Sun et al. 202T). These methods are not directly applicable in offline training.
classical approach against perturbed state observation is to train robust policies against worst-case disturbances (see the left subplot in Figure [I]. which may lead to cons-connervaaiim (Zhang et al. 2020 (2021). In pioneering work (Yang et al. 2022). the authors propose an alternative approach that adopts the conservative smoothing method to smoothen the Q-value and regularize the policy, preventing the agent from taking catastrophic movements under adversarial attacks in the test. The performance of the aforementioned approach may decay quickly with the increasing noise scale, especially in complicated environments with high-dimensional action and state spaces.
"Corresponding author

---

Published as a conference paper at ICLR 2024
Classical approach
Our DMBP against observation perturbations
against observation perturbations
a2
Universal Agent
Environment g a,)
Environment ( a.)
13a;13,)
Trajectory Cache
a:
Ft (
St+1
Robust Agent m(a,15;)
Sc+1
DMBP
X Attacker
X
Attacker
p3 a F2
N
5c1 = Sc41 E
c+1 = sct1 +
Tt-1
Tt
c++
Q--1
a;
8r
3.+1
Figure 1: Overview diagram of classical approach against observation perturbations (left) and the proposed DMBP improved RL decision process against observation perturbations (right).
For online image-based deep RL, Lin et al. (2117) propose a model-based approach to "denoise" the observations by predicting the actual states. They construct multiple-layer perceptron (MLP) neural network to detect the adversarial attack on image-based observations and predict the original states for decision-making in Atari games. For state-based RL tasks, similar MLP-based prediction methods have been used as data augmentation in online (Janner et al. (2019) and offline (Yu et al.) 2020, [022]) settings instead of denoising tools. In general, MLP-based prediction methods cannot be applied to complicated state-based tasks (like Mujoco) which are sensitive to observation noise and prone to eтrоr accumulation.
Recently, diffusion- based-aadddddeee generative models (Sohl-Dickstein et al. 2020b) are widely used in offline eL/decision-maken problems as trajectory generators (Janner et 2022; Ajay et al. 2002 Liang et al. 2023) and behavior cloners (Wang et al. (022). We note that potential of diffusion models facilitate decision making via state denoising has not been fully explored.
2015 Ho et al. 2020 Song et al.
Towards this end, we propose a new framework that predicts the actual states against observation perturbations for state based offline RL, which is referred to as Diffusion Model- Baleed Predictor (DMBP). Different from the aforementioned works, the proposed approach utilizes di diffusion models noise reduction tools rather than generation models, and can therefore enhance the robustness of existing offline RL algorithms against different scales of perturbations on state observations.
A diagram of the proposed approach is shown in the right subplot of Figure T Given the past- estimated state trajectory, last-step geenr-geeeeated action, and the noised current state from the environment, DMBP utilizes conditioned diffusion model to estimate the current state by reversely denoising data. To mitigate the error accumulation issue in state estimation, we propose a new non-Markovian loss function that minimizes the sum entropy of denoised states the RI. trajectory (cf. Section 4p. In order to well capture the relationship between the noised current state and the denoised state trajectory (especially the last RL timestep denoised state), we propose an Unet-MLP neural network structure to predict noise information (cf. Appendix [.J.. The output of DMBP is an estimation of the current state, which is fed into an offline RL algorithm generate the action. To our knowledge, this is the first state denoising framework for offline RL against observation perturbations state-based tasks.
The proposed framework has several advantages over existing offline RI. methods against noisy observations. First, with an objective recovering the actual state, DMBP can significantly strengthen the robustness of existing offline RL algorithms against different scales of random noises and adversarial attacks. The proposed approach does not lead to over-conservative policies, compared with counterparts that train robust policies against worst-case (or adversarial) perturbations.
Further, by virtue the capability diffusion models to infill the missing regions (i.e., image inpainting), DMBP facilitates the decision making under incomplete state observations with random combinations of multiple unobserved dimensions the test. Such a situation common in reality, for example, when robots continue work with compromised sensors.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORKS
Robust RL. Robust RL can be categorized into two taxonomies: thining-time and testing-timi robustness. Training-ting time robust RL involves perturbations during the training process, while evaluating the agent in a clean environment (Zhang et al.) 2022b] Ye et al., 2023). Conversely, testing-time robust RL focuses on training the agent with unperturbed datasets environments and then testing its performance in the presence of disturbances (Yang et al. [0222] Panaganti et al. 2022).
Our work primarily aims enhancing the testing-time robustness of existing offline RL algorithms.
Testing-time robust RL formulations can generally be divided into three categories (Xu et al. 2022)
Uncertain observations: In online settings, Zhang et al. (2020) propose state- adversarial Markoy decision process (SA-MDP) framework, which is advanded by Zhang et al. (2021); [Sun et al. (2021)
that adopt neural networks to simulate worst-case observation attacks for the training of more robust policies. In offline settings. Yang et al. (2022) utilize the conservative smoothing method to make the agent take similar actions when the perturbations on state observation are relatively small. ii) Uncertain actions: Tessler et al. (2019) explore the training of robust policies against two types of action uncertainties, i.e., occasional and constant adversarial perturbations. [Tan et al. (2020) utilize adversarial training on actions to enhance the robustness against action perturbations. ii) Uncertain transitions and rewards: The computation of optimal policies against uncertain environment parameters has been explored under the robust Markov Decision Process (MDP) (Xu & Mannor 2006; Roy et al. 2017 Ho et al., 2018) and the distributionally robust MDP frameworks (Xu
& Mannor 2010 Yu & Ku150 In online RL settings. Pinto et al. (2017) and Gleave al. (2019)
train the agent under adversarial model uncertainty through two-player Markov game approach
(Littman 1994). For offline RL training, Panaganti et al. (2022) propose a dual reformulated robust
Bellman operator to deal with the uncertain transition probability.
For models in the first two categories, the true states and transition probabilities of the environments are not influenced by the action, which is not the case for the robust approaches against model uncertainties developed in the third category. Our work belongs to the first category.
Diffusion models in offline RL. The diffusion model was originally proposed as an iterative denoising procedure for image generation in computer vision (Sohi-Dickstein al. 2015 Ho et al., 2020). Recently, diffusion model has been adopted in docisiionmaking for state-based tasks. Diffuser (Janner et al. 2022) and Decision Diffuser (Ajay et al., 2022) utilize the conditional diffusion model trajectory generator to facilitate the decision making of the agent. |Wang et al. (2022) propose the Diffusion- iasiiionnQQ.aa algorithm that adopts the diffusion model to regulate the policy not to be far away from the one used in datasets, in a similar spirit to Fujimoto al. (2019); Wu et al. 2019). Different from the aforementioned works, the proposed approach utilizes the diffusion model as a denoiser (against state observation perturbations) rather than a generator, for robust offline training of RL agents.
3 PRELIMINARIES
Offline RL. RL tasks are generally modeled as Markovian Decision Processes (MDP) in the form of M = (S, (S, A, r, P, 7, do), where is the state space, A is the action space, r S x AA + R represents the reward function, P is the model dynamics,
[0. 1) is the discount factor, and do E A(S) is the distribution of initial state sp (the set of the probability distribution over X' is denoted as A(Y)). P(s' (s, a) S x A + A(S) represents the transition function from state s to s' when taking action a. The state-action-reww
transitions over trajectory are recorded as t == (s,,0,, (e,,,,,,eeee re)e The goal of RL is to learn a policy T. that maximizes the expectation of the cumulated discounted reward R(T) = Dio 7 (rr( at), denoted by T = arg max, Bao-do,a-n [R(T)].
commonly used iteration method for btate-based tasks is under the actor-critic framework, where the Q-value of policy is defined (s,, ax) := Ea-a EE (6--))((s) a.)] and is modeled using neural networks (recorded as (u, ).)). To approach an optimal policy, the temporal difference (TD) method is adopted to update the critic Q-value by minimizing the TD loss: LTD (1))
E ((s.an))))D((( (r + max (s',
- ( {.. The actor is updated by Lactor (0) =
E.ED, [=) a)), where the dataset D records historical interactions between agent and environment, and is continuously updated in the alternate training of the actor and the critic. In offline RL settings, the training is performed on a statistic dataset D. := {(s, r. r, s s')}, which is obtained from behavior policy n. without any interaction with the environment.
3

---

Published as a conference paper at ICLR 2024
Direct adoption of the actor-critic approach may lead to severe distributional shift between the trained policy Fg and the behavior policy Ty due to the over-estimation of the Q-value of actions unseen in datasets. To mitigate this issue, policy regularization has been adopted to update the actor
through constrained policy loss (Wu et al. 2019; Kumar et al. 2019 Fujimoto et al. 2019 Fujimoto
& Gu 2021 Wang al. 2022): (())=
La( Olactor Factor (+), where La( is the behavior
cloning loss representing the nominal distance between the trained policy and the behavior policy,
and Qacttr is the coefficient for the Q-value term. Alternatively, conservative Q estimation updates
the critic through minimizing the constrained Q-value loss (Kumar et al. 2020; An An et al. 2021 Lyu
al., 2022; Yang et al. )022): ((²) == Cy(()) + CeriticLTpl (v)) (),, where [g(1) is the penalty on
Q-value for out-of-distribution actions, and O'eritic is the coefficient for the TD loss term.
Diffusion model. Diffusion based generative models have been widely used for synthesizing high- quality images from text descriptions. The forward process, i.e., the noising process, is Markov chain that gradually adds Gaussian noise to data according to a variance schedule "i....,k??
g(>1.K zo)
II q xg-1)), k=1
q(>k x == Na V 1
The reverse process, i.e., the denoising process, is a Markov chain with learned Gaussian transitions that usually starts at p(rk) Noxx; I):
Po (Zo.k) == p(xx) II po æk).
po xx) := 11 k), Eg (Ik, k)).
Ho et al. (2020) derive a simplified surrogate loss for the reverse process denoising:
Ldenoise (0) := -x.. (eleirllllleeeeee [0(x,
k) <||²].
(I)
The Gaussian noise e, which perturbs the original data æn to noised data æy. is estimated through the neural network based predictor (2x,, k). -x-1 is sampled from the reverse process as ( k) and Eg (Ix, k) are functions of c,( k). It straightforward to extend diffusion models to conditional ones with (x Ze, c) (conditioned on information c), where the noise prediction is given by co k, c).
DIFFUSION MODEL BASED PREDICTOR
We express the perturbed version of the original state s as 8, where Ba E) [ d(s 8) S <)} is the perturbation set and the metric d(...) -) is based on Ep norm, as in Shen et al. (2020). An adversarial attack on state is introduced in Yang et al. (2022): s* arg B.ggmaggB,..c.. where (-Ill-) is the divergence of two distributions. The targets of both works are to minimize the smoothness regularizer for the policy: Rg EAED maXgeB,(s.e) D(T(-11))] (+)),, and to mini- mize the smoothness regularizer for the value function: Ry = R IEED ma e) (Q a) QB3, a)), against the perturbations on state observations. We remark that we do not normalize the state observations when applying the perturbations as in Shen et al. (2020): : Sun et al. (2021) in
contrast to Zhang et al. (2020); Yang et al. (2022).
In Section 4.1
we propose DMBP to recover the actual state for decision (which is fun- damentally different from the technical approaches in aforementioned works). In Section
4- we
propose a new non-Markovian loss function to mitigate error accumulation. In Section 4.3 we apply DMBP to RL tasks under incomplete state observations with unobserved dimension(s).
4.1 CONDITIONAL DIFFUSION FOR PREDICTING REAL STATE
As there are two timesteps involved in our framework, we use superseripts i, h E {L.... K} to denote diffusion timestep and subscript & (L...., T' to denote trajectory timestep in RL tasks.
DMBP is inspired by the diffusion model framework originally proposed for image generations (Ho al. 2020). As the proposed framework essentially deals with information with small medium scale noises instead of generating data from pure noise, we redesign the variance schedule as:
B = 1- - as e +c
II
k
Ois
A
ai-1
Bi,
4

---

Published as a conference paper at ICLR 2024
where a, b, are hyperparameters (ef. Appendix B.2 noise scale to be small in the diffusion process and limits the total number of diffusion timesteps K for predictor training. We use the conditional diffusion model to obtain the denoised state &, from the noised state &t, with the condition on last step action d;-1 and the previously denoised state trajectory --_1 (&1, $2, ...; $t-1), )- The denoised state is sampled from the reverse denoising process, which can be expressed as a Markov chain:
The redesigned variance schedule restricts the
81 (
0ik
T( = f(a(s)) II po( 8% a: 0, T(-1):
where (Rt) Vāgšt. The transitions (0 8% a: Tf-1) can be modeled using Gaus-
sian distribution (s; (3) a1-1, T (3( a:-1, 1, 1;)), with the following mean and variance (Ho et al. 2020):
var(1 di-1)
vāi-13 u(1) 1 a
a1-1: a 1)
+
St
Eg(s a1-1, T(-1, 6) = B,I.
1 ag
((1) is the state directly recovered from the current diffusion step noise prediction, which is Here, 8 given by
0(1)
vI CnEa
a. a,,
C2s
The reverse diffusion chain is given by
3)
Ea( a: a: 7 + /Be. (a -
B,
(4)
where E -N I) and is set be at the final denoising step (i 1). For the final step denoising output of Eq. 4 o (i.e., the output of DMBP), we refer it to as 8,. 8; can be used for decision-maie making by any offline-trained agent according to a, T-(- &c). & is stored in the trajectory cache T. and the pair of (T ac) will be utilized for the next step denoising. In practice, on account of the stochasticity involved in the diffusion process, denoise the state 50 times in parallel and take the average value as the final output 8, to prevent the denoised state from falling out of the distribution.
We find that directly inputting state trajectories and action into neural networks leads poor noise estimation (cf. Appendix C for ablation study on network structure), partially due to the fact that $;-1 more closely related to 8; than $, with captured by neural networks. Therefore, we first extract the information from the trajectory with U-net (Ronneberger et al. 2015 Janner al. 2022) (recorded as U8(,), r(-1)), and then utilize an MLP-based neural network to predict the noise through co Tf- 8% a;-1, 8,-1, i), which is represented by eg G8t at - i) for notational convenience. See Appendix B. for details.
1. and that this information cannot be well
4.2 NON-MARKOVIAN LOSS FUNCTION
The accuracy of the current denoising result š, highly dependent on the accuracy of the diffusion condition T A straightforward adoption of the loss function in the denoising diffusion probabilis- tic model (DDPM) (Ho al. 2020) may lead severe error accumulation in online testing, due the mismatch between the training process noise prediction ea ca a,-1: 7/-1. and the testing- process
noise prediction eØ ( ae T. DMBP, we propose a non-Markovian training objective to minimize the sum entropy denoised states over the RL trajectory т:
i). To mitigate error accumulation and enhance the robustness of
Centropy Σ E BaCEr.((aL) [- log P(s, az-1, T(]]
where P(
a T is the distribution of state after denoising at RL timestep Following the setting in Ho et al. 2020). establish closed-form expression of the training objective that minimizes Eq. 5 which can be simplified as (cf. the details in Appendix A}:
0t-1, (0-1,T(-1>> pa (8
T
Csimpie (0)= = E, ~do -.
Ee (8), a TH - =
(6)
5

---

Published as a conference paper at ICLR 2024
where Ug is the uniform distribution over discrete set {1, ................................ K}, and the noised states for all terms are sampled through:
Va,st + -I- dsej, ~V
For computational convenience, we further simplify Eq. 6 and derive our non Markovian loss function by sampling the partial trajectory (St-N Qt-N, +1: ........ M(+M-1) from the offline dataset D, (N is the condition trajectory length and M the sample trajectory length):
[(0) EEiili aN N M DE,,
[E a.-1, T(-1,) ei11 +
Le
++M-1 Σ 370 1111
[ep(S
am T 1,
m
where the state trajectory condition for the predictor 6g in Lg is the original Ti-1 = ..I.NN,..,.11... from the offline dataset Dr, and the state trajectory condition in I. can be expressed as T m- { j E {m- ....., m 1}}. with:
if <t
v1 03 aj a1 otherwise (jE (t.....,t t M -))).
Different from the loss function in Ho et al. (2020) that concerns only single-step diffusion accuracy (for data generation under conditions on ground-truth states), the proposed non-Markovian loss function trades off between the diffusion accuracy the current RL timestep and the condition shift in long RL time horizon (to avoid error accumulation).
4.3 DIFFUSION BASED STATE INFILLING FOR UNOBSERVED DIMENSION
Inspired by the application of diffusion models in image inpainting (uugmayr al. 2022 propose a state infilling procedure for DMBP facilitated decision making, which is shown to work well based RL tasks with incomplete state observations (cf. Section 5.2)
We denote the ground truth state as 8t, the unobserved state information as (1-m) @ 8t, and the observed state information 6, m 8t, which is incomplete with some masked di- mensions. Given the recovered state trajectory T agent generated ac- tion a, and the known information of the current state 6., DMBP aims to recover the original state &, for deci-
Algorithm Diffusion based state infilling for DMBP
Require: st (N, I), ét. a:-1, 7:-1,, for i K, do
for I,..., do
3:
<~ N0,, if > clse c 0 3-1 *t,noonn / Vā.li /1 d.E
4:
5:
~~ No,, if 1, clse 2 = 0
Epred = 60( a:-1, T
7:
unkknwwn vor
6f
(1,(1-a,) Epred VBz
sion making. Following the inpaint-
8:
S m C $ (1 - m) O uunknown
ing method of Lugmayr et al. (2022).
9:
if u < U and
then
DMBP infills the missing state infor-
10:
4 ~ N 1 - 1 4: 1BB
mation through Algorithm]] [] For each
11:
end if
diffusion timestep. the known region
12: end
of the state is determined from the for- 13:
end for
ward process (noising process) line 14: return s. = 5
4, and the unknown region of the state is determined from the reverse process (denoising process) in line 7. To avoid the disharmony of the forward and reverse process generated information. the com- bined information (in line 8) takes one diffusion forward step in line 10, which is called "resampling" The resampling is performed by U times for one diffusion timestep. More resampling times may lead to more accurate and harmonious diffusion information at the cost of higher computational load.
5 EXPERIMENTS

---

Published as a conference paper at ICLR 2024
We evaluate the proposed DMBP together with sev- eral state-of-the-art baseline offline RL algorithms
Original States ($y):
on D4RL Gym benchmark (Fu et al. 2020) against different types of attacks on state observations. The baseline algorithms include Batch Constrained deep Q-learning (BCQ) (Fujimoto et al., 2019), Conser- vative Q-Learnng (CQL) (Kumar et al. 2020), TD3 with Behavior Cloning (TD3+BC) (Fujimoto & Gu
2021 Diffusion Q-Learning (Diffisionn @L) (Wang
Observed Noised States (3,):
et al. 2022). and Robust Offline Reinforcement
Learning (RORL) Yang et al., 2022).
We train DMBP for 300 epochs (1000 gradient steps with a batch size of 256 for each epoch) with hy-
perparameters defined in Appendix B.2 We train the baseline algorithms with the corresponding sug-
gested hyperparameters in specific enviromments and datasets. We perform two tests, on robustness against noised state observations (in Section 5.1) and on bustness against incomplete state observations with unobserved dimension(s) (in Section (52).. We uti- lize the same DMBP for all baseline algorithms (cf. the framework in Figure D. and benchmark their performance against the original baseline algorithms (without DMBP). We present partial results on the D4RL Mujoco benchmark, and the results of other datasets (including medium-experr, medium, and full-replay) and other benchmarks (including Adroit and Franka Kitchen) can be found in Appendix [D]
DMBP Recovered States (8,):
Figure 2: Visualization of the denoising ef-
fect of DMBP with Diffusion QL (trained on the dataset of observation is perturbed with Gaussian dis- tributed random noise with std of 0.10.
The
5.1 ROBUSTNESS AGAINST NOISED STATE OBSERVATIONS
Firstly, we evaluate the performance of DMBP in basic setting with Gaussian noises of standard deviation k: & significantly enhance the robustness of all baseline algorithms, especially on the dataset "medium- replay". where DMBP strengthened bascline algorithms achieve similar scores as in the corresponding noise-free cases. To demonstrate the powerful denoising effect of DMBP, we visualize a partial trajectory of "hopper" during the test in Figure 2
- K (O, I). The evaluation results in Table [ indicate that DMBP can
Table I: D4RL score ones ("DMBP" recorded in blue) trained with expert (c) and medium-replay (m-r) datasets under different scales of Gaussian random noises on state observation. The evaluation results are averaged over random checkpoints (20 tests for each checkpoint).
score of the baseline algorithms ("base" recorded in black) and the DMBP strengthened
Noise scale
RCQ
CQI.
TD3+BC
Diffision QI.
Env Dataset
base
DMBP
base
DMBP
base
DMBP
base
DMBP
base
DMBP
HalfCheetah
9.8
9300-6.1
8,.8
92.9
108.51-1122
0.05
4.5+2.6
60.2+239
18.1+8.6
60.9422.5
7.3+666
77.1+15.5
4.6+366
75.24:2077
15.4+39
$5,7=29.2
0.10
4.54:2,
26.8:16.2
7.444.0
40.54:166
4.74:36
47,5422.2
3.342.5
39.8=21.8
3.7=1.9
32.8=20.4
41.62:4.
47.0110
45:21:0.9
41.710.8
66.711.4
0.10
20.6
38.511122
35.61
45.8.1.
28.51...55
44.31100
30.1:14.1
45.6.10.9
43.5:12.4
61.91
0.15
14.8+10022
35.14887
28.8+1.5
44.6+1.11
24.0+8.9
42.5422.6
24.247.6
44.6433.0
30.345.9
58.4-122
88.41:22.4
109.1113.7
108.9-10.5
104.115511
110.413.1
Happer
0.05
34.3+1134
61.0422522
41.2421.8
85.7422622
32.24-884
79.1422.2
38.2412.4
84.84-7.4
56.943349
69.3+19.8
0.10
24.3=1009
18.1
24.3+11.8
48.8+2204
22.7+11.6
12.6
24.0+9.3
16.1=17.3
24212200
37.5=10.5
0
78.7+196
96.188
80.9122455
95.7417.2
103.1108
m--
0.10
15.7-99.0
66.8+17.3
47.5+21.6
19.1412,4
14.4+12.3
71.9424.5
25.94 12.4
85,9=2009
85.9+29.5
103.2-13
0.15
11.1=7.2
64.5217.2
33.7+21.2
80.7+:6.5
9.6+7.3
66.1
17.9:11.5
72.2=222.
51.1
104,2=322
0
111.6:50.6
1088=119
110.700.5
109.6:0.5
104
Walker2d
0.10
77.9=37.6
11033200
97.6+21.9
94.3+22.3
72.9#39.4
109.2411
93.3+27.2
109.11440
19.7
97. 27.8
0.15
28.2=3224
104.2413.5
78.9133.2
83.41:23.3
9.24+:3.6
107.54:5.2
30.513325
94.5=18.1
81.6126.4
84.5=26.4
30.61316
81.1998
93.11109
$8.7.199
m-r
0.10
14.7+1111
53.1428.5
708++18.
78.747.2
40.7+-5.3
84.448.7
59.6+33.88
92.6+1006
88.6+1.1
88.4+2,5
11.245.9 0.15
52.9+20.9
48.6126.5
73.6+
16.5+12.8
77.9=17.2
19.2415.7
91.349.6
19,4+1.
89.024.5
7

---

Published as a conference paper at ICLR 2024
Table 2: D4RL score of the baseline algorithms and the DMBP strengthened under uniformly
distributed random noise (U-rand), maximum action-difference attack (MAD), and minimum Q-value attack (MinQ) on state observations.
Env
Dataset/
Naise
BCQ
CQI.
TD3+BC
Diffusion
RORI
Noise Scale
Type
base
DMBP
base
DMBP
base
DMBP
base
DMBP
base
DMBP
HaliCheetah
U-aad
69.14-215
27.2-644
69.64-22.4
16.3413.1
11,6+10.9
77.84-1.8
24.3475
MAD 36+1.77
52,5+1799
1244669
61.2+19.7
4.743,5
65.4+160
4.+32
62.9+132
14,1+2.5
54.3=27.1
0.05
MinQ 12.8=9.3
51,8+23.9
194=11.3
60.4+19.4
18.0442
88.2+113
8.046.7
71.1215.
933488
71.0=29.1
U-rand 31,5210,6
40.315.9
40.9226
46.4118
36,9166
46911.
38.5:5.7
4688009
39.911.3
61.22111
m--7
MAD 19.248.2
29.426.9
29.0=2.6
46.340.9
27.1434
36.240.9
22.343.8
34.545.5
22.5+15
62.3=1.0
0.10
MiaQ 5.145.2
36.71888
39.2-0.8
46.24.1
36.7168
3861111
34.0114
63,222.3
U-rand 46.1=20.7
66.9426.3
59.6=29,4
95.7423.8
42.6428.4
84.02214
53.2=2088
84,4425.3
85,343700
819925.2
Hopper
0.05
MAD 31.1=1444
53,21:24.
22.6=13.9
73.94277.9
27,2110.9
60.342122
36.8-9.0
31.1212.3
36.6422.2
59.0=13.8
MinQ 47,41.1899
62.51.279
32.7=13.5
58.71:17.9
45,3427.5
95.71-276
66.74336
59.21123.9
79.8132.7
29.4122.1
U-rand 18.54822
68.9
66.1-20.1
95.488
20.49911
6544220
333+ 10.7
949+177
80.7428.0
1033-13
m-r
MAD 5.145.00
37,5426.1
32.1=1599
88.9413.7
6.145.5
64.34218
99+8.1
38.3+1588
51.6430,7 97,5=2.5
MinQ 5.3455.4
18.32:84
84.6=14.1
87,516.6
118876
80.511811
512225,1
62,5427.3
98.3-26.
103.2-24
U-rand 102.14188
1104=08
106.1499
1060::7.4
1429
1100-003
107.2
1094:055
95.1 97.229.
Walker2d
0.10
MAD 50,5=43.7
70.5+:3..
64.1=27.0
97.6216.1
19.9=22.7
69.7=175
36.6235,5
88.2424.8
61,9429,2 83.8=19.9
MinQ 99.9422222
105.6+11
99.9-1188
102.4-69
91,94224
105.5-13
101.142.0
102.4+1.3
91 91,8-28.0 89.3+133
U-rand 17.3=1222
34.9225.7
69.2=209
1992
51222283
83.61143
64.2=278
91.112.1 89.11I
88.7221
11-1
MAD 6.613,3
43.41:2..8
19.7=14.7
78.4888
8.824.4
20.811911
7.222.3
66.1124.2
8199111 90.5
0.15
MinQ 7.314.2 30.31.6.1
21.7115,9
47.2423.2
89.611.7
We consider three additional types of noise attacks that are commonly used on state observations, where DMBP-strengthheee algorithms also outperform the corresponding baselines (cf. Table 2 i) Uniform random noise distributed inside the lo ball with the norm of <: 8, 8; + K 21 (d(-I,1). ii) Maximum action diffonddiffeeence (adversarial) attack: The noises selected inside the &oo ball with the norm of k, such that &, 8, + arg maxicB.(e.r) (+(() [| (+())) Among 20 samples of & in the ball, we choose the one with the largest [|Tx(-|s) (-)))
iii) Minimum Q-value (adversarial) attack: The noises are selected inside the too ball with the norm of k such that, & = + arg minaeBa(s.n) Q(,,, na (-())) Again, we sample 20 times and choose the one with the minimum Q to be the perturbed state St-
The latter two adversarial attacks have been considered in the literature (Pinto et al. 2017 Zhang al. 2020 Yang et al., (0022). For fair comparison, when we use DMBP against adversarial attacks, first sample 20 noised states and denoise them using DMBP, and then choose the denoised states & with the maximum action difference or the minimum Q-value as the perturbed state.
CQL DMBP-COL
COL Iwithout mask)
(without mask)
halfcheetah-exppeee exxpettt22
halltheecah-mediuu--eeeeey
100
75-
50
20
25
mosper.expert.vg
hooper-medlum replay-v2
100
100
25
2ellkerddexpprt-222
75
75
25
number
number
Figure 3: The performance of CQL, DMBP-CQL, RORL, and DMBP-RORL with incomplete state observations that have 1-5 unobserved dimensions. The dash-dot lines represent the performance of the corresponding baseline algorithms in the original environment with fully observable states. (The total state observation dimension 11 for hopper, and 17 for both halfcheetah and walker2d.)
8

---

Published as a conference paper at ICLR 2024
5.2 ROBUSTNESS AGAINST INCOMPLETE STATE OBSERVATIONS WITH UNOBSERVED DIMENSION
We utilize DMBP to recover the missing state information for decision-making. In D4RL benchmark problems, we mask some dimensional state information that cannot be observed by the tested policy (i.e., the masked dimensions of the state are set as 0 for E {2.3. 32........ ..... T)). The baseline algorithms make decisions based on the observed (incomplete) states, and DMBP- iinroved counterparts take actions according to the recovered states. For each dimension of the state, we make the dimension unobserved and conduct 10 tests. When multiple state dimensions cannot be observed, we randomly select 30 groups of dimensions and conduct 10 tests on each group. The experiment results of CQL, RORL with offline "expert" and "medium-replay" datasets are shown in Figure 3 DMBP significantly enhances the performance of all baseline algorithms by accurately predicting the missing state information. On "medium "medium-replaa"" datasets, the DMBP strengthened algorithms incur little performance degradation in masked environments, compared with that achieved in the original environments with complete and accurate observations.
5.3
ABLATION STUDY
(Markavian loss)
Checkpaint eval
Training evall (std=0.05)
Training eval (std-0.10)
Training (std=C.15)
100
100
60
60
60
80
40
40-
40
40
20
20-
0--
0-
100
200
100
200
300
Gauzsien 3td (x1001)
Gradient stege (x10")
Gredient steps 1x10")
Gradient steps (x30")
Figure 4: The checkpoint and training process evaluations of random noises, where DMBP is trained on hopper-expert-v2 with different sample trajectory lengths (M). The curves are averaged over 5 random seeds, and the checkpoints are selected randomly after gradient steps of x 10
QL under Gaussian
In Figure 4 we conduct ablation studies on dataset "hopper-expert-v2" where algorithms are more prone to error accumulation than in other danaaett/enniirommentss, to demonstrate the efficacy of the proposed non-Markovian loss function and evaluate the impact of the non- non- -on-Mroovvinn sampling length (M in Eq. 7) We utilize pre-trained Diffusion QL for decision-making to evaluate the performance of DMBP under the framework in Figure Other hyperparameters and DMBP training follow the basic settings in Appendix B.2 and D.I respectively.
When M 1, the DMBP training objective 7 reduces to the classical training objective of diffusion models in Eq. (i.e. LM = 0 in Eq. [7). From the second to the fourth subplots of Figure 4 we observe that the direct adoption of classical conditional diffusion models suffers from severe error accumulation as training proceeds. The proposed non- non-Markovian training objective significantly en- hances the robustness of the baseline RL algorithm against state observation perturbations, especially when the noise scale is large. When M is no less than 6, the performance of DMBP remains almost the same. To expedite the computation, we M for the "hopper" environment. More ablations studies on neural network structure and condition trajectory lengths (N) can be found in Appendix
6 CONCLUSION
In this work, we propose the first framework of state-denoising for offline RL against observation perturbations in state-based tasks. Leveraging conditional diffusion models, we develop Diffusion Model-Based Predictor (DMBP) to recover the actual state for decision-making. To reduce the error accumulation during test, we propose a new non-Markovian loss function that minimizes the sum entropy of denoised states along the trajectory. Experiments on D4RL benchmarks demonstrate that the proposed DMBP can significantly enhance the robustness of existing offline RL algorithms against different scales of random noises and even adversarial attacks. The proposed framework is shown to be able to effectively deal with the cases of incomplete state observations (with multiple unobserved dimensions) for btase RI. tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the General Research Fund (GRF) project 14200720 of the Hong Kong University Grants Committee and the National Natural Science Foundation of China (NSFC) Project 62073273. The authors would like to thank the anonymous reviewers for valuable discussion.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision makiio?? arXiv preprint arXiv:2211.1565) 2022.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified Q-ensemble. Advances in Neural Information Processing Systems, 34:7436-7447) 2021.
David M Chan, Roshan Rao, Forrest Huang, and John F Canny. t-sne-cuda: Gpu-accelerated t-sne and its applications to modern data. In 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pp. 330-338. IEFE, 2018.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffss--faa conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12413-12422. 2022.
Accelerating
Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? study on diffusion times in score-based generative models. arXiv preprint arXiv:2206.051775 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 14420012--01444 2021.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052- 2062. PMLR, 2019.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial 2019.
policies: Attacking deep reinforcement learning. arXiv preprint
Chin Pang Ho, Marek Petrik, and Wolfram Wiesemann. Fast Bellman updates for robust MDPs, In International Conference on Machine Learning, pp. 1979-1988. PMLR, 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances Neural Information Processing Systems. 33:1 139-11991 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.0/6433 2020.
10

---

Published as a conference paper at ICLR 2024
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877, 2023.
Yen-Chen Lin, Ming- Yu Liu, Min Sun, and Jia-Bin Huang. Detecting adversarial attacks on neural
network policies with visual foresight. arXiv preprint arXiv: 0010144 2017.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine Learning Proceedings 1994, pp. 157-163. Elsevier, 1994.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.04744 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529- -339995333323 2015.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. arXiv preprint arXiv:2208. 05129, 2022.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial rein- forcement learning. In International Conference on Machine Learning, pp. 2817-2826. PMLR. 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisee Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015.
Aurko Roy, Huan Xu, and Sebastian Pokutta. Reinforcement learning under model mismatch. Advances in Neural Information Processing Systems, 30, 2017.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon
Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et Mastering Atari, Go, chess and shogi by planning with learned model. Nature, 588(7839);644-69, 2020.
Qianli Shen, Yan Li. Haoming Jiang. Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pp. 8707-8718. PMLR, 2020.
David Silver, Julian Schrittweeser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lueas Baker, Mattew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354-35, 2017.
Jascha -all-Diksstein Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcore-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020b.
Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? Towards optimal and efficient evasion attacks in deep RI.. arXiv preprint arXiv: 106.000.500 156805087, 2021.
11

---

Published as a conference paper at ICLR 2024
Kai Liang Tan, Yasaman Esfandiari, Xian Yeow Lee, Soumik Sarkar, et al. Robustifying reinforcement learning agents via action space adversarial training. In 2020 American Control Conference (ACC), pp. 3959-3964. IEEE, 2020.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applica- tions in continuous control. In International Conference on Machine Learning, pp. 6215-6224.
PMLR, 2019.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06/993 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint
2019.
Huan Xu and Shie Mannor. The obuusness-pefformance tradeoff in Markov decision processes. Advances in Neural Information Processing Systems, 19, 2006.
Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. Advances in Neural Information Processing Systems, 23, 2010.
Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.0800 2022.
Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. RORL: Robust offline reinforcement learning via conservative smoothing. Advances Neural Information Processing Systems, 2022.
Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robust offline reinforcement learning with general function approximation. arXiv preprint arXiv:2310. 2023.
Pengqian Yu and Huan Xu. Distributionally robust counterpart in Markov decision processes. IEEE Transactions on Automatic Control, 61(9):2538-2555 2543, 2015.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 29-14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. COMBO: Conservative offline model- mosel-bbsed policy optimization. Advances in Neural Information Processing Systems, 34:28954-28967, 2021.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui
Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems,
21037, 2020.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning
state observations with learned optimal adversary. arXiv preprint arXiv:2101.08455) 2021.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit
models. arXiv preprint arXiv:2206.05564 2022a.
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robus offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 5757-5773, PMLR, 2022b.
12

---

Published as a conference paper at ICLR 2024
A DERIVATION OF THE TRAINING OBJECTIVE IN EQ.
Following the standard setting of diffusion models with Gaussian distributed noised states (Sohl Dickstein et al. 2015 Ho et al. 2020), below we derive the non-Markovian training objective via conditional diffusion models.
Suppose that the initial state is perfectly known for the agent, i.e., 81 known and does need to be estimated. Th-1 = {&j J < ( }} represents the denoised state trajectory. Our learning objective is to minimize the cross-entropy of the denoised states along the RL trajectory:
Centropy
By( |- log Po(8 T[-1].
We denote the state the forward diffusion process as sj and the state in the reverse diffusion process as st. We refer po (s a;-1, Tf-1) ) to as po (o() below. Following Ho et (2020), we adopt the variational lower bound (VLB) to optimize the negative log-likelihood:
T
LVLB=
E ((x)) | loo po()) + DKl(a(s 1:K
sp)0(((( K:K
| a(,,
(s 80)
By [10)) + Es x
log
Pp (
89, at T
(. log Po(
g(s)
Eg [log po()))
E -g(
89. a;-1, T
)o (5))
q(s !
E x
log
Po (o($0: 80:K
(84 at-1, T(
i= ((8)
ss 1
Σ
E 1.kk ~ (
log
(0( I
(s
8) a,-1, T
t=2
g(8) 1-1
8/ 80)
E f 1 ((2))
log Po
53 a1,11111 1(11) +
log
p
8), a1-1, T/-1)
t=
q(8t |8])
log
po(K))
T
Σ
E k - ((
log Po( (
s7 a,-,
2
K
DEL (g(s)
(;) po 7())),,(_1))
+ DeL (g( K (l)(
(8)
where (xz(-I) represents the Kullback-Leible divergence and it is guaranteed that LVLB Centropy (oo et al. 2020).
2
As in Ho et al. 2020), for the first term in the last line of Eq g we let
( (N 5
31 Co (8) a -,,,), B,
Po (s
8. a a, - -
(9)
Vai(1 a1)
To directly produce the noised state under Gaussian distributed perturbations, obtain from the forward diffusion process that
Vo,8t + /1 11-10,6 G d~N((, i (1,2,.... ..., K).
(10)
13

---

Published as a conference paper at ICLR 2024
For the expression in Eq.
9 have
log pa
8/ ax-11
T
81
81
log (2T)
Σ
[1s
co($),
co(s), a1,1 7-- 1) (11)
Val
Vai(1 - ā1)
1 log (2T)
1
|1312 Σ
[|e((5) a:-1, 1, c: - (z)
20
where the first equality follows from Eq. 9 and the second equality holds due to Eq. 10
For the second term in the last line of Eq. we let (Ho et al. 2020
[(5) 8.) = =N Vas Ōi-1) 1) 8 + v va,-13
1
8;
8
BiΣ
(12)
ai
1 āj
and
B) 01, (
8 Vas
3
1 - ai-1 3
po p0
Eo a;-1, T(-1; 7
0z,
0:)
(k
(13)
Utilizing the Kullback-Leibler divergence for Gaussian distributions in Eq.
12 and Eq.
13
we obtain
DKL (g(s)-1 [1,=%] (
8% a1-1- -))
1 /o; ā;-1)
8 + vāi
8
B.
8, Vai
+
Va,(1
az)
c0(3) a
(- a.
n,
B,
|eea (5): a;-1, T/ = 1,
20; ā;-1)
(14)
where the last equality follows from Eq. 10
As in (Ho et al., 2020), for the third term in the last line of Eq. 8 we let
q K (.)
A (Vars) (1 ax)
(15)
In a similar spirit to the pure Gaussian distributed noise assumption made in Ho et al. (2020) assume that the observed noised state follows Gaussian distribution, i.e.,
we
(=(
K - (ms? nZ))
(16)
According to Eq.
15 and Eq.
16
obtain the KL divergence of Gaussian distributions
d= āk d dloog
1 - āg
DEL (g(
2 po 8
|1( [[Vaa==)ss) -
(17)
which is constant and does not depend the neural network parameters EØ. Substituting Eqs.
to the last line in Eq. we can express LVLB as
CVLB
Band
+
(8),
7 cil
2=
(18)
T
T
K
Eg et
Σ
C
+
> 1|
a 7( -,
1=2
2
where Ct is a constant for E (1,2.....
1}:
C. C== [
IlVax - m)s? 11āk
āk d log
1 āk log [2 | 8
n
n
and the coefficients for the norm terms are given by
14

---

Published as a conference paper at ICLR 2024
if==1,
201
B
otherwise (i ;
20;
0:-1
Following the setting in Ho et al. (2020), we ignore the constant term and the coefficients. The
training objective in Eq. 18 be simplified
K
Csimple (0) = Es, (0o.)
||0(3) 7.-1, ctl?
E Nod.
[[o0(5) a1 -,,,,, c/ll
(19)
E - ar
Σ
|co (5) a T/ -;:1?
where is the uniform distribution over discrete set 21,2.... K}, and for T == {8z 1 t-1}
we have
81
if j=
83
(x(5))
otherwise (j (2...., 1}).
From 19 it clear that the simplified target make E (8), a:-1: T/-1 i) close as possible e for any f. From the forward noising process in Eq. T0, we have the following approximation
((#)
Va,
en ( a.-1; l (2...., 1}. (20)
04
VI- a
Therefore, Eq. 19 can be expressed as:
Ssimple = df
||E0(02 a1, 81,
e/112
terml
(21)
(en(8 a b T- 6
teri22
where T--1 == (8) 1--5-11 1), and & along the RI trajectory can be calculated through Eq. 20 iteratively from == 2 to = T. However, this is time-consuming the sampling process needs to be looped for (T'22 - 2) times. In addition, the historical state trajectories multiple RL timesteps ago have little impact
the current denoising results. To expedite the computation, extract partial trajectory with a window size of + (N is the condition trajectory length and M the sample trajectory length) in Eq. 21 and set the training objective as in Eq.
(( = EE-iti eo. N...
[ a 1. e/ll
Le
4+M-1 Σ
am am-11 1, T 1, (co(s
m=t+1
Lm
where the state trajectory condition for the predictor eo in L, is the original T/ (SI-N,..., from the offline dataset D,, and the state trajectory condition in Lm can be expressed - {ă, j E {m N N...., m 1}}. with:
if <t,
8
$ =
/I a,co(3", 11 T/
otherwise (je (t...., i M -2)).
a;
15

---

Published as a conference paper at ICLR 2024
B IMPLEMENTATION DETAILS
B.1
DMBP NETWORK STRUCTURE
Information Extraction
Noise Prediction
State Encoder
Predicted Noise
State Encoder
Layer MLP (Mish)
7pt(5)) t
Action Ennedeer
9c-1
Sin-Pos Emb
Residual Temporal Block
Inputs for Information Extraction Block
Network:
rpt(4f): Repeated state
Residual Temporal Block
Sin-Emb
timestap t and Difuuiion timestep
Down Sample (Convid)
cf Previoudly
trajectory
Group Mish
Up-Sample (ConTTnnsposcld)
Cond
Mish
Extra Inputs for Noise Prediction Block
Operation:
3f Noosee state at timestep c
and Diffusion timestep
Copy and Stack
Re-3: Recovered state at RL timestep t
Flatten
Gg-y timestep
Diffusion timestep
Figure 5: Neural network structure of DMBP.
We implement DMBP based on classical diffusion models (Ho et al. 2020). As discussed in Section 4.1 the naive adoption of MLP neural network structure, which stacks all inputs in one dimension, leads to poor estimation of noise, partially due to the fact that & is more closely related $; than &, with j < t--1 1, and that this information cannot be well captured by the MLP structure. Therefore, we first extract the information from the trajectory with U-net (Ronneberger al. 2015 Janner et al. 2022) (see "Information Extraction" block in Figure 5). Then, the output U-net Ug Ti-1 1 is flattened on one dimension and fed into "Noise Prediction" block together with the noised current state 51, last-step agent generated action a.-1, last-step denoised state 1; and the diffusion timestep
It should be mentioned that directly inputting state and action neural networks may lead poor estimation performance, partially due to the much higher dimensionality of state than that of action. We utilize MI.P-baeed neural networks encoders match the dimensions of action and states. The final output of our Unet-MLP neural network is recorded as 6 6 at-1, T i). Detailed hyperparameter settings be found in Appendix B.2 and ablation studies network structure be found in Appendix
B.2 DMBP HYPERPARAMETERS AND TRAINING
As discussed in Section 4.1 the proposed framework essentially deals with information with small to medium scale noises instead of generating data from pure noise. We redesign the variance schedule to restrict the noise scale in the diffusion process and limit the total number of diffusion timesteps K for predictor training. The generic neural network and the variance schedule hyperparameters be found in B
Since action and state dimensions vary greatly in different environments, choose different numbers of neurons in the proposed MLP-basee encoders in various environments. The corresponding hyperparameter is recorded the embedded dimension Ç, and the action/state encoder can be
16

---

Published as a conference paper at ICLR 2024
expressed FC(2Ç. () with Mish activations. Besides, for different benchmark environments and datasets, we choose different condition trajectory lengths (N) and sample trajectory lengths (MZ to achieve the best performance (ef. Table 4h.
Table 3: Generic hyperparameters of DMBP
Hyper-parametent parameters
Value
Noise prediction network
FC(256,256,256) with Mish activations 0.1
Dropout for predictor network
Learning rate
3e-4
Batch size
64
Variance schedule
3.065, b= 24.552, -
Total diffusion step (K)
10 for denoising tasks
100 for infilling tasks
Table 4: Hyperparameters of DMBP for different benchmark environments and datasets
Embedded dimension
Condition
Sample
Domain
Environment
Dataset
trajectory length (N)
trajectory
(C)
length (M)
halfcheetah
all
64
Mujoco
hopper all
64
walker2d all
64
pen
expert"
128
hammer
"xpertt"
128
Adroit
door
expert"
128
6
relocate
"expert"
128
kitchen
"mixed"
256
Franka
kitchen "complete"
256
Kitchen
kitchen "partial"
256
Different from classical conditional diffusion models, our proposed DMBP utilizes non- Markovian loss function for updating- For each iteration, we sample the partial trajectory (8r-N, Ot-N, 8t-N+1. 8-4) from offline dataset D,. Following the setting in Section 4.2 4.2 we present the procedure for DMBP training in Algorithm
Algorithm 2 DMBP algorithm
Require: offline dataset D., initialized noise predictor Eg
1: for each iteration do
2: Sample trajectory mini-batch ((: at-N, St-N1,......... ..,..-11)11)) D.
3: Sample uniformly distributed diffusion timestep i (1, ..)
4: Sample random Gaussian noise e (0,
5: Produce noised state through 8 Va,st vI a
6: Get Trajectory T { 8
7:
L, | a:-1, 1)
8:
Recover the noised state through 8, Vov
[ VI asco(s) 7 -()1
9:
T POP(PUSH(TT 8,), N:
10:
for m = + 1,,...,,..... M do
11:
Sample en ~
12:
3"n / + vI- a, m
I.m ( am T 1,
c 1|
14:
šm Von m VI- C,5co m a am -.
15:
T = POP(PUSS (TPPUSH(TT-1111 3m), 8m-
16: end for
Update noise predictor eo by minimizing M+ Ln
17:
18:
end for
19:
return Eg
17

---

Published as a conference paper at ICLR 2024
C ADDITIONAL ABLATION STUDIES
As introduced in Section 5.3 our proposed non-Markovian training objective is essential for avoiding eroor accumulation model-based estimation. To demonstrate the efficacy of the proposed approach, we visualize partial trajectory of "hopper" during the test in Figure
Original States (Sy):
Observed Noised States (5p):
DMBP (trained with Markoovian loss function, M 6) Recovered States ($²):
DMBP (trained with classical Markovian loss function, M 1) Recovered States (3²):
Figure 6: Visualization of the denoising effect of DMBP (trained with different sample trajectory length M) alongside Diffusion QL (all trained on the dataset hopper-expert-v2) During the test, the observation is perturbed with Gaussian distributed random noise with std of 0.10, and the action is determined based on the denoising results DMBP trained with M 6.
When the proposed non-Markovian loss function is adopted to train DMBP (see the third subplot of Figure (). DMBP can give accurate predictions the original states in long RL trajectory. Even though prediction errors exist for some RL timesteps, such errors do not collapse future prediction results. However, when the classical Markovian loss function is used to train DMBP (in the fourth subplot of (6) prediction error accumulates, leading to severe prediction errors and significant deviation of the predicted trajectory (away from the actual one).
18

---

Published as a conference paper at ICLR 2024
We further conduct ablation studies the condition trajectory length (N) in Figure When N the noise prediction Eø( ac-1: -;- ,, decays to (g at-1, 1,-1, i.e, the state condition for DMBP reduces to only the last step denoised state 8 (instead of the trajectory Tィー1). Theoretically, this is enough for predicting the noise information reinforcement learning problems modeled as Markov decision processes (MDP), and most moodel-based methods follow this assumption (Lin al. Danner et al., 2019 Yu al. 2020 202T). However, we observe that DMBP performs poorly In the conventional MDP training setting with N = 1. On the contrary. larger condition noised states. This partially due the fact that when predicting the current state (&,), the prediction enror of the previous RI. timestep denoised state
always
trajectory length N leads to better prediction
($ can be compensated by the information extracted on the state trajectory (Tff1). We set N 4 for the "hopper" environment.
Checkpoint
Training (std=0.05)
Training (std-0.10)
Training
50-
80
100
100
60-
60-
60
60
40-
40
40
40
20
20-
20
200
Gaussien 3rd (x10"")
Credient stege (x10")
Gredient steps 1x109
Grediennt steps (x39')
Figure 7:
Ablation studies of condition trajectory length (N) on dataset "hopper--eperrtv2" M is set be for all DMBP.
Lastly, we conduct the ablation studies on our proposed Unet-MLP structure. As discussed in Section 4.1 directly inputting state trajectories and action into MLP networks leads to poor noise estimation. In Figure 8 compare the performance of DMBP with MLP network structure and DMBP with the proposed Unet-MLP network structure. For the MLP network, we stack all information in one dimension as input, and the network hyperparameters are "Noise Prediction" module of our Unet-MIPP structure (cf. Figure 5). For the Unet-MLP network, we follow the basic settings in Appendix B We observe that the MLP network converges faster but achieves much worse denoising performance than Unet-MLP.
be the same the
MLP (N=1 M=
MLP (N=4 M= Unet-MLP (N=1 M=6)
Unet MLP (N=4 M=
Checkpaint
Training (std=0.05)
Training eval (std=0.10)
Training (std=0. 151)
80
80
60-
60-
60
60
40
20
20-
20
20
15
100
200
200
200
Geursion Std (x10"")
Gradient stegs (x10!!
Gredient steps 1x10")
Gradient stepe (x30")
Figure 8: Ablation studies the proposed Unet-MLP structure on dataset "hopper-expert-v2"
D EXPERIMENTAL DETAILS
D. EXPERIMENT SETTINGS
For the baseline RL algorithms (including BCQ (Fujimoto et al. 2019). CQL Kumar et al. 2020). TD3+BC (Fujimoto & Gu 2021), Diffuusion QL (Wang et al. 2022), and RORL YYng et al. 2022 ),
we train them with the suggested hyperparameters for specific environments and datasets. For the sake of fair comparison, we set checkpoints every 10 epochs (1000 gradient steps for each epoch) during the training process of each algorithm, and randomly select from the last 10 checkpoint policies for robustness evaluation.
19

---

Published as a conference paper at ICLR 2024
As for the training process of DMBP, follow the hyperparameter settings in Appendix B.2 We train DMBP for 300 epochs and set the checkpoints every 10 epochs. Analogously, we randomly select 5 from the last 10 checkpoint DMBPs and pair them randomly with the previously selected policies for robustness evaluation.
When DMBP is adopted improve the robustness against noised state observations (denoising tasks introduced Section 5.1). we utilize different diffusion start timesteps (k) (see Eq. (2) for different types and scales of noises (cf. Table|5p. When DMBP is used for improving the robustness of policy against incomplete state observations with unobserved dimension(s) (the infilling tasks introduced in Section 5.2), we set the diffusion start timestep be 100 and the resampling times to be for all benchmark experiments.
Table Choices of the diffusion start timestep under different types and scales of noises
Gaussian
Uniform
Adversarial
Noise Scale
Diffusion start
Noise Scale
Diffusion start
Noise Scale
Diffusion start
(std)
timestep (k)
(norm)
timestep (k)
(norm)
timestep (k)
(0.00, 0.03]
(0.00, 0.04]
(0.00, 0.05]
1
(0.03, 0.06]
2
(0.04, 0.08]
(0.06, 0.09]
3
(0.05, 0.10]
2
(0.08, 0.12]
(0.09, 0.12]
4
(0.12, 0.15]
(0.10, 0.15]
3
(0.12, 0.15]
5
D.2 ADDITIONAL EXPERIMENTAL RESULTS ON MUJOCO
We provide more experimental results in addition the robustness evaluation of DMBP against noised state observation (in Section 5.T and against incomplete state observations with unobserved dimensions (in Section 5.2
Following the basic setting with Gaussian noises defined in Section 5.1 the full robustness evaluation results of five baseline algorithms (BCQ, CQL, TD3+BC Diffusion QL, and RORL) on the Mujoco domain with and without DMBP are presented in Table 6 where we include the training datasets "medium-expert". "medium". and "full-replay". In Table 7 further provide full evaluation results of three other types of noise attacks that have been commonly adopted on state observations (including uniform random noise, maximum It is clear that DMBP greatly enhances the robustness of all baseline algorithms against different types of attacks on state observation for all environments and datasets. Almost all baseline algorithms (without robustness consideration) strengthened by DMBP outperform the SOTA robust offline RL algorithm RORL.
attack, and minimum Q-value attack).
To better demonstrate the robustness enhancement of DMBP on the baseline algorithms under different noise scales, we present the robustness evaluation of CQL and RORI. against four different attacks in Figure
For the robustness evaluation against incomplete state observations with unobserved dimensions (see experiment settings in Section 5.2), we perform the robustness evaluation test under three baseline RL algorithmms, BCQ, TD3+BC, and Diffusion QL, on halfcheetah (cf. Figure I0.. hopper (cf. Figure and walker2d (cf. Figure 12); the training datasets chosen be "medium-expent" "medium" and "full-replay". The evaluation results highlight the robust adaptability of DMBP, i.e., its compatibility with wide range of offline RI. algorithms and diverse datasets.
20

---

Published as a conference paper at ICLR 2024
Table 6: D4RL score of the baseline algorithms and the DMBP strengthened ones trained with expert (c), medium-expert (m-e), medium (m), medium-replay (m-r), and full-replay (f-r) datasets in Mujoco domain under different scales of Gaussian random noise on state observation.
Noise scale
BCQ
CQL
TD3+BC
Dillusion QL
RORL
Eav Dataset
hase
DMBP
DMBP
hase
DMBP
DMBP
base
DMBP
0 96.9+1.8
93.0+6.1
95
92.9-1077
108.+11.22
0.05
4.542.6
60,2423.9
18.1
60.9-22.5
7.3+66.6
77.1
4.843.6
75.2420.7
15.4:3.
55.7=29.2
0.10
4.542.5
26.8+16.2
7.444.0
40.5=16.6
4.743.6
47.54:222
3,342.5
39.8:21.8
3,741,9
32.8=20.4
0 94.333.
920010.6
89.0:12.1
94,4213
106.523.6
m-
0.05
38.915.3
64.8+17.9
34.52998
66.2=19.6
38.84:4,2
72.6+12.7
38.0=7.4
80.1499
55,2+17..3
69.8-15.0
HaliChoctah
0.10
28.9+4.2
48.7+999
26.8+5.3
43.6+16.1
26.91-5.4
52,1+16.7
28.5+3.8
63.5+10.3
21.5++6..
46.2-9.7
0 47.6E09
09.4:009
48.41088
52.5:077
64.2233
0,10
29.31:599
46.241.1
31.7-6.2
41.7=11
28.94118
47.020.8
29.52:5.0
18.1=:16
43.8:9.5
5922-42
0,15 22.14:3.8
43.64:411
246+-588
46.34-26
21.94-5.9
45.4+1.1
21.74588
45.5+6.3
30,5+7.6
57.96.8
416654
4700010
45,2:009
66.7714
0.10 20.6+6.99
38.++1122
35.6+1.
45.8+1.
28.5+:5.5
44.3+-1.
30,1+4.1
45.6+009
43.5+224
619-1122
0,15
14.8+1022
35.143.77
28.8
1..6
24.018.9
42.54226
24.27.66
44.623.00
30.3:5.9
584=122
0 75.5118
78.722.3
75.7ET.
17.E1.
88.1
For
0,10
27.447.4
598444,44
31,8+322
63,4+277
27.94-83
58.3+8844
30,7+333
62.842.1
43.2+6.5
67.5-5.8
0,15
21.246.9
48.411244
25.0=2.00
55.8=24
23,7433.
51.916.3
22.4499
55.142.5
33,92:66
(233779
D 88.44224
109.1+13.7
108.9
104.9+15.1
110443.1
0.05 34 610025.2
41:2-118
79.1128.2
38.2#124
84.82274
56.9134.9
64.3=1988
0,10 24,311099
17.1
24.3-1188
48.8=20.
11.74
32.6118.7
24.02-9.
56.1=17,33
24.1=20.2
37,5=10,5
0 9T.7
106.7113.3
107
105.2116.0
111.32899
m-e
0.05 45.61179
74,43317
35,5=17.5
78.2-26.6
42.6+:4.11
80.7425.0
42.0#14.6
74.3423.5
65.4+31.00
81.0-23.9
0,10
33.4+163
57.2+22.1
30,9=1333
31.5211936.2+333
37,2+19.9
29.8+83
526+1733
30.6=16.7
66.9-26.3
Hopper
0 39.1212.6
73.6214.
39.719.3
804
1026:11511
0.10 29,549.2
50.1
37.8=1399
32.9-2177
30.2412.1
51.3114.1
13.912.3
57.9222.2
42.3+22.99
57.3-21.3
0,15
20.7+8.3
43.9+1888
25,8+11.3
30.9-20.0
20.249.3
42.5+17.2
21.9+799
51.3+21,9
1..+21.9116++130
21.6+13.0
339-17.0
0 78.7
969E88
80.91243
95.7E17.2
103.H008
m-F
0.10
15.7+9.0
6681173
47.5+=21.6
89.1=124
14.4+12.3
71.91:2.5
15.9
85.9-20.9
85.9
103.24133
0,15
11.1772
64.51172
33.7=2122
80.7-16.5
9.6
66.14:22.8
1799115
722422.9511-22..
104.243.
O 89.72239
T021
788:2272
106.740.1
1075433
0.10 17.4+7.3
73.1
$4.3+25.00
103.3+4.22
19.4+7.7
67.2430.3
349+1002
96.0+11.4
69.2+19.8
1016+7.5
0,15 11.7669
64.9225.6
311111.11
97.3
9.4552
64.2427.88
12,9
83.2222.52.5222
83.2422.6
52.5+26.3
99.5=10.8
O 111.6:0.66
108.:11
110.7:03
109.6:0.5
104.:11255
0,10 77,9137.6
110.3-20
97.612199
94.3
72.9139.4
109.2::.55
93.3+27.2
109.114.0
95.4 97.8-20.2
28.24324 0.15
1042-13.5
78.9-33.2
83.4223.3
9.24136
107.545.2
30.54:2.55
94.5118.1
81.6+26.4
84.5226.4
107
11062009
110.240.
114.1111
m-e
0,10
60,2+36.2
7-8.-
90.8-213
105.1+6.7
78.5+-26.1
108.144.5
91.8+
108.643.5
110.6+3.77
113.2+122
Walker2d
0.15
40.7+32.3
105.3=6.2
73.4+29.2
91.3=10.4
37.34-28.
104.51144
50.7+35.9
1068448.4
111443.6
105.348.2
0 71.31133
32.31311
83.6:10.1
86.4212
1013200
0,10 67.2+22.0
74.4+18.3
76.8=15.8
8..3-8.1
81,5++11.3
84.9.7.7
77.7=2004
85.247.3
98.677.5
93.3-11.0
0.15
48.74:28.6
65.8+20.7
64.4+22.9
71.2-9.2
62.34-29.8
80.6+:0.2
23.6+228.3
78.3+14.4
82.3+18.3
14,0
0 50.6131.6
79.92:48
84.719.8
93.121009
88.7119
0,10
14.7=11.11
$3.1
10.
78.7=1.2
40.7422.3
84.418.7
59.6#31.8
92.6
026::10.8886:11.
88.4-25
0,15
11.24-59
$2.9129.9
48.6126.5
73.6=10.11 10.1
16.5+12.8
77.9117.2
19.2+15.7
91.319.6
89.4+12
89.0-4.5
82.24233
928-22
96.7E1.7
992:114
104.55077
fr
0,10
38.5+:28.5
85.6
89.2=69
91.3=332
69.9+-28.3
96.3+222
79.2=29.5
98.728
99.2+7.4
101.5+6.7
0,15
19.3
77.1
68.6128.1
89.1
23.3422.11
92.21555
18.5123.4
97.624477
71.21339
98.5-11.8
21

---

Published as a conference paper at ICLR 2024
Table 7: D4RL score of the baseline algorithms and the DMBP strengthened ones in Mujoco domain under uniformly distributed random noise (U-rand), maximum action-difference attack (MAD), and minimum Q-value attack (MinQ) on state observation.
Dataset/
Noise
BCQ
COL.
TD4BC
Diffasion QL
RORL
Noise Scale
Type
base
DMBP
base
DMBP
base
DMBP
base
DMBP
base
DMBP
U-rand 7.414.9
69.1221.5
27,2164
69.62224
16.311311
842
116:109
1781218
24.32.5
66.8127.0
<
MAD 3.6
52,5+17.9
12.4#6.9
61.2=19.7
4.723.5
65441660
4,343.2
62.92132
14.1=2,5
54,3427.1 1
0.05
Man 12.81933
51,81:2.9
19.4111.3
6044199
18.0142
21.
8.066.7
71.11:52
933888
71,04:29.
U-rand 39332004
08.2211.33
29.1213.1
09.7211
4222289
769229
44.32900
80.7113
41.311311
73.3
I1-0
MAD 38.71288
$4.51:1.0
10.9154
195216.0
34.5132
72.51.00
326-57
76.21111
22.5::4
69.5115.6
HalfCheetah
0.05
MinQ 36.3=6.1
$1.7#12.9
26.9:12.5
58.5-24.0
30.8:7.3
78.3+12.6
45.0=2.0
71521022
55.8=1322
79.544.55
U-rand 44.912.1
47.510.1
44.7:52
m
MAD 40.3=11.
46.442.5
41.0+11
4899=09
39.1220
48.3
36.9=3.1
49.220.88
43.5=19
49.945.3
0.05
ManQ 34.31988
46.11:2.7
49.0111
48.8
55.315.0
U-rand 31.5
40.3539
40.9226
464213
36.926.6
46.911
383
46.E09
39.923
G1.E1T
MAD 19.218.2
29.416.9
29.0126
27.1334
22
34515.5
62.31100
0.10
MinQ 5.1+5.2
36.7488
39.220.8
46.2=11
36.726.8
44.8:111
37.0=4.8
38.61111
34.0=
63.242.3
Fi
U-rand 36.5158
MAD 26.5==3.6
58.455.5
24.2+442
62.3=19
3222113
19.6:7.6
53.324.4
218-23
69.118
63.221.
15
25.3=132
66,7214.3
0.10
MinQ 13.8192 46.1111.9
53.41377
272115
53.11
1355169
7244662
Uradn 46 11 20.7 66.91266.
59.61299
95.71288
42.61244
8401
53.1
20.8
3441255
0.05
MAD 14-144
22.6+13.9
73.9427.9
27.24
342222
368-9.00
37.1-123
36.6-222
59.04338
MinQ 47.4 18.9 62.5
32.7.13.55
58.7-17.9
45.3
95.71.276
66.7-336
59.2123399
79.8 32.7 19.4
U-rand 62. 28.6 54.72215
64,8429.4
96.1=258
00.222993
88.-230
64.82253
1622
35
me
MAD 18.31 13.1 50.1
69.6
85.4
22.2.144
73.31.044
35.6-113
59.51112
65.2-
29.0 63.5
0.05
MinQ
43,2113.9
74.1128.2
8122429.4
8841315
10111221.
38.32256
69.91323
93.3-27.9
85.9228.2
Hogper
radd 47.8 13.2 62.2
62.916611
12.1
48.11113
5851120
472-172
6121177
82.1176
MAD 37.3 48.3113.2
49.8114.7
36.61175
45.7116.0 48.11111
2922162
45.8112
66.312772
719120.5
0.05
MinQ 27.91 10.8 45.21 10.5
59.9118.7
87.0-27.9
19.741826131122
59.2-186
75.41129
63.9-217
18.5
U-rand 18.318.2 68.1199
6541220
339
919ETTY
m.r
MAD 5.1.
37.51
12.1
88.9-1337
6.155.55 6431218
9.918.1
38.31158
5 97.5
0.10
MinQ 5,315,44
18.31184
84.6114.1
87.5166
80.118.1
21.2-25.1
62.5127.3
98.2
2032424
Er
Urrand 28.31 10.3
75.212668
10L7175
26.7410.0
73.31 29.9
46.7-186
105811011
99.9
9866132
MAD 8.615.5
55.91288
1955117.
96.1110
6.3158
69.912388
16.2-733
58.5126.2
65.4126.7
77.31280
0.10
MinQ 14.7.8.1 17.04 18.1 99.141.8
102.344.6
6.8660
77.9129.6
66.0-2442
80.21:226
82.5-20.9
925.229911.416. 11.74
U-rand 102.1=18 110.=08
1499
1060274
106, 1429
110.:003
101.2:10
109.4:05
95.1-15.7 97.2495
0.10
0
MinQ 99.9=22222
MAD 50.5=43.7 10.5
105.6=1.1
99.9++1.8
64.1+27.0
102.4+6.9
97.6=1611
91.94224
19.9#22.7
105.5+13
69.7-175
36.6=35.5
101.+220
88.2-24.8
102.4+13
61.9=292
91
83.8+19.9
13.3
U-rand 91. 91.922665 110.7=111
102.7411.2
108.7775
90.7
110.1110
103.311377
100.=0.
108.3#32
106
0.10
MAD 38.3+33.7 98.54755
73.2433.7
102.5+16.
22.8+2100
373-34.1
99.5-11.3
88.6-117
101
MinQ 75.5=223.7 105.0=5.
73.7::99.8
100.9210.2
70.2+20.3
92.9-14.1
96.7=19.1
101.9=12.1
1365
95,7+14,2
Walker2d
U-rand 77 77.8:1622 75.1
82.6::5.5
815=337
82.6
849:42
855=59
85.3
99.7=38
1010=006
0.10
m
MAD 38.6=27.4 74.7=116.8 59.3+229.5
80.5-5.6 6
41.1=34.3
74.1=8.1
32.2-2955
61.5:1377
77.3-26.6
79.2421.3
MinQ 69.24-1888 11.
71.7420
68.7-49
77.6492
74.6+32
742-124
82.8+5.1
90.0-
U-rand 17.3212.254.92220
69.2420099
78.1-92
$1.2428.3
83.611888
642-218
91.1-121
89.9
88.742.1
0.15
MAD 6.6+3.3 43.44:298
19.7+14.7
784488
88+4.4
10.8-19.1
7242.3
16242
8199
90.5+3.5
MinQ 7,344422 30.3+:261
66.511188
7855=4
21.7+1599
76441149
4722-332
68.0=19.
82.3=144
89.6+1.7
0.15
f-r
U-rand 409=30.6
MAD 5.81-4.5
66.1426.3
85.77777
36.6+29.7
919-13
86.3-.0
00.2228111
76618
26
76.3=29
85.7221
5
63.9-009
98
72.222337294206
MinQ 11.177.1 58.6+31.3
79.342.8
84.1=25
66.65555
63.8-778
79.3+2.00
39.55145,7
22

---

Published as a conference paper at ICLR 2024
CQL
DMBP-CQL
RORL
DMBP-RORL
HalfCheetah
HeltCheetah
Hopper
Hopper
medium-replyy
100-
100
D4NL Score
0.00
0.15
0.05
0.15
0.00
0.00
0.20
0:15
0.00
0.08
0.15
0.05
(a) Performance under Gaussian random noise attack
COL
DMBP-COL
RORL
DMBP-RORL
HelfChestah expert
HollChestah medium repley
Hopper eapert
Hopper medium repley
We kerkd
Walker2d
expert
medium replay
120
120
120
120
120
120
100
300
100
100
102
DaIL Score
80-
80
80
80-
80
60-
60
60
60-
60-
60-
40
40
40-
40-
20-
20-
20-
20-
20-
0-
0-
0-
0-
0.00
0.03
0.10
0.15
0.00
0.05
0.10
0:15
0.00
0.02
0.30
0:15
0.00
0.08
0.20
0:15
0.00
0.08
0.10
0.15
0.00
0.05
0.50
0.15
norm
norm
norm
norm
norm
(b) Performance under uniform random noise attack
CQL
DMBP-CQL
RORL
DMBP-RORL
HelfCheetah ixpert
Hallcheetah
Hopper espert
Hopper neddum--mplay
kerkd
Wiikerdd medium-epply
medium-epply
120
120
120
120
120
320
100-
300
3.0
300
000
100
D4 Score
80
80
80
40
40
20
20
20
0
0.00
0.05
0.10
0.15
0.00
0.05
0.30
0.15
0.00
0.05
0.30
0.15
0.00
0.05
0.30
0.15
0.00
0.05
0.10
0.15
0.00
0.05
0.00
0.15
norm
norm
norm
norm
norm
norm
(c) Performance under the maximum aution-ddifferenn attack
CQL
DMBP-CQL
RORL
DMBP-RORL
HalfCheetah expert
HelfCheeteh medium repley
Hopper medium medium-replay replay
kerkd expert
Walker2d
expert
medium
120
100
100
000
DARL Scate
80-
60
60
40
20
20
20.
0.05
0.15
0.05
0.30
0.10
0.15
0.05
0.15
0.05
0.10
(d) Performance under the minimum Q-value attack
Figure 9: The performance of CQL, DMBP-CQL, RORL, and DMBP-RORL under different scales of (a) Gaussian random noise attack, (b) uniform random noise attack, (c) maximum action difference attack, and (d) minimum Q-value attack. The curves are smoothed with a window size of 3 and the shaded region represents half standard deviation over 5 random checkpoints (20 tests for each checkpoint).
23

---

Published as a conference paper at ICLR 2024
DMBP-BCQ
BCQ (without mask!
TD3+BC QL
DMBB-TD3+BC
TD3+BC BBC (without mask)
Diffuiiifusion QL
Diffusion (without mask)
1000
hallchheatoh-medluumz2
22-
halfcheetah--fflf full--e---l---------
Figure 10: The performance of BCQ, DMBP-BCQ, TD3+BC, DMBP-TD3+BC, Diffusion QL, and DMBP-Diffusion QL (trained on halfcheetah medium expert, medium, and full-replay datasets) with incomplete state observations that have 1-5 unobserved dimension(s). The dimension of state observation halfcheetah is 17.
nedium-eepert,
DMBP-BCQ
BCQ (without maski
TD3+BC
DMBR-TD3+BC
TD3+BC (without mask)
QL
Diffusion mask)
hopper medium expert-V2
100
happer-med um-22
100
happar-tull-rpplayya0
score
7E
30
Figure 11: The performance of BCQ, DMBP-BCQ, TD3+BC, DMBP-TD3+BC, Diffusion QL, and DMBP-Diffusion QL (trained on hopper medium-expert, medium, and full-replay datasets) with incomplete state observations that have 1-5 unobserved dimension(s). The dimension of state observation hopper 11.
24

---

Published as a conference paper at ICLR 2024
DMBP-BCQ
BCQ (without mask!
TD3+BC QL
DMBB-TDD+BCC
DC3+BC (without mask)
DMBP-Diffuzion QL
Diffusion (without mask)
molkkr2d-madlumm mww.uueex-e--e-t2
100
wolkear2d-medlum-
100
75
score
72-
50
number
Figure 12: The performance of BCQ, DMBP-BCQ, TD3+BC, DMBP-TD3+BC, Diffusion QL, and DMBP-Diffusion QL (trained on walker2d medium-expert, medium, and full-replay datasets) with incomplete state observations that have 1-5 unobserved dimension(s). The dimension of state observation walker2d is 17.
D.3 EXPERIMENTAL RESULTS ON ADROIT
We further evaluate DMBP the Adroit domain, where agents are required to control 24-DoF robotic hand to manipulate pen, a hammer, door, and a ball. The action and state spaces in the Adroit domain have much higher dimensions than those in the Mujoco domain, and most of the baseline algorithms perform poorly on "human" and "cloned" datasets. Therefore, only present the robustness evaluation results on the "expert" dataset as shown Table g Again, DMBP significantly enhances the robustness of all baseline algorithms. Stronger robustness enhancement is observed for "policy regularization" offline RL algorithms (i.e., BCQ and Diffusion QL), compared to "Q-value constrain" offline RL algorithms (i.e., CQL and RORL).
For the robustness evaluation against in-complete state observations (cf. Figure 13). we mask more state dimensions (ranging from 20), and the evaluation results demonstrate that DMBP performs well even when roughly half of the state dimensions are masked, especially in "hammer" and "door" environments.
D.4 EXPERIMENTAL RESULTS FRANKA KITCHEN
To demonstrate the efficacy of DMBP, conduct experiments the Franka Kitchen domain, where agents are required to control 9-DoF Franka robot kitchen environment to finish multiple tasks common household items. It is noticed that the Franka Kitchen domain has sparse reward environments, i.e., agent scores 25 when finishes given task and scores otherwise. Therefore, challenging train agent in such environments, and becomes extremely difficult for an agent to obtain any rewards with perturbed state observations. We present the robustness evaluation results of DMBP against noised state observations in Table 1 and against incomplete state observations with unobserved dimensions in Figure [4] We note that DMBP significantly enhances the robustness of all baseline algorithms.
25

---

Published as a conference paper at ICLR 2024
Table D4RL score the baseline algorithms and the DMBP strengthened ones Adroit domain under Gaussian distributed random noise (G-rand), uniformly distributed random noise (U-rand), maximum action-difference attack (MAD), and minimum Q-value attack (MinQ) on state observa-
tions.
Env/ Noise Noise dataset
BCQ
CQL
Diffusion QL
RORL
Scale
Type
base
DMBP
base
DMBP
DMBP
base
DMBP
321.2438.3
606.5
153.4
126.1
G-rand
114.3148.3
111.5 41.2 69.714554
35.7
131.0137,6
21.
121.1131.0 133.7
pen-expent-vl
U-rand 116.9+38.1
114.8=39.2
83.3+73.2
95.5=62.8
130.7433.3
137.8=39.9
128.3+54.9
131.9+46.9
0.10
MAD 112.6+442.9
113.5-46.2
76.9+55.5
87.6-59.8
122.7458.0
131.5+25.9
123.5+47.1
121.8+32.2
MinQ
118.7442.2 121.1=497 83.5266.3
76.1=66.9
129.0:4339
1426:2337
114.9248.7
126.4141.6
G-rand
108.4=50.9
114.1-3332
51.35:4.5
71.6=65.9
128.3:443.9
143.1=39.3
102.6=36.]
124.0=48.5
0.15
U-rand 115.2:149.5 119.8-39.5
68.8166.3
83.0-66.9
131.7.19922
130.8.1887
121.51:7.55 121.0135.8
MAD
111.6162.9 117.1440.
58.3119.88
65.6=55.2
116.6:54.9
129.4-328
103.0257.1 110.2453.5
MinQ 108.4+442.8
113.1=38.3
69.3+55.9
97.5-59.5
119.6:+49.5
135.4=45.6
111.3=448.8
122.4+38.2
0
108.5+1.2
85.4+31.5
105.7+4.8
111.7=53.1
hammer-xxpert-v
G-rand 29.7+-4.77
109.1+
5.7+1.0
48.3-15.5
68.6+-49.8
103.6+16.7
11.7+0.2
38.3+56.5
U-rand 86.9:2282
107.3=13.0
29.6#1.0
76.0=21.7
84.9147.5
107.3
33.2438.4
60,5-40.2
0.10
MAD 5.8+11.9
84.9+31.5
-0.1+0.3
18.5=12.5
16.44184
78.5+31.0
0.0+:0.1
1899-119
MinQ 83.31:37.
109.91122
15.5123.0
$2.2-26.8
33.3132.2
103.5:11.7
11.211.7
46.9137.3
G-rand 0.4::0.6
1018154
0.02:0.3
21.6:0077
33.3146.5
100.1226.2
0.1::0.2
117::0.4
U-rand 51.1=56.66
109.0+5.99
10.6:99.
55.9=23,9
79.0#56.5
106.1
4.5+6.1
40.3=35.7
0.15
MAD 0.11000
69.0112.3
-0.110.0
11.9-139
4.0.11.3
66.2134.6
-0.110.0
9.2.5.2
MinQ
26.4147.3
99.3=1.2
0.911.2
31.3-22.5
0,540.9
99.2432.9
0.140.1
22,2-13.3
0
104.747.7
106.6:00.9
103.647.3
117.844.6
G-rand 88,2433.2
83.5+36.0
88.3423.6
1032299
95.6+21.
103.5:9.2
65.3448.2
95.3=27.4
dour-cxpert-vi
U-rand 104.4+5.3
102.9-10.6
103.0+3.7
103.6+14.3
101.5+18.5
105.1+14
86.5+440.4
98.5+26.7
0.10
MAD 97.31:9.6
103.246.9
31.0121.7
74.5-26.6
92.943588
103.747.8
30.4123.8
86.6136.5
MinQ 95.6++25.0
99.3+19.9
53.5+41.7
105.1=3,1
100.6+13.88
103.2+5.1
86.1+35.8
102.0=14.6
G-rand 44.1144..6
82.5119.2
25.5133.2
102.9_8.6
53.2146.1
101.5116.0
14.0131.1
95.7-25.2
0.15
U-rand 95.9122477
100.919.8
89.2426.2
98.7-20.4
93.7422.0
101.111.00
73.7442.2
95.9-18.1
MAD 37.7133.77
84.2.115.8
0.2.10.4
62.8-38.5
54.8133888
81.9
7.411177
73.7
MinQ 74.3144.5
94.8.127.2
0.111.11
97.3117.
86.1125.5
103.713.2
42.7138.3
100.5114.9
66.2430.8
76.2436.6
101.2+1277
43.3+20.5
relocate-cxpert-v
G-rand 2.0:44.9
4210:33.5
0.9.2.9
34.6-317
13.3425.1
96.6124.3
1.1.0.1
27.311.3
U-rand 50.9+37.88
65.8+39.0
2.343.9
41.4=2.11
69.8+42.9
98.0+13.6
12.1=2.1
35.7=31.7
0.10
MAD 0.110.2
34.7122.8
0.21102
24.2-21.0
0.110.5
66.813077
0.010.1
15.8121.0
MinQ
57.7135.3
52.9429.8
0.2400.
37,5=25.8
3,4422
70.6137.1
0.940.8
34.2-22.1
G-rand 0.2.104
48.3.133.7
0.0.1.11
34.2-112
1.313.5
87.2.2035
4..110.0
22.011555
U-rand 8.1111..3
47.4125.5
0.811.9
36.1-30.0
32.5122.5
93.116.77
0.110.1
30.3122.7
0.15
MAD
0.0100.1
33.6111.5
0.010.0
28.2-219
0.0.0.3
67.1132.5
-0.120.1
16.3-18.5
MinQ
17.2
42.2:123.5
0.1:0.11
25.6
0.710.4
50.2131.6
-0.110.0
9,415.4
26

---

Published as a conference paper at ICLR 2024
BCQ
DMBP-BCO DMBP-DITrusio
BCQ (without mask)
Dimusioo
(without
pen-expert-v1
150
DMHL
70
(S4RL 11314
10
20
10
36
10
20
expe-expert.v1
DMAL 81270
50
8
10
12
14
16
18
20
2
10
12
14
36
18
20
palocate-oxparte
DNR MCOT
100
30
20
20
Figure 13: The performance of BCQ, DMBP-BCQ, Diffusion QL, and DMBP-Diffusiin QL in Adroit domain with incomplete state observations that have 2-20 unobserved dimensions. (The dimension of state observation 45 pen, 46 in hammer, and 39 in both door and relocate.)
DMBP-COL DMBP. Diffusion QL
(without
Diffusion QL
Diffusion QL (without mask)
kitchen-complete-vn
75
D4-2
25
(168) sx0re
75
50
25
20
kitchen-mioed-v0
100
DARL store
50
1
10
12
18
18
20
3
10
14
36
10
20
Moskied diimension number
Figure 14: The performance of CQL, DMBP-CQL, Diffusion QL, and DMBP-Diffusion QL in Franka Kitchen domain with incomplete state observations that have 2-20 unobserved dimensions. (The dimension of state observation in kitchen is 59.)
27

---

Published as a conference paper at ICLR 2024
Table 9: D4RL score of the baseline algorithms and the DMBP strengthened in Franka Kitchen domain under Gaussian distributed random noise (G-rand), uniformly distributed random noise
(U-rand), maximum defion-differrnce attack (MAD), and minimum Q-vauue attack (MinQ) on observation.
Enw/
Noise Scale
Noise Type
BCQ
CQL
Diffusion QL
RORL
dataset
base
DMBP
base
DMBP
base
DMBP
basc
DMBP
7,54:15,9
47.3=32.4
84.5:127.7
22.81126.8
kitchen-complee--w
G-rand
0.844.3
5.0112.9
0.0=0.0
34.3125.6
0.8:14.3
65.8122.4
0.010.00
12.3=17.8
U-rand
0.8+4..3
4.5+9.6
0.8=4.3
31.8+24.0
0.8444.3
60.0:29.6
0.0=0.00
13.0=18.1
0.05
MAD
0.8144.3
12.5117.8
0.0-0.0
10.81184
2.547.5
42.0420.0
0.0:0..
6,511488
MinQ
0.04:0.0
6.341288
0.8-4.3
17.3420.2
0.0+:0.0
56.84288.0
0.0=0.0
9.8=17.9
G-rand
0.342.5
L3234
0.0=0.0
32.3=24.4
0.0:00.0
610+330.5
0.0:0.0
6.0=14.2
U-rand
0.8+-4.3
3.3+8.4
0.0-0.0
23.5+27.1
0.0++0.0
53.0+225.4
0.0+0.0
8.0+16.4
0.10
MAD
0.04-0.0
4.0+9.2
0.0-0.0
9.8-11888
0.0.0..0
34.3+-25.9
0.0+0.0
3.5+8.7
MinQ
0.01:0.0
5.8.1006
0.0=0.0
23.311770
0.0.1..0
47,8122.4
0.0.:..0
65114.6
29.3+18.3
31.334.5
71.42232
30,3423.0
G-rand
6,3+-1177
1188+1.0
0.0-0.0
32.5+24.4
12.5+17.8
65.842224
0.0+0.0
18.8+20.9
kitchen-partial-or
U-rand
3.318.44
10.0118.1
0,8
28.5122.77
22.5:126.3
60.01129.6
1.5.5.8
21,5
0.05
MAD
6.54:4488
5.311300
0.0-0.0
22,5426.3
5.0:112.7
48.3122.7
13155
13.3-18.1
MinQ
5.04:129
5.0+12.7
3.3884
29.8-22.9
4.34:12.1
40.0428.6
0.0+0.0
15.3-19.
G-rand
0.3.14.3
1.311T5
0.0
26.82223
7,311558
64.31213
0.010..0
16.0
U-rand
2.3:17.2
7,5
0.0=0.0
23.3427.1
4.3.112.3
55.8128.5
0.513.5
17.3=20.2
0.10
MAD
2.547.55
4.349.4
0.0=0.0
14.341888
0.844.3
41,3419.6
0.0=0.0
9.84179
MinQ
5.0+:10.0
10.8:18.4
0.0=0.0
12.5+20.0
3.348.4
39.5+18.1
0.0=0.00
12.0=19.8
18.3122.22
51.334.55
65.31121.
33,54:2499
kitchen--iise camnen-mixed-vi -voodd-vv
G-rand
7.3ETI.3
26.32200
0:0-0:0
30.82233
11.311888
49.3422.8
0.825..
19.5
U-rand
9.34
18.8+20.9
0.0=0.0
32.8224.6
22.5:+26.2
55.0426.9
0.0=0..0
12.3=19.8
0.05
MAD
5,8+144.0
15.0+1922
0.0-0.0
26.3422.0
13.3+18.2
42.3420.1
2.3+7.2
13.0=18.1
MinQ
0.8+-4.3
15.8+19.7
34.3+22.6
9.34-7.66
40.0+-28.6
1.5+5.8
G-rand
1.315,4
14.91189
1.8-64
28.31122.5
0.0.1..0
50.0113.7
0.0.0.0
9.11
U-rand 5.0+:10.0
16.8+200.0
1.5-588
30.3++22.9
1.545588
50.8+34.3
0.0=0.0
11.5=1888
0.10
MAD
0.0+00.0
12.5+2002
0.0-0.0
1788+20.3
2.5+7.5
40.0++28.6
0.8+4.3
12.8-17.8
MinQ
0.844.3
13.3118.
25.81219
D.5 COMPUTATIONAL COST
In Table T0 we compare the computational cost of DMBP for multiple baseline algorithms single machine with one GPU (NVIDIA RTX4090 24GB) and one CPU (AMD Ryzen 7950X). For each algorithm, perform the training halfcchetah- time (1000 gradient steps for each epoch) and the GPU memory usage over 10 epochs. Although the training of DMBP requires 200-300 epochs) of the baseline algorithms (which usually converge in 1000-3000 epochs), i.e., the total training time of DMBP is similar that of the baseline algorithms in comparison.
dataset, and measure the average epoch
computing time per epoch, DMBP expedites the convergence (in
Table 10: Computational cost comparison
Algorithms
Runtime (s/epoch) GPU Memory (GB)
BCQ
4.33
1.36
CQL
19.58
1.33
TD3+BC
2,82
1.29
Diffusion QL
12.05
1.69
RORL
22.63
1.44
DMBP
194.08
2.22
Table 11: Decision time consumption comparison (1000 timesteps)
Runtime(s)
Runtime(s)
Runtime(s) (I 5)
Decision algorithms
Decision algorithms
(k = 1)
BCQ CQL
0.543
DMBP-BCO
2,411
9.980
0.491
DMBP-COL.
2.323
9.776
TD3+BC
0.365
DMBP-TD3+BC
2.215
9.698
Diffusion QL
3.388
DMBP-Diffusion QL
4.786
12.241
RORL
0.525
DMBP-RORL
2.439
10.143
28

---

Published as a conference paper at ICLR 2024
We also perform experiments to measure the decision time consumption for baseline RL algorithms and the DMBP improved ones (see Table [T). We compare the total runtime over 1000 RL timesteps of the baseline algorithms (trained on malficheetah-eppertee v2 dataset) with and without the DMBP denoising framework. For the start diffusion timestep of DMBP in denoising tasks, we select two values of k and k 5. DMBP consumes more decision time within acceptable range.
To expedite the training process of DMBP, one can optimize the training schedule by adjusting the number of diffusion steps (Chung et al. 2022] Franzese al. 2022) and the design of noise scale (Nichol & Dhariwal 2 through learning, which have been proven effective in speeding up the convergence rate. Analogously, to expedite the testing process of DMBP, one can employ implicit sampling techniques, such as Song et al. (2020a); (Zhang et al. (2022a). which aim to reduce the number of reverse denoising steps required during the testing phase. The proposed methods will be subject to verification in our future work.
D.6 EXPERIMENTAL RESULT ANALYSIS
In our experiments, we find that DMBP achieves better robustness enhancement performance when it is combined with offlin RI algorithms based on policy regularization (BCQ, TD3BC, Diffusion QL), as opposed to those based on conservative Q estimation (CQL, RORL). Among the baseline algorithms we evaluate, Diffissin QL demonstrates the best performance when paired with DMBP, while the least improvement exhibits for RORL. For the purpose of understanding the underlying mechanisms, employ t-SNE (Van der Maaten & Hinton [0088] Chan et al.) (2018) to visualize the distributions of states and actions encountered during the training and testing processes of these offline RL algorithms.
As shown in Figure 15. we train various baseline offline RL algorithms using the halfcheetah-expertt v2 dataset. The visualization depicts the distributions of states and actions in the dataset, well as during the testing process without any perturbations, across 20 epochs (each epoch consisting of 1000 RL timesteps). The test results indicate that policy regularization based algorithms tend to replicate the policies observed in the dataset during the testing process. They also exhibit preference for accessing states that are present in the dataset. This phenomenon is particularly evident in the case of Diffusion QL. where the training and testing sets largely overlap. For conservative Q based algorithms, on the other hand, the overlap between the training and testing sets is less significant (particularly for RORL). By the nature of the Diffusion Models, DMBP has a tendency recover states that frequently appear in the dataset. This explains why DMBP exhibits better robustness enhancement performance when paired with policy regularization based offline RL algorithms.
D4RL deteset
Policy in clean env
Diniiutioon
Accion Diniiuuiion
Figure 15: Visualization of the state and action distributions of the training datasets (halfcheetah-
expert-v2), and the generated data during the testing process in the clean environment for various baseline offline RI. algorithms (BCQ, CQL, TD3+BC, Diffusion QI. RORI).
29

---

Published as a conference paper at ICLR 2024
We note that DMBP demonstrates certain level of generalization ability for unseen states and actions in the dataset. We visualize the distributions of states and actions for the agent in the clean environments, the agent in the noised environments with the assistance of DMBP, and the agent in the noised environments without the assistance DMBP (see Figure 16) There is significant overlap between the state and action distributions obtained by the agent in the clean environment (the green scatter plot) and those from the agent assisted by DMBP in the noisy environment (the red scatter plot). This indicates that DMBP is able accurately infer the original states during testing, even in cases where some states are not observed in the training set. Without the assistance of DMBP, the distribution of actions taken by the agent in the noisy environment, as well as the distribution of states encountered during the process (the purple scatter plot), are completely different from those observed in the clean environment.
Policy clean city
Policy noised
DMBP-Policy nolsed
TRUNC
Oiftatinn
KORL
State
Figure 16: Visualization of the state and action distributions for the agent in the clean environments, the agent in the noised environments with the assistance DMBP, and the agent in the noised environments without the assistance of DMBP. The noise is set to be Gaussian distributed with a std of 0.03. The baseline RL algorithms (BCQ, CQL, TD3+BC, Diffusion QL, RORL) and DMBP are all trained on halfchhettah-expert-22.
E LIMITATIONS
One limitation of our present work is that the noise scale related hyperparameter (i.e., the diffusion start timestep & Eq. 2) is manually defined when DMBP is deployed for online tests in environments with different scales of noises. In future work, we will consider self-adaptive diffusion start timestep to make DMBP more adaptive for environments with varying noise scales.
Another limitation of this work is the lack of validation of DMBP's performance with normalized state observations (as in /Zhang et al.] (2020); Yang et al. (022)). Without normalizing the state observation, adding noise of the same scale may introduce a bias the corruption's 's effect on different dimensions. In future work, we will validate and optimize the performance of DMBP with normalized state observations.
Note that the recorded states during the testing process in the noised environment are the original states other than the noised ones.
30


Reviews:
[{"review_id": "Rrxri7LrJp", "reviewer": "Reviewer_ad2c", "summary": "When facing observation perturbations, reinforcement learning agent may perform very poorly. Unlike many prior work which train a robust policy, this paper instead utilizes diffusion model as noise reduction tool to retrieve accurate state information. Then the proposed framework feeds the de-noised state to any offline RL algorithm. Additional techniques are introduced, e.g., specialized loss to facilitate multi-step diffusion accuracy. Extensive simulations are conducted on MuJoCo control tasks.", "questions": "1. When training different algorithms, do you use the same dataset size and batch size for all algorithms? \n\n2. In offline RL, especially the theoretical community, we are often concerned with the data coverage problem (full coverage vs partial coverage). I am interested in the performance of DMBP in this scenario. In particular, D4RL has millions of transitions in its dataset. Although one can argue that this is not that much for a continuous control task, we sometimes see more extreme case in tabular setting where the data is extremely limited, e.g., barely supporting the optimal policy's state-action visitation. I conjecture that if DMBP is paired with algorithm like VI-LCB[1] which is proven to be statistically efficient, it can also facilitate offline RL with scarce data setting. Do you have any insight on this?\n\n[1]Li, G., Shi, L., Chen, Y., Chi, Y., and Wei, Y. (2022a). Settling the sample complexity of model-based\noffline reinforcement learning. arXiv preprint arXiv:2204.05275\n\nOverall I think this is a great paper, and it should be a good contribution to ICLR.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed framework uses the latest diffusion model to solve offline RL with state perturbations. I believe there is a strong motivation behind this approach. Indeed, an important RL problem is studied with state-of-the-art tool.\n\n2. The paper is well-written. The motivations behind each section is quite clear. I enjoyed reading the paper a lot.\n\n3. I believe the contribution is solid in this paper: (a) proposed a non-Markovian loss that facilitates multi-step diffusion accuracy; (b) a tractable version of its VLB is proposed; (c) thorough simulation and ablation studies\n\n4. The proposed framework is very flexible and can work with any good offline RL algorithm.", "weaknesses": "There is no notable weakness in my opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EEFfIX0qcs", "reviewer": "Reviewer_YkkL", "summary": "This paper addresses the challenges in offline reinforcement learning (RL), particularly the robustness against state perturbations. The authors introduce the Diffusion Model-Based Predictor (DMBP), a novel framework that employs conditional diffusion models to recover actual states in state-based RL tasks and proposes a non-markovian loss function to mitigate error accumulation. The framework is designed to handle incomplete state observations and various scales of random noises and adversarial attacks, improving the robustness of existing offline RL algorithms. DMBP is empirically evaluated, demonstrating significant performance improvements in terms of robustness against different perturbations on state observations without leading to over-conservative policies.", "questions": "1. In the experiments, results show DMBP significantly improve the robustness performance of existing offline RL methods. The review noticed that most offline RL methods used are value-based. How will DMBP perform when used with weighted imitation learning methods, e.g., IQL?\n2. Is DMBP sensitive to hyperparameters (i.e., a, b, c) of the variance schedule? Any guidance to choose these hyperparameters?\n3. It is intutitve that diffusion denoising can perform well for random noise. Can the authors provide insights why it also works well for adversarial attacks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is significant for this paper to promote the robustness of offline RL methods.\n2. This paper shows its novelty by applying diffusion method to improve the robustness of offline RL against state perturbations, especially designing the non-Markovian loss function to reduce the accumulated error in state estimation.\n3. The proposed method demonstrates strong performance and has been extensively evaluated with different offline RL methods and noise types and in various benchmark tasks. \n4. This paper is generally well written and easy to follow and covers well related works.", "weaknesses": "1. One main concern is that observation perturbations can be potentially addressed by traditional methods, especially when we assume observation perturbations does not affect reward and transition functions. The reviewer is curious of how offline RL methods with Kalman Filter perform compared to DMBP?\n2. Minor issue: all results in tables should be shown with standard deviations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bBbBIj1hW5", "reviewer": "Reviewer_FLnj", "summary": "Offline reinforcement learning (RL) enables training without real-world interactions, but faces challenges with robustness against state observation perturbations caused by factors like sensor errors and adversarial attacks. The Diffusion Model-Based Predictor (DMBP) framework has been introduced to address these issues by predicting actual states using conditional diffusion models, focusing on state-based RL tasks. Unlike traditional methods, DMBP leverages diffusion models as noise reduction tools, enhancing the resilience of existing offline RL methods against various state observation perturbations. The framework utilizes a conditioned diffusion model to estimate current states by reversely denoising data and incorporates a non-Markovian loss function to prevent error accumulation. DMBP's advantages include improved robustness against different scales of noise and adversarial attacks, as well as the ability to manage incomplete state observations, making it suitable for real-world scenarios like robots operating with malfunctioning sensors.", "questions": "Same as the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. DMBP strengthens the resilience of existing offline RL algorithms, allowing them to handle different scales of random noises and adversarial attacks effectively.\n2. Unlike traditional approaches, DMBP leverages diffusion models primarily for noise reduction, rather than as generation models. This innovative use helps in better state prediction and recovery against observation perturbations.\n3. The framework introduces a non-Markovian loss function, specifically designed to prevent the accumulation of estimation errors over the RL trajectory.\n4. DMBP offers a more balanced approach than methods that train robust policies against worst-case disturbances. This ensures that policies do not become overly conservative, which can hinder performance in certain scenarios.\n5. DMBP's inherent ability, derived from the properties of diffusion models, allows it to effectively manage situations with incomplete state observations. This is particularly valuable in real-world applications, such as when robots operate with compromised sensors.", "weaknesses": "1. The paper has the assumption that perturbation on state space follows a Gaussian distribution. However, in practical settings, such perturbations might be biased and skewed. For example, a water drop on a camera might lead to distortion of captured images. Therefore, it might be better if the authors could elaborate on the assumption of the perturbations and how they are produced. \n2. Introducing a diffusion model for noise reduction could increase the complexity of the model, which may lead to longer training time and resource-intensive computations. The authors might want to include more details regarding training time and computational resources.\n3. Scaling to very large state spaces or handling very noisy environments might pose challenges, especially when using diffusion models. It would be better for the authors to consider testing on tasks with larger state spaces such as the humanoid. \n4. Eq.(2) is wrong as the recovered state should not be a probability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X1OXpY9WR4", "reviewer": "Reviewer_As3y", "summary": "The paper presents DMBP, a novel framework for robust offline RL. DMBP addresses the challenge of handling state observation perturbations by employing conditional diffusion models to recover the true states from perturbed observations. Additionally, it proposes a non-Markovian training objective to reduce the cumulative errors by minimizing the sum entropy of denoised states along the RL trajectory. Experimental results show that DMBP significantly enhances the robustness of current offline RL algorithms, effectively addressing various perturbations and incomplete state observations.", "questions": "My questions are listed in the \"Weakness\" part:\n\n* The authors are suggested to provide a more comprehensive literature review.\n\n* It is necessary for the authors to clarify whether the observations are normalized. If not, it would be beneficial to include a comparison and provide further clarification.\n\n* The reviewer is interested in knowing if the diffusion model can also handle training with incomplete observations.\n\n* The authors are recommended to include information about the training time and inference time of the proposed method.\n\n* Fix the typo.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Overall, this paper is well-organized and easy to follow.\n\n* The introduction of the diffusion model as a means to recover true states from perturbed observations, addressing the challenge of observation perturbation, is both novel and sound. Besides, the inclusion of the regularization of sum entropy for denoised states across the RL trajectory is highly meaningful in the context of sequential decision-making, effectively mitigating error accumulation.\n\n* The proposed method can be applied to many model-free offline RL algorithms and significantly improves over prior works in terms of observation robustness and masked observations.\n\nI believe this paper represents a significant contribution to the field and will have a substantial impact on the research community.", "weaknesses": "* The literature review section lacks a discussion of another taxonomy: training-time and testing-time robustness. While this paper and many others focus on offline RL for testing-time robustness, there is another group of offline RL works that investigate training-time robustness [1] [2] [3]. These works involve corrupting the offline dataset and evaluating it in a clean environment. It would be beneficial to include the training-time works for a more comprehensive literature review.\n\n* It is not clear whether the states in this paper are normalized. Normalizing the observations would ensure fair observation corruption and may potentially impact the recovery ability of the diffusion model.\n\n* Currently, it seems that the diffusion model is trained using complete observations even in the experiments with incomplete observations. I am curious to know if the diffusion model can handle training with incomplete observations as well.\n\n* Additionally, the authors are suggested to report the training time and the inference time of the proposed method.\n\n* Typo: page 2 'A diagram of the proposed approach is shown in the fight subplot of Figure 1' --> 'A diagram of the proposed approach is shown in the right subplot of Figure 1'.\n\n[1] Wu F, Li L, Xu C, et al. Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks[J]. arXiv preprint arXiv:2203.08398, 2022.\n\n[2] Zhang X, Chen Y, Zhu X, et al. Corruption-robust offline reinforcement learning[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2022: 5757-5773.\n\n[3] Ye C, Yang R, Gu Q, Zhang T. Corruption-Robust Offline Reinforcement Learning with General Function Approximation. arXiv:2310.14550.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)