ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
Authors: Zeyuan Liu
Zhihe Yang
Jiawei Xu
Rui Yang
Jiafei Lyu
Baoxiang Wang
Yunjian Xu
Xiu Li
Venue: NeurIPS 2025 poster ()
Keywords: Offline RL
Diffusion Model
Corruption-robust Offline RL
Primary area: reinforcement_learning

Abstract: Real-world datasets collected from sensors or human inputs are prone to noise and errors, posing significant challenges for applying offline reinforcement learning (RL). While existing methods have made progress in addressing corrupted actions and rewards, they remain insufficient for handling corruption in high-dimensional state spaces and for cases where multiple elements in the dataset are corrupted simultaneously. Diffusion models, known for their strong denoising capabilities, offer a promising direction for this problem—but their tendency to overfit noisy samples limits their direct applicability.
To overcome this, we propose **A**mbient **D**iffusion-**G**uided Dataset Recovery (**ADG**), a novel approach that pioneers the use of diffusion models to tackle data corruption in offline RL. First, we introduce Ambient Denoising Diffusion Probabilistic Models (DDPM) from approximated distributions, which enable learning on partially corrupted datasets with theoretical guarantees. Second, we use the noise-prediction property of Ambient DDPM to distinguish between clean and corrupted data, and then use the clean subset to train a standard DDPM. Third, we employ the trained standard DDPM to refine the previously identified corrupted data, enhancing data quality for subsequent offline RL training. A notable strength of ADG is its versatility—it can be seamlessly integrated with any offline RL algorithm. Experiments on a range of benchmarks, including MuJoCo, Kitchen, and Adroit, demonstrate that ADG effectively mitigates the impact of corrupted data and improves the robustness of offline RL under various noise settings, achieving state-of-the-art results.

ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
Zeyuan Zhihe Yang?; : Jiawei Xu²; Rui Yang", Jiafei Lyu', Baoxiang Wang', Yunjian Xu Xiu Li
Tsinghua Shenzhen International Graduate School, Tsinghua University.
The Chinese University of Hong Kong,
'The Chinese University of Hong Kong, Shenzhen,
University of Illinois Jibana-Champann
Abstract
Real-world datasets collected from sensors or human inputs are prone to noise and errors, posing significant challenges for applying offline reinforcement learning (RL). While existing methods have made progress in addressing corrupted actions and rewards, they remain insufficient for handling corruption in high-dimensional state spaces and for cases where multiple elements in the dataset are corrupted simultaneously. Diffusion models, known for their strong denoising capabilities, offer a promising direction for this problem-but their tendency to overfit noisy samples limits their direct applicability. To overcome this, we propose Ambient Diffusion- Guided Dataset Recovery (ADG), novel approach that pioneers the use of diffusion models to tackle data corruption in offline RL. First, we introduce Ambient Denoising Diffusion Probabilistic Models (DDPM) from approximated distributions, which enable learning on partially corrupted datasets with theoreti- cal guarantees. Second, use the prede-prr property of Ambient DDPM to distinguish between clean and corrupted data, and then use the clean subset to train standard DDPM. Third, employ the trained standard DDPM to refine the previously identified corrupted data, enhancing data quality for subsequent offline RL training. A notable strength of ADG is its versatility- it can be seamlessly integrated with any offline RL algorithm. Experiments on range of benchmarks, including MuloCo, Kitchen, and Adroit, demonstrate that ADG effectively miti- gates the impact of corrupted data and improves the robustness of offline RL un- der various noise settings, achieving state-of-the-an results. Our code available at https::/ / thub. com/ //s//anddnn nine/ ADG.
Introduction
Offline reinforcement learning (RL) has emerged as a prominent paradigm for learning decision- making policies from offline datasets [20, 11]. Existing approaches can be broadly categorized
into MDP-based methods [11, 10, 18, 19, 4, 5, 12] and non-MDP methods [6, 16, 29). However, due the data-dependent nature of offline RL, it encounters significant challenges when dealing
with offline data subjected to random noise or adversarial corruption [42, 41, 22, 36, 37]. Such disturbances can cause substantial performance degradation result in pronounced deviation from
Equal contribution Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Step Detector Training with
Step3: Dutaset Recovery Collaboration and Denoiser
Noised Slice
Slice Window
Detector Fg(t)
Detecter
ty(f)
Slice Window
Step2: Denoiser Training with Naive DDPM
Recevered Siice
Denoiser
*a((9)
Mask
Ves
Judde
Denoiser +9
Capy original data
Clonn Sa
CampleddSpppla
Raceoard Sample
Overview of the training processes for the detector and denoiser (left) and the dataset recovery process (right) in the proposed ADG method.
Figure 1:
the intended policy objectives. Therefore, ensuring robust policy learning is crucial for offline RL to function effectively in real-world scenarios.
Several previous studies have focused on the theoretical properties and certification of offline RL under corrupted data [43, 40, 33. 7, 39]. Empirical efforts have resulted in the development of uncertainty- algorithms utilizing Q-ensembles [40] and robust value learning through Hu- ber loss and quantile estimators [37]. Additionally, sequence modeling techniques have been applied to mitigate the effects of data corruption while iteratively refining noisy actions and rewards in the dataset [34]. However, as noted in [34], recovering observations remains challenging due to the high dimensionality.
In recent years, diffusion-based generative models [30, 14, 31] have gained considerable attention for their ability to effectively model complex data distributions, making them increasingly important in offline RL [15, 23]. One promising application of these models is their usage in reducing or mitigating noise in data, thanks to their inherent denoising capabilities. For instance, DMBP [38] proposed a diffusion- based framework aimed minimizing noise in observations during the testing phase. Despite its success, this approach is specifically designed to work with clean training data and only addresses data perturbations during testing. cannot manage perturbations the training dataset, as current diffusion models typically assume the dataset is entirely clean or has a consistent noise distribution across all data points [1, 8). Therefore, naive diffusion methods can encounter challenges, such as overfitting to noise data, when applied to partially corrupted data during training.
To gain deeper insight into which aspects of corrupted datasets degrade the performance of offline RL algorithms, we evaluate several offline RL methods on three types of datasets: a clean dataset, a partially corrupted dataset, and filtered dataset, which is created by removing the noisy portions from the corrupted data, as illustrated in Figure 2 (a-c). Surprisingly. We find that current offline RL algorithms fail to fully restore their performance on the filtered dataset. particularly when the dataset size is limited. These results imply that the loss of critical sequential information plays a key role in performance degrading of decision- makinn.. Additionally, while sequence modeling methods [6, 34] exhibit robustness against data corruption, they fail to function when faced with incomplete trajectories. This finding indicates that simply filtering out noisy samples is insufficient for handling corrupted datasets, underscoring the importance of recovering corrupted data.
To address this issue, we introduce the first diffusion-bused deneising framework for handling cor- ruption robust offline RI. Our approach recovers clean data purely from the corrupted dataset, with- out requiring any external information or supervision. We name this novel three-stage diffusion- Certusion-Guidee Dataset Recovery (ADG). The detailed diagram is pre-
based method Ambient sented in Figure 1. In the first stage, we introduce Ambient Denoising Diffusion Probabilistic Mod- els (DDPM) from approximated distributions, which enable diffusion training on partially corrupted datasets with theoretical guarantees. In the second stage, leveraging the coise-prediction property of the well-trained ambient DDPM, we identify the corrupted data within the dataset. The remaining clean data then used to train a denoiser within the framework standard DDPM. Finally, in the third stage, we apply the standard DDPM to refine the previously identified corrupted data. The corrected data combined with the clean data to form high-quality dataset, which is subsequently used for offline RL training.

---

80
80
80
80
Normaliz anzliized Score
60
60
20
20
20
20
Dataset
100%
(a)
(b)
(c)
(d)
COL IOL
RIOL
ROT
Figure 2: The performance baseline algorithms (CQL, IQL, RIQL, DT, and RDT) is evaluated under four dataset conditions: Clean (original dataset without corruption), Noised (dataset corrupted using Random State Attack as described in Appendix C.I), ), Filtered (Noised dataset with corrupted samples removed), and Recovered (dataset recovered using ADG). The comparison results show the average normalized scores across three MuJoCo tasks (halfcheetah, hopper, and walker2d) using the "medium-replay-v2" datasets. We include the results for both the 100% dataset and 10% dataset settings. ADG effectively restores performance close to the results on clean datasets.
We find that ADG is overall competitive with the ideal case of perfectly filtering out all noisy samples for MDP-based algorithms and significantly outperforms the filtering method for sequence modeling methods, as shown in Figure 2(d). Additionally, we provide comprehensive analysis of ADG's per- formance under both random and adversarial data corruption scenarios, examining various levels of data availability, whether full limited dataset size, as described in Section 5. Our empirical studies demonstrate that ADG consistently enhances the performance of all baseline algorithms, achieving remarkable overall improvement. These findings indicate that ADG versatile and compatible with any offline RL approach, including their robust variants. Notably, ADG exhibits consistent and robust performance across a variety of dataset qualities, corrupted scales, and corrupted ratios.
2 Related Works
Robust Offline RL. Several works have focused on testing-time robustness against environment shifts [28, 35, 26, 38, 34]. For training-tine time robustness, Li et al. [21] investigate various reward attack strategies in offline RL. and reveal that certain biases can unintentionally enhance robustness to reward corruption. Wu et al. [33] introduce a certification framework to determine the tolerable number of poisoning trajectories based on different certification criteria. From theoretical per- spective, Zhang et al. [43] propose a robust offline RL algorithm utilizing robust supervised learning oracles. et al. [40] introduce uncertainty weighting to address reward and dynamics corruption, offering theoretical guarantees. Ackermann et al. [2] develop contrastive predictive coding-based approach to tackle non-stationarity offline RL datasets. Yang et al. [37] utilize the Huber loss to manage heavy-tailedneet and adopt quantile estimators balance penalization for corrupted data. Additionally, Xu et al. [34] introduce a sequential modeling method to iteratively correct corrupted data for offline RL.
Diffusion Models in Offllie RL. Diffusion-based generative models [30, 14, 31] have been ex- tensively utilized for synthesizing high-quality images from text descriptions [27]. More recently, they have gained significant attention in the RI. community, serving as behavior policy replicators [32, 13), trajectory generators [15, 3, 23], and state denoisers [38].
3 Preliminaries
RL and Offline RL. Reinforcement Learning (RL) is typically formulated as Markov Decision Process (MDP) defined by the tuple (5, A, P, 7), where and A represent the state and action spaces, P denotes the transition function, is the reward function, and 7 [0, 1] is the discount factor. The objective of RI. is to learn policy (a[s) that maximizes the expected cumulative return. In offline RL, access to the online environment is restricted. Instead, the objective is to
optimize the RI. objective using a previously collected dataset, D { {st of, rg, +f+1)) 1- which consists of N transitions in total.

---

Corruption-robust Offline RL. We adopt a unified trajectory-based storage approach, as proposed prior works [34]. An original trajectory is represented as 87 -,- 1",-,,4-1;;" 11 PT))
where each trajectory consists of three components: states, ac- tions, and rewards. This trajectory can be reorganized into sequence data for DT [6] and RDT [34), or split into transitions (st. ag. rt, S1+1) T-2
(80: 0o, To, ........
T-2 for Markov Decision Process (MDP) based methods
such CQL [19] and IQL [18].
We investigate the impact of injecting random or adversarial noise into the dataset under two cor- ruption scenarios. First, we examine state corruption, where only the states in the trajectories are affected. Second, we introduce noise into state-action-reward triplets, which we refer to as "full- element" in the following context.
Random corruption refers the addition of noise drawn from a uniform distribution. For ex- ample, corrupting the state with uniform noise of corruption scale can be written as šu = So + --t(ss),, A Uniformi-an represents the d. d.-dimensional standard deviation of all states in the offline dataset. On the other hand, adversarial corruption employs Projected Gradient Descent (PGD) attack [25] with pre- trained value functions. We build upon prior work [34] by introducing learnable noise to the target elements and optimizing it through gradient descent to minimize the pretrained value functions. Further details refer to Appendix C.I.
where d, is the dimensionality of the state, and std(s)
Diffusion Models. Given any clean sample æ, the the forward process of diffusion models a Markov chain that gradually adds Gaussian noise to data according to a variance schedule , O(g
K
((x K
x."
II g(x R=1
ax 1)
g(a
1) = N(a"; vaua ,(1 0z T))
(1)
The reverse process is likewise a Markov chain characterized by learned Gaussian transitions (pa- rameterized by 0), typically initiated at p(x*) Wo<=:00 I):
po (e(x0:K) = p(a*) II (o x*), po (a x²) == N(xx uo(a" k), Eo(x", k)).
To ensure high-quality generation, the diffusion model learning process typically demands clean original data, i.e., have direct access to æ. Building on the foundation of initial Denoising Diffusion Probabilistic Models (DDPM) introduced by Ho et al. [14), the most widely adopted training loss for diffusion models is formulated as:
LDDPM (0) = ee
k)) c||3)
where * Oaga + vI 036 with oi == I og and e is randomly sampled Gaussian noise. Nevertheless, obtaining completely clean data poses significant challenge in certain circumstances, making the loss in Eq. (3) not directly applicable. Recently, several studies have proposed methods for handling corrupted datasets, while they assume uniform noise scale across the dataset [1, 8), leaving the problem of training diffusion models on partially corrupted datasets unresolved.
4 Methodologies
4.1 Motivation
A natural approach to addressing the partially corrupted dataset is to identify the corrupted samples within the dataset and leverage the uncorrupted samples to train a denoiser, which can subsequently be used to correct the corrupted samples. However, naive noise detectors trained with supervised methods relying solely on manually labeled clean and noisy samples may easily fail due to the inherent ambiguity of noise.
As an alternative, diffusion models have garnered significant attention due to their demonstrated effectiveness in training on fully corrupted datasets and leveraging the trained models to extract uncorrupted samples [1, 8). Moreover, diffusion models have also been shown to be effective as state denoisers in the realm of offline RL [38].

---

Nevertheless, key challenge is that existing diffusion models for corrupted samples typically as- sume a consistent noise distribution across the dataset, which does not hold for corruption- robust offline RI. On the other hand, if the naive diffusion process is applied to corrupted samples during training, even small amounts of noise in the training set can cause the diffusion model overfit to these perturbed points, leading to poor performance during subsequent sampling. To address the discrepancy between clean and corrupted data within the same dataset, the development of a new diffusion training method is urgently needed.
4.2 Diffusion for Partially Corrupt data
Extending Ambient Difffusion Models to DDPM. Daras et al. [8] demonstrate that diffusson models can be trained on datasets corrupted by a consistent scale of noise in the context of score- based continuous diffusion models. We observe that their conclusions can be seamlessly extended to the discrete DDPM framework. Building upon Theorem A.5 in [8], which addresses score- based variance preserving diffusion, we derive the following corollary that extends the result to the discrete DDPM.
Corollary 4.1. (Ambient DDPM) Let k be munually defined constant representing noise-added diffusion timestep. Suppose we are given samples x= = v Vas..2 x +vI - ãr,, e. Let x = vāaæ" + VI- aaE be further en-noised samples with S Ka < k. Then, the unique minimizer of the objective
āka 2tth Vox Ock
(ākk āā āx)co( =x""'²" 112)
Eggta K3)
(4)
V Vāk. ãā σxk)
have 6g- (or k) B[e]x=*), Ak Ka tef. Appendix A.I for detailed proof).
Notably, Eq. (4) cannot be directly applied in our settings because the noise-consistent data, x=,,, are not directly accessible. As discussed in Section 5.1, some portions of our dataset are corrupted while others remain clean.
Ambient DDPM from Approximated Distribution. It is worth noting that all samples involved in the Ambient DDPM training process follow Gaussian distributions with parameters that are func- tions of k. While direct access to noise-consistent data is not available, an ideal alternative is to ap- proximate these distributions and train the diffusion model using the approximations, Let g( |x²) denote the ground- truth distribution of samples generated by the DDPM forward process at diffusion timestep k, as described in Eq. (I), and let e( (x") represent an approximation of this distribution. To ensure effective learning under this approximation, we make the following.
Assumption 4.2. There exists positive constant e such that, for any ls 2 kas if the Kullback-Leibhe (KL) divergence satisfies Drr, (xrlg((")) |a )ll((*))))))))) < e, then the umbient DDPM with 2 kin as introduced in Corollary 4.1, can be effectively learned from samples drawn from the approximated distribution e( (a²))
Following the standard setup of corruption-robus offline RI, we are provided with samples that may may not contain scaled Gaussian noise The distribution of such samples at diffusion timestep & in the DDPM forward process is denoted as q(*1 We have the following.
Theorem 4.3. Let Assumption 4.2 hold. For any bounded noise scale one can always find diffusion timestep li such that ambient DDPM with k. 2 kig which should have been learned from samples drawn from q(<1|x0) can instead be effectively learned from samples drawn from q(a* (..). That is, for any >> O, one can always find ka such that for any k. " kis the following inequality holds: Dxilg(=>|a' ( <.
We provide the detailed proof in Appendix A.2. It is evident that smaller ka retains more of the original information, thereby improving the accuracy the noise predictor. However, a smaller hi also introduces a more relaxed threshold e, which may result in a larger discrepancy between the original distribution and the approximated distribution. The choice of k2 necessitates trade-off between these two factors. See Section 5.3 for detailed ablation studies.
5

---

4.3 Corrupted Samples Detection
Given a sample & that may or may not contain scaled Gaussian noise a e, we propose using the squared Frobenius norm of the noise predictor, ||Eo 11 determine whether noise is present.
Proposition 4.4. Assume the noise prediction error for co follows 63 NN(00 (T). Define the difference between the noisy and noise-free cases as: A Eillca mns 1)]11=1 Calloo 1)))) where a and th denote noisy and noise-free samples with original information consistency oper- ation, respectively. The Signal-to- to-oiss Ratio (SNR) of the prediction is then expressed as:
2 0
SNR(K)
(5)
((l 1)
(1 - a o
See Appendixa A.3 for detailed proof. Assume that the noise prediction error are the same across all diffusion timesteps, i.e., 0% = a for any k then (NR(k) achieves maximum value at k ka. as ag is a strictly monotonically decreasing function of k.
4.4 Ambient Diflusion-Guided Dataset Recovery (ADG)
Having established the theoretical foundation, we now proceed to introduce our proposed ADG method for corruption-rous! offline RL. As there are two timesteps involved, we use superscripts k to denote the diffusion timesteps and subscripts & to denote the RL timesteps for clarity.
The ground truth trajectory matrix is defined T: == [z (z.- Zt+ H1] E (2M+22 where z E RR represents the RI. component (which can correspond to an observation 8 or a state-action- reward triplet (s, a, r), M denotes the dimensionality of z, and H specifies the temporal slice size. Let &, denote the RI. component that may or may not contain scaled noise L. We only have access to the observed (partially corrupted) trajectory tे- = [ z+##l E Mx(H++1)
Ambient DDPM for Corrupted Samples Detection. Following Theorem 4.3, given a partially corrputed offline RL datastet, we pre-define ka and train the ambient DDPM through
E Byte D. K),t)
[I] Vax Ctk.
Otke
(aLk. - (+. k) t
ko 12
(6)
Vāk." āx (1 (1-aa) kk)
where 7/*h V ana
t/i Beg ai
Once the training converges, obtain the noise predictor
Gba
cs(Te, k). which achieves the largest SNR k ka described in Proposition 4.4. It should be noted that for each sample à, we focus solely on whether the RL component at the center position (ž,) is corrupted, rather than evaluating the entire trajectory slice. For this purpose, we further define (o(,)) = ||ea( +r; hig 12 I2 where (+) represents the (H+1)-th column of the matrix. Subsequently, we utilize eo (#,) to evaluate every samples within the partially corrupted dataset, and rescale the prediction range to [0, 1j. With a manually defined threshold Ç. samples with ep ee())) > are classified as noised samples (D,),, while the remaining samples are considered clean (D.).
Naive DDPM for Corrupted Samples Recovery. Once the corrupted samples have been identi- fied, the remaining uncorrupted samples can be utilized to train a denoiser (through naive DDPM), which can subsequently be applied correct the corrupted samples.
To avoid overfitting of DDPM to misclassified noisy data, we reuse ( to filter out training data for the naive DDPM. We denote Il, as a binary indicator variable that specifies whether ž, is corrupted (I = for (z)) > () or not (I, 1 for co( ( < (). Given the mask defined as m, :=
I- H I+ the training loss for naive DDPM follows
Br.-D....r.1.K,ee......... (*) k) e]] @ mm
(7)
where c is the Hadamard product. We refer to this training process as selective training in the following discussion. After the training coverges, we then conduct revese DDPM process pa( as described Eq. (2) to all samples within D,,. Finally, we combine the denoised D,, with D, to form the final dataset. More implementation details of ADG can be found in Appendices B and C.

---

5 Experiments
In this section, we conduct comprehensive experiments empirically evaluate ADG by exploring three key questions: (1) How does ADG enhance the performance of both non-robust and robust offline RL methods across various data corruption scenarios?
(2) What is the individual contribu- tion of ambient loss and selective training to the overall effectiveness of ADG? (3) What are the advantages of ADG's structure, which incorporates two independent diffusion models?
Table 1: Performance under random data corruption. Results are averaged over four random seeds.
Blemen
Altack
Task
Naive
CDr.
ADG
Naive
IQI.
ADG
Naive
RIQI.
ADG
Naive
DT
ADG
Naive
RDYT
ADG
balfeheetah
15.9218
23.915.6
19.252.2
2.7811.7
19912.1
278144
27.5.225
2..8
308118
34,211,4
hepper
45,496.4
788++11.1
47.6-7.1
72.2+7.1
34.0+134
66.3=15.9
51.3+14.0
79.1-6.7
566629
65.2+7.5
39.9552
46.0+9.7
17.566.8
27.933.7
14.24122
39.56433
47.6+49
55.1=8.9
534440
66.4#29
10llehocnsh(00))
11.0+111
17.3+27
6,1+13
12,0+0.9
4,4+0.9
83++22
63+004
26,4-10
8.3-155
22,4+088
(opper(108)
18+017
3.1+06
1333+3.3
1966663
15.5+54
157+6.1
3611776
37.4-94
408+335
42,4+79
State
walker2dt (0%)
4010.
1.111.1
10.927.2
18.712.8
9.31444
10.1143
18.0:2.5
34,0.5.5
203328
20.5:27
Richetsosmplete
38+28
1500+688
33.6-73
51.3+4.5
87.5+664
52,5+43
57,0H662
61.9-124
528+11
SL1+10.1
kinchen paraal
00100
16.11
13.51344
238.122
25.9134
26.91766
$1.018.11
1.
368158
19.1111
kinchen miseee--iied
00100
0.01000
16.255.6
41.2:4411
21663377
41.9132
31,8134
16.3:13.1
418:44
57.1176
doort
46.6+175
63.3+9.8
39.0=164
77.5+102
0460+4
1029-04
1028+22
1089-22
hamment (%%
02100
02100
64.11173
78221733
70.01126
884155
978-10-3
1187107
1138116
1268126.9
relocstor 1%)
43400.1
03
9,4-3.
14.424.9
52450
278:44
6166-56
67.6-4.0
650062
67.2488
Average Soore
24.7
51.
Improvement
62.5%
balfcheetah
03201
33
15.5115
37.4
22.52.3
32.3336
280:550
2.22-0.9
207229
M.1112
hopper 2d
0.4000
20.317.3
2633119.
36.1-20.1
164132
194
5301114.
55.11119
184
51.212011
Full Felemeeennt
41+00
22.2+42
21.1+8.2
175436
199459
51.+1145
55.3-19
56.5-110
161+154
09202
20.314.
42
18.3220
2.0.8.5
14.6424
1..1:4.5
2.63-16
163253
25.2421
hoppen 10%)
20111
109112.1
9.1103
19.1210.
116424
1422148
28.3188
15.9.6.0
36.1147
18.41766
walker 24d (0%)
09+16
1,2+20
5.++2
104+2.1
4.7
1977+82
189+40
$2.0-166
24.5559
43.4+9.3
kitchen complete
4.4332
13.145.4
23.86.01
48.3238
26.15.
488145
51.31994
55.0...3
6066-5.77
00.014.7
kinchen-pariall
19+221
0.0+00
A1++1
27.5+6.8
0.5+0.9
156++77
34.4+744
37.5-7.3
45.6-2177
418+94
kithen-missdd
06600
1.2422
088+08
37.5+50
84+11
43.1+662
21.9+11.3
23.8-22.4
39.4-17.1
35.6++977
doont
-03+000
-03-00
47.1-1
16.3-130
64 14.++1111
71.++08
33.0+34.3
1-1--66.
92.1-17.5
105
hammer)1))
02+00
0.2+0.0
65.7+12.7
84.5-12.2
05.0426.3
26.3+16.9
43.5-38.4
118.8-238
102.++17.3
relocated 1%)
0.3..00
03100
7,211.0
10.61.77
12.5.15
20.3
35.9118.77
42.0.121.1
56.71132
61.8.1344
Avrrage Sonre
22.9
55.7
Improvement
1033,33
61.57%
25.38
5.1 Experimental Setups
We assess ADG on various widely used offline RL benchmarks [9], including MuloCo, Kitchen, and Adroit. Since prior work [34] has shown that the impact of data corruption becomes more severe when data is limited, we further evaluate the effectiveness of ADG across different dataset scales by conducting experiments on down-sampled tasks. Specifically, we down-sample 10% and 1% of the data from MuJoCo and Adroit tasks as the new testbed. We also include results under different down- sample ratio of 2% and 5% in Appendix D.7. For MuJoCo. we select "medium-rege -eruuuuyrrer"""r- datasets. Additional results on "medium" and "expert" datasets are deferred to Appendix D.4.
For data corruption, consider two data corruption scenarios: random and adversarial corruption, as introduced in Section 3. These scenarios are applied either to states alone or to state-action-rewaard triplets (full-element). Following the settings of prior works [37], we the corruption rate 17 0.3 and the corruption scale a to 1.0. We provide the further implementation details on data corruption in Appendix C. We also investigate ADG under Gaussian noise corruption and under different noise ratio and scales in Appendix D.3, D.8.
We evaluate diverse set of offline RL methods, including pessimistic value estimation method CQL [19], policy constraint methods along with their robust variants IQL [18] and RIQL [37], as well as sequence modeling methods such as DT [6] and RDT [34]. To ensure the robustness of the findings, each experiment is conducted using four distinct random seeds, with the standard deviation across these seeds also reported.
5.2 Evaluation under Different Data Corruption
Results under Random Corruption. We evaluate the improvement brought by ADG on various offline RL algorithms under random data corruption. The average scores presented in Table 1 show that ADG provides benefits to both non-robust and robust algorithms across all scenarios. Notably, ADG brings significant improvements to bDs-dased algorithms (CQL, IQL, and RIQL), with an
7

---

average performance boost of 69.1%, demonstrating the effectiveness of sequence completion for missing information. ADG also provides substantial improvements to non-MDP algorithms (DT and RDT), with an average performance boost of 17.4%. When equipped with ADG, both IQL. and DT outperform their Naive robust variants in nearly all scenarios. This highlights ADG's core strength: by modifying only the dataset, it enables standard algorithms to surpass their robust variants explic- itly designed for noise resistance. Moreover, this advantage can be further amplified using robust variants, achieving nearly 60% improvement RIQL. Additionally, when equipped with ADG, most algorithms achieve similar performance under both state and full-element corruption, further demonstrating the scalability of ADG. We provide visualizations of ADG's s detection and denoising in Appendices D.I and D.2 to clearly explain its effectiveness, and investigate the effect of using ADG solely for filtering in Appendix D.5.
Table 2: Performance under adversarial data corruption. Results averaged over four random seeds.
CQL.
RIQL
Attack
Naive
ADG
Naive
ADG
Naive
ADG
Naive
ADG
Naive
ADG
balfchectah
$1 11.2#114
35.6m213
23.1=6.7
29.4#222
26.545.3
27.4113
35.6#2.1
34.5=1.9
35.221.3
hopper
55.3:113
5
51.0-6.9
51.24-2677
41.2422.7
11.1118.1
27.3+10.1
55.8+21.
55.2-78.3
70:57788
walker2d
44.21..66
1.
36.21140
42.417.4
16.723.2
32.4115.4
44.118.11
62.213.2
42.416.4
62.21..77
hulfeheetsh(10%)
10002.0
18
56217
13.0144
3.6105
5.400
26.412.1
7.5-0.4
22.3118
hopper(10%%)
202111
2,4228
200-388
199122
19.1259
226-11.3
38.644.7
40.315.9
39.325.1
42.216.99
State
walker20/155
02-0.11
0.2+0.5
9.8-3.3
140+44.
10.8444
104+4.
22.3424
48.4-1344
21.1-2.6
41.5+4.88
kicchen-compltt
38+2.8
75+73
45.6-211
53,8+577
51,0+333
51.34-.8
48.416.7
7388+38
58.4-37
600+3.11
kitchen purtial
00100.0
25514
26.9-16
33.11135
35.41588
44.418.7
32.615.11
39.41219
36.5
42.51100.5
kitchen-inixal
00+0.0
1.9432
41.2-4.5
42.54556
33.9+11.2
38.8+4.5
26.2+9.
38.8+996
30.0=5.5
45 65.6 10.1
-03+00
49.3+114
18.1+10.
47.3+248
6166-112
990+00
103.++20
104.
105
homeient
02400
784+89
60.14234
14.1-11.7
960+2.
12013-43
1166+74
13.4-13.1
relocated 1%)
02.00
0.2.0.11
7.011
18.7188.7
17.1
20.519.0
762150
80.8.559
69.044
709188
Average
31.1
46.2
57.6
Improvement
14.75%
17.70%
30.74%
12.28%
halleheetah
02:007
2..3-30
1.
17.015
34.514.1
370120
39,5:105
1.
110.3
hopper
12.32133
087777
27.81125
16.41113
66.9118.5
64.711055
walker
17.6+130
52741.8
49.3+65
574-14.7
52.4+6.7
60.1+74
588-660
65.2+6.0
Full-Eclement
1allechertsk(10%)
13.1=19
1711+34
481221
1734377
12.7424
14:2-15
hepperc (00%
10+0.4
3.8+2.
20.5+2.7
30.3+53
35.142.4
400+12.1
40.2+33.
30.7-18
40.4-13.2
walker2d/10%)
0.7.1.1
285174
37.4111.3
14,017.3
38.0110.7
kitchen complete
38128
7.5.14.7
46.2-6.2
48.116..5
50,01198
56.299.4
55.0.6.1
65.618.00
59426.2
58.11338
kitchen-ponial
4.4476
29.42.7
53.1+6.5
19.4+2.1
3811-128
20+1770
42.5+10.2
22.5-21.4
10.61120
20.6.148
38.112.11
36.1
doort 155)
0.3.0.0
0.410.0
66.9115.0
68.1111.1
37.71332
55.315.7
96.51113
98.9133
87.9-20.5
94.1170
0.1+0.11
0.24300
61.5=10.1
926+1135
50.9+3005
N0.7=13.2
75.1+201
75.2=25.5
62.5-26.5
45.35.
relocated 1%)
-0.2:0.0
-0.3::0.
5.0:3.4
7.93333
74=4.1
54.5:1433
67.7453
4.7=49
6.866.3
Average Store
0.8
9.5
33.6
42.0
27.5
39.4
47.8
54.1
41.7
49,3
Imprivement
1087.50%
25.00%
43.27%
13.18%
18.23%
Results under Adversarial Corruption. We further examine the robustness of ADG under adver- sarial data corruption. The results, summarized in Table 2, show that ADG consistently improves baseline performance by an average of 24.41 %. Notably, when equipped with ADG, the baselines IQL and DT outperform their Naive robust variants RIQL and RDT across all scenarios. This fur- ther supports the conclusions drawn from the random corruption scenarios. These findings highlight ADG's ' ability adapt and mitigate adversarial data corruption.
Kicchen-complet
Relocate
Roe Scuure
0.2
40
20
0.0
Kihe (Q1)
Kihr (OT)
Re (3%, /QLI
Re (1% DT)
Nalve
Ours 155)
Durs 30)
Our 501
Nalve
Ours 15g
Ours Pu
ours 0u-500
The FN rate during detector training (left), and the performance of IQL and DT using these detectors (right). "Kit-c" denotes Sitchen-comples and "Re" denotes Relocate.
Figure 3:
5.3 Ablation Study
We conduct ablation studies analyze the impact of each component ADG's performance. Impact of Ambient Loss. We assess the impact of ambient loss on detector performance by vary- ing t, {15,30, 50} on the "kitchen-complete-vO"" and "relocate-expert-vi" datasets, selected for
8

---

their complexity. Following Section 5.1, Random State Attacks are introduced. Samples detected as corrupted are labeled positive and others as negative. The detector trained for 5k steps. We detect the dataset with (0(²) V (as in Section 4.4) at each step and plot false negatives (FNs), which represent the proportion of undetected corrupted samples. We also evaluate the D4RL. scores of baseline algorithms the datasets recovered using ADG with these trained detectors. As shown in Figure 3, the detector trained with the naive diffusion loss shows some detection capability ini- tially, but quickly overfits to the corrupted portion of the training data. Ambient loss significantly improves it. Notably, in 30, which is also used in the main experiments, achieves the lowest false negatives (FNs), implying that nearly all selected samples remain unaltered by attacks. This ensures that the denoiser trained on nearly clean data, making the naive diffusion loss feasible. Moreover, the D4RL scores of baseline algorithms exhibit clear correlation with detection performance.
Impact of Selective Training. To evaluate the impact of selective training on denoiser, we vary Ç within the range {0.05, 0.10, 0.20, 0.50) and measure the mean squared error (MSE) between the recovered dataset and the ground truth. The results are shown in Figure 4. A very low Ç = 0.05 leads to poor denoiser per- formance and eventual overfitting, likely due to small dataset. the insufficient information 0.50 also de- Therefore, very high grades performance, possibly due to excessive corrupted data in the dataset. Moderate values of Ç = 0.10 or 0.20 yield similar, good per- formance, supporting the necessity of selective training and suggesting that is somewhat ro- bust, performing well within a certain range.
C- 0.05
{- 7=0.10
[=0,20
7- 0.50
Nomaaizzed Score
0.6
30
WISE
0.5
20
0.4
0.3
0k
10k
15k
20k
Update be Steps
(a)
(b)
Figure 4: Results on munkkerddmmddium-rpplhee
v2" with Random State Attack (Appendix C.1): (a) MSE vs. ground truth, and (b) DT perfor- mance across Ç. Best results at 0.20.
The D4RL scores of the baseline algorithms ex- hibit a strong correlation with the performance of the denoiser, further supporting the necessity of using Ç for the selective training of the denoiser.
Impact of Using Two Separate Diffusion Mod- els for Detection and Denoising. As described in Section 4.4, approach employs structure with two independent diffusion models: one serv- ing as the detector and the other as the denoiser. However, it is technically possible to employ a single diffusion model for both detection and de-
Normalized Score
30
10
noising tasks by training it concurrently with both
Walker2d (IQL)
Walker2d (10%, (QL)
Eq. 4 and Eq. 7. We conduct ablation experiments on the der Random State Attack, using both full and limited dataset sizes to evaluate this configura- tion. The results are presented in Figure Al- though ADG with single diffusion consistently improves the performance of baselines, ADG with two independent diffusion models always outperforms ADG with a single model. This re- sult is intuitive, applying two different losses to single diffusion model can casily cause inter- ference, leading the network converge to local minima. Using two independent models effec- tively addresses this issue by decoupling the mu- tual interference. For more implementation de- tails and results, refer Appendix D.10.
dataset un-
Mlormalized Score
40
Walker2d (DT)
Walker2d (10%, DT)
Naive
ADG (w/ SD)
ADG
Comparison results among Naive, ADG with single diffusion model, and ADG with separate diffusion models for IQL (upper) and DT (lower). ADG (w/ SD) denotes ADG using single diffusion model.
Figure 5:
Additionally, we also include the ablation study on the length of the slice window by varying the values of H in Appendix D.6. The performance of ADG shows a positive correlation with the hyperparameter as increases from O, and becomes

---

robust to further changes once H reaches a certain range. This demonstrates the importance of incorporating sequential information and the robustness of ADG.
Conclusion
We propose Ambient Diffusion-Guideer Dataset Recovery (ADG), the first diffusion-baann based denoising framework for offinn RL under data corruption during the training process. We introduce Ambient Denoising Diffusion Probabilistic Models (DDPM), which enable the diffusion model to distinguish between corrupted and clean samples. This mechanism effectively filters the training data, allowing training a naive diffusion model to serve as a denoiser. Comprehensive empirical studies on D4RL benchmarks demonstrate that ADG consistently improves the performance of existing offiine RL algorithms across various types, scales, and ratios of data corruption, and in most cases, allows baseline algorithms to outperform their robust variants. We hope that this work establishes new paradigm for more robust learning from noisy or corrupted data, ultimately benefiting the application of offine RL in real-world scenarios.
References
[1] Asad Aali, Giannis Daras, Brett Levac, Sidharth Kumar, Alexandros G Dimakis, and Jonathan I Tamir. Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data. arXiv preprint arXiv:2403.08778, 2024.
[2] Johannes Ackermann, Takayuki Osa, and Masashi Sugiyama. Offline reinforcement learning from datasets with structured non-stationarity. arXiv preprint 2rXiv:2405.1411 2024.
[3] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023.
[4] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Advances in neural information process- ing systems, 34:7436-7447, 2021.
[5] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learn- ing. arXiv preprint arXiv:2202.11566, 2022.
[6] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 50845004-50097, 2021.
[7] Yiding Chen, Xuezhou Zhang, Qiaomin Xie, and Xiaojin Zhu. Exact policy recovery in offline with both heavy-tailed rewards and data corruption. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1411-11124, 2024.
[8] Giannis Daras, Alex Dimakis, and Constantinos Costis Daskalakis. Consistent diffusion meets tweedie: Training exact ambient diffusion models with noisy data. In Forty-first International Conference on Machine Learning, 2024.
[9] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[10] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learn- ing. Advances in neural information processing systems, 34:20132-20145 2021.
[11] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning with- out exploration. In International conference on machine learning, pages 2052-2062, PMLR, 2019.
10

---

[12] Kamyar Ghasemipour, Shixiang Shane Gu, and Ofrr Nachum. Why so pessimistic? estimating uncertainties for offline through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 35: 35:18267-1828, 2022.
[13] Philippe Hansen-Estruch, llya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Ad- vances in Neural Information Processing Systems, 33:6840-6851, 2020.
[15] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902- 9915. PMLR, 2022.
[16] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273- 1286, 2021.
[17] Cevahir Koprulu, Franck Djeumou, et al. Neural stochastic differential equations for uncertainty-aware offline In The Thirteenth International Conference on Learning Rep- resentations.
[18] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learnig. arXiv preprint arXiv:2110.061163 2021.
[19] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179 1191, 2020.
[20] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
[21] Angi Li, Dipendra Misra, Andrey Kolobov, and Ching-An Cheng. Survival instinct in offline reinforcement learning. Advances in neural information processing systems, 36, 2024.
[22] Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for 1lms. arXiv preprint arXiv:2404.04102. 2024.
[23] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adapt- diffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, pages 21725-20745. PMLR, 2023.
[24] Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guider policy regularization for offline reinforcement learning. arXiv preprint wwX1922405.19909 2024.
[25] Aleksander Madry. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1eXivv1706.0600 2017.
[26] Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust re- inforcement learning using offline data. Advances in neural information processing systems, 35:32211-32224, 2022.
[27] Robin Rombach, Andreas Blatmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[28] Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25(200):1-91, 2024.

---

[29] Ruizhe Shi, Yuyao Liu, Yanjie Ze, Simon S Du, and Huazhe Xu. Unleashing the power of pre- trained language models for offline reinforcement learning. arXiv preprint arXiv:2310.20587, 2023.
[30] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pages 2256-2265, PMLR, 2015.
[31] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[32] Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusson policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference Learning Representations, 2022.
[33] Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. Copa: Certifying robust policies for offline reinforcement learning against poisoning attacks. arXiv preprint (xXxx:2033008998 2022.
[34] Jiawei Xu, Rui Yang, Shuang Qiu, Feng Luo, Meng Fang, Baoxiang Wang, and Lei Han. Tackling data corruption in offline reinforcement learning via sequence modeling. In The Thirteenth International Conference on Learning Representations, 2025.
[35] Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing, Advances in neural infor- mation processing systems, 35:23851-23866, 2022.
[36] Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang, Regularizing hidden states enables learning generalizable reward model for 1lms. arXiv preprint arXiv:2-06.102266, 2024.
[37] Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption. In The Twelfth International Conference on Learning Representations, 2024.
[38] Zhihe Yang and Yunjian Xu. Dmbp: Diffusion model-based predictor for robust offline rein- forcement learning against state observation perturbations. In The Twelfth International Con- ference on Learning Representaatonss, 2024.
[39] Chenlu Ye, Jiafan Hc, Quanquan Gu and Tong Zhang. Towards robust model-based reinforce- ment learning against adversarial corruption. arXiv preprint arXiv:2402.0888 08991, 2024.
[40] Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robus offline reinforcement learning with general function approximation. Advances in Neural Information Processing Systems, 36, 2024.
[41] Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinffrreeet learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452 2021.
[42] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observa- tions. Advances in Neural Information Processing Systems, 2-21002--2137. 2020.
[43] Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline rein- forcement learning. In International Conference on Artificial Intelligence and Statistics, pages 5757-5773. PMLR, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: There are three main claims laid out in the abstract. These concern ADG's ability recover near-original trajectories under heavy comruption, the tradeoff between corruption detection precision and policy learning stability, and improved performance from boundary-biased diffusion sampling. All of these points addressed in the main text through theoretical analysis and empirical validation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations our work in Appendix E.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The proof of most statements are layed out in the supplemental, to save on space.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple-
mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided or not)?
Answer: [Yes]
Justification:
We explained our settings in Section and heprparrameters in Appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We upload the codes and instructions to recover the results. Once the blind review period is finished, we'll 'll open- open-source all codes, instructions, and model checkpoints.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niis.... cc/ cccppblic//aacccccccccccccccacca cabbic/guudes httgguee/cooeduubmmi ssi cotmmisssoooooio onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Training and test details (including hyperparameters, optimizer settings, and data splits) are specified in Section 5 and Appendix C.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We show the standard error most training curves, with an average of over four random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have a Appendix C.5 on this.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttt::://eurippte httppppbllc/thcccccieeeeeeee
Answer: [Yes]
Justification: We do not believe that our work has any harmful consequences as layed out in the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work involves small models and simulated datasets. It does not impact society at large, beyond improving our understanding of certain aspects of deep learning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such models or datasets are involved.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, haperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: We have no human participants in our study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy pettps:/neurips: ct/////feerencce//2////////// for what should or should not be described.
19

---

A Theoretical Interpretations
A.1 Proof for Corollary 4.1
dgEg, where E1, E2 ~ Firstly, we define z² = vākk x + vI- ak, E1 and x = Vāza + vI
(0 I). According to the forward process in DDPM, x* can be expressed as a function of x**:
x² Vāgz" + vI - OgC2
Vāx vh VI- (kk E1
+ v1 24.2
Vākko
vāk x +
2
vāxv I Okki
(8)
Otka
Var.
v Vaka
Ok
ku
aha
Ork
ōka
Ōk.
Following the proof sketch of Lemma A.4 in [8], we apply Tweedie's formula to the pair of x and
z
V logpp<<")) = Vā |x*] x
(9)
āk Similarly, applying Tweedie's formula to the pair of xh and tho we derive:
de TE
(ar Ni |a=*|]
x
log (xrt
(10)
Okk
Equating Eq. (9) and Eq. (10), we ohtain:
El xx1
Otk.
0e xx
13 20
Vak, (1 C.a [x]]
xx
(1I)
Vāa.
Vā -(1 - C.
Since we aim to predict the Gaussian noise rather than the original signal, the forward process of DDPM provides the relationship:
E xh vI (x
(12)
ax Substituting Eq. (12) into Eq. (11), we relate R[e[a*] to ta |zx"
Elaeii [x*] ōkk w Vas OCk
(āx. - āx 18 [[([(=*]
(13)
Vāka O&k (1 (ikc
Building on Theorem A.3 in [8], we minimize the objective:
Eyzaa Ex-it(k.. Eott [llga(a**, k:) .z
(14)
where the minimizer satisfies go (x* k) = E[ [ Substituting Eq. (13) into Eq. (14), we have the final form of the optimization objective:
(āx, ax ax) (x. ks)
E Ex Exoeliik. K) Eeg ko ||
x Ne
(15)
Vau oek.
Vāk, az (1 axk
with the minimizer eg- (a* k) = [E[E|=*], Vk 2 hia This completes the proof.
A.2 Proof for Theorem 4.3
Let è be the samples that may or may not contain scaled Gaussian noise (. e. It can be expressed in the form of:
क = 0 + Unaise e, where x E RR is the noise-free matrix, is the noise scale, E m is the matrix of Gaussian noise with i.i.d. entries éij (0, 1), and Tnoise is an unknown indicator variable that determines whether noise is present. Accordingly, we define the samples that contain consistent Gaussian noise t. e, which is expressed as:
(16)
ã == x LL.E.
(17)
20

---

Lemma A.1. Given è and 2 defined in Eqs. (16) and (17) respectively, let q(-b1-) denote the distribution of DDPM forward process diffusion timestep k, the following inequality always holds:
DKLlg(æ* (11_======))((11(1111))))))))))1 [ I Dxzlg((" (x))))()*)++)))
(18)
Proof. We first denote the probability density function for distirbution ((=*1x²)) ((*1)=)) and (() as Pr(-), Pr(-), and Pr(-) respectively. According to the definition of KL divergence, we have:
D DKLla lg(("* [²)] - DD
Pr(æ)
Pr(x)
Pr(x)) log
Pr(æ) log
Pr(æ))
Pr(xr)
(19)
Pr(x)
Pr(x:) log
Pr(æ))
Note that the probability density function for q(**11"" can be expressed as a mixture of two com-
ponents: Pr(æ) ?Pr(x) + (1 (1--)) Pr(xx), where E [0, 1] represents the probability that the indicator variableInnosee Substituting this relation into Eq. (19), we have:
Declg(a** (=)))1((")1)))))))))))))
Po(x)log >Pr(x) + (1 Pr((()))
Pr(æ))
/ Pr(x)) log PP((x) + - Pr(x))
Pr(x)
Pr(a:)
2 log
Pr(æ) PP((x) (1 Pr(x))
(20)
#
Pr(æ:)
2 log
Pr(x) Pr(x)" 7 Pr(x) (1-2)) (1-7)
Pr(x)" .Pr(x) (1-7)
= log
i 2 log yPr(x) + +(1 Pr(z)
log/r + (1 7)] =
where the inequality (2) holds due to Jensen's inequality, and inequality (f, (i) follows from weighted Arithnetic-Geometri mean inequality. They all hold with equality if 7 1, i.c, the indicator takes Inoise = 1 for all samples.
Lemma A.2. For any c > 0, with the bounded Gaussian noise scale i and the samples & from Eq. (17), one can always find ki such that for any k 2 kms the KL divergence between g( x²) and
((**)00) satisfying:
DKI[g(æ* (1(((")))))))
<
(21)
Proof. According to the definition of DDPM forward process in Eq. (8), have
x o
/ax + VI âBE
(22)
a k Vanaa + v1 04.1
(x +L. E) + vI 64
(23)
Varxa + / - ãs + ōye
Note that e E |R is the matrix of Gaussian noise with i.i.d. entries eij NO, 1). By flattening the matrix into vector, can explicitly express the probability density function of the distribution q(=*1x") as follows:
(r()) (2TT) mom / 11/22 exp ( (at
vana")
T
--* vana²)
(24)
21

---

where E E R is a diagonal matrix with all diagonal elements equal to ag. Similarly, the probability density function of the distribution q( (z") follows:
(2 )mm/ Σ 1/2 exp (- -
Pr(0"|<" (1")
Vara") - Vara²)
(25)
where E Rm is a diagonal matrix with all diagonal elements equal to ce + Lā.. Then we derive the KL divergence between q(x=)) æ²)) and q( )=)):
D ( xr
(x=)) )l (11 [+))))
Σ
log
+ 1r Σ EΣ} - mm
Σ
(26)
mn
0g
log
1 0g + 2.x
Let f(k)
1- 1-6, +. where āg is monotonically decreasing function of k with 0g [0,
1-ax
is straightforward deduce that f(k) is also a monotonically decreasing function of k, with f(k) E oc) if is bounded. Substituting this expression into Eq. (26), we obtain:
DKL[g(æ"1ar (1q(("))))))))))))))))))))))(( x²) (lla(( mn log (kk
1
-1].
(27)
s(k)
which is monotonically increasing function of fk) within the range of flk). It attains its minimum value 0 when f(k) = 1. Therefore, for any c O, we can always find kg such that for any k 2 kas the inequality in Eq. (21) holds.
By combining Lemma and Lemma A.2, and under the validity of Assumption 4.2, the proof of Theorem 4.3 is complete.
A.3 Proof for Proposition 4.4
We begin with the noise prediction at diffusion timestep k. If the noise is perfectly predicted, it should follow the form:
a x
Epred
(28)
ag
Notebly, any noised/un-noisel sample can be expressed as Ø = x + Inoise E, where x" e mxx the noise-free matrix, i is the noise scale, Rmxx the matrix of Gaussian noise with i.i.d. entries Eij ~ (0, 1), and Inoise is unknown indicator variable that determines whether noise present. To predict the noise within ø using diffusion timestep k. the first step is to perform the original information consistency operation:
hy
Ouk = Vork (a: + Unoise 6).
(29)
Then substituting (29) into (28), we obtain:
Vou
Inoise
Epred
(30)
V1 Okk
Assume have noise predictor co(:, k)) with prediction error 5% ~ ((0,of1) Given sample ē that may or may not contain noise, we have the following two criteria:
Case 1: Noise-free data (Inoise 0) Under such circumstance, the model's prediction is entirely determined by the prediction error:
co ( k) Cpred + 5 55.
(31)
thereby the expected Frobenius norm squared of the prediction is:
Elllea
k)]
5 5 |
E
_, Σ, (b.,,)
(32)
77 12 o
22

---

Case 2: Noisy data (Inoise 1) For noisy data, the model's prediction includes both the actual noise and the prediction error. Specifically, have:
Vāx L Inoise
co(0" k)) pred + 5
c 8
(33)
V1 Oz
The expected squared Frobenius norm of the prediction is then given by:
=Ell V Ct L Inoise
6
VI Oik
E[[] Vak L. Inois
2 L. Inoise E[Tr(E 88)]
+
v1 ak
v1 ax
(34)
B[|le|?.] + m. ok +0
(1 @x) (xx))
2 ak
-n m-
(1 (+)
The last term in equality (i) vanishes because and o are mutually independent. Combining the
results of Eq. (32) and Eq. (34), and substituting into the definition of the Signal-to-Noise Ratio
(SNR) from Eq. (5), we derive:
SNR(K) = [E[ Eo a
1111
60 D of: E
E Bilco ( 1>1
(1-aa)
3.ng
m + m.
of m n2- of
(35)
m. of
2. ..0
(1 kxk .0
which complete the proof.
B Algorithm Pseudocode
We provide the pseudocode of our proposed Ambient Diffusion- Dataset Recovery (ADG) Algorithm for a comprehensive overview.
C Implementation Details
C.1 Data Corruption Details during Training Phase
We study two types of corruption: random noise and adversarial noise. For each type, we in- vestigate two categories of elements to attack: (1) State corruption, where only the states in portion of the samples are corrupted. According to Section 3, an original trajectory is defined as (so, do, r.....,, 1r-,,,11 In MDP-based methods like IQL and CQL, corrupting state 8, affects two transitions: (st-1, (t----11 41-1, 11, 8t)) and (st, de, Tt, 8141) ). (2) Full-element corrup- tion, where the states, actions, and rewards (st, de, rz) in portion of the samples are corrupted, introducing a stronger challenge for robust offline RL algorithms.
We consider the tasks including MuJoCo, Kitchen and Adroit [9]. We select the "medium-replay v2" datasets in the MuJoCo tasks for our main experiments, using both the full datasets and down- sampled versions (reduced to 10% of the original size). For Adroit tasks, we choose "expert-vO" vO" datasets and sample them to 1% of their original dataset. We use the full datasets for the tasks in the Kitchen, as their original dataset size is already limited. To control the overall level of corruption within the datasets, we introduce two parameters 1) and a. The parameter represents the proportion of corrupted data within dataset, while indicates the scale of corruption across each individual dimension. These settings are consistent with prior works [40, 37, 34]. We outline two types of random data corruption follows:
23

---

Algorithm Ambient Sertusion-Guideee Dataset Recovery (ADG)
Require: Offlin partially corrupted dataset D, initialized noise predictors detector and the de-
noiser Ed-
Step 1: Update the detector using Ambient Loss
for each iteration do
Sample a trajectory mini-batch B = { +( ~ D, where ă represents either the state or the concatenation of (s, a, r) may or may not contain noise Sample uniformly distributed diffusion timestep l: ~ (Ka,..., K} Sample random Gaussian noise E ~ No
Produce noised element through 2 = vā + VV-
Get trajectory t; ( H E (=Mx(2H+1) Update the eg through Eq. 4. end for
Step 2: Update the denoiser G using Naive loss for each iteration do
Sample trajectory mini-batch B {(3 11, ................................ 2(+1)} D, where represents either the state or the concatenation of (8, a, r) may or may not contain noise
Sample uniformly distributed diffusion timestep Sample random Gaussian noise e NN I)
{K....,K}
Produce noised element through _ V + vI-vatt
Get trajectory t; [ ..... H+ E Mx
Get (2,) using detector (g k) with k = hi
Get mask m, ;= [1+ -H- ................................ [e where 1, for c0(2) > ( and 4e = 1 for e0( <C Update the Eo through Eq. 7. end for
Step 3: Detect and Recover the noised dataset for each in the noised dataset with eg (=)2) > { do
Get the trajectory tg := [ He......,. L
1 [+] v og (+², k)] Recover the trajectory H2 Voc
Replace ε, with (#+) which represents the (H+1)-th column of end for
Random State Attack: We randomly sample r). N.T states from all trajectories, where N refers to the number of trajectories and 7 represents the number of steps in a trajectory. The selected states are then modified d, represents the dimension of states, and std(s) (s) is the d dd-dimensional standard deviation of all states in the offline dataset. The noise is scaled based on the standard deviation of each dimension and is independently added to each respective dimension.
s + X- std(s), where À Uniform[ ajd., Here,
Random Full-element Attack:
We randomly sample 1) N T state-action-reward triplets (8t, Tt) from all trajectories, and modify the action à = a (td)sst((a).FF Uniform[-30 ar 30 ar where À ~ Uniform[-a, a)), da represents the dimension of actions and std(a) is the da-dimensional standard deviation of all actions in the offline dataset. The corruption to rewards is multiplied by 30, following the setting in RDT [34], since offline RL algorithms tend to be resilient small-scale random rewards corruption.
The two types of adversarial data corruption are detailed as follows:
Adversarial State Attack: We first pretrain IQI. agents with a Q-function Qp and policy function x, on clean datasets. Then, we randomly sample 17 N states and modify them as follows. Specifically, we perform the attack by solving for mingeB.. Qp a). Here, Ba(s, c)
IS c std(s)} regularizes the maximum difference for each state dimension. The optimization is implemented through Projected Gradient Descent, similar prior works [25, 42, 37, 34]. In this approach, we first initialize learnable vector u a),, a)d., and then conduct a 100-step gradient descent with a step size of 0.01 for = 8 + v std(s). After each update, we clip each dimension of within the range [aa,a]-
{
24

---

Adversarial Full-element Attack: We use the pretrained IQL agent with Q-function Qp and policy function Zp- Then, we randomly sample r N state-action-rewall triplets (st, (t), and modify = (s, a) to minaeBa(u..)) Qp Here, Ba(u, o)) {ūllūuu كا a s-st regularizes the maximum difference for each dimension of u. The optimization is implemented through Projected Gradient Descent, discussed above. The rewards in these triplets are also modified to: -
C.2 ADG Network Structure
Both detector and denoiser in ADG is classical diffusion models [31] with Unet Structure as shown in The diffusion models are unconditional, which do not need to introduce extra knowledge except the noised imput, make ADG can be simply applied generation, we utilize a Layer MLP with Mish activation. Further details including the hyper- parameters refer Section C.3.
scenarios. For the noised prediction
Residual Tereporal Block
Group Nomn Mish
Group Norn Mish
Revidaal
sample (Cousld)
Up Sp-Sampps Tamppould)
Figure 6: Neural network structure of ADG.
C.3 Hyperparameters
We present the hyperparameters and other details of ADG in Table 3 and 4.
Table 3: Generic hyperparameters ADG.
Value
Batch size
256
Total diffusion step (K)
100
Ambient Nature Timestep (Ka)
30
Threshold
0.20
Temporal slice size H
5
Learning Rate (Ir)
Noise prediction network
FC(256,256,266) with Mish activations
Dropout for predictor network
0.1
Variance schedule
Variance Preserving (VP)
Learning Rate
le-4
Table 4: Hyperparameters of ADG for different benchmark environments and datasets.
Tasks
Embedded Dimension
Denoising Diffusion Steps
Total Training Steps
MuJoCo MuJoCo (10%)
128
100
20k
128
100
20k
Kitchen
512
10
40k
Adroit (1%)
512
10
40k
C.4 Dataset Details
Since we conduct the experiments on both the full and down-sampled datasets, we provide detailed information about the number of transitions and trajectories shown in Table 5.
25

---

Dataset
halfcheetah 202000 202
hopper
walker2d
larger number of # Transitions
402000
302000
Trajectories
2041
1093
Dataset
halfcheetah(100%) 20000
Mopper(10%%
walker2d(10s))
# Transitions
32921
27937
# Trajectories
20
204
109
Dataset
kitchen-complete
kitchen-partial
kitchen-mixed
# Transitions
3680
136950
136950
# Trajectories
19
613
613
Dataset
door
hammer
relocate
# Transitions 10000
10000
10000
# Trajectories
50
50
50
Table 5: Detailed information about the number of transitions and trajectories.
C,5 Computation Overhead
In Table 6, we compare the computational cost of ADG with baseline algorithms on single GPU (P40). Each algorithm is trained on the "vallke22d-edduu-eelpy-222 dataset, and we record the total training time. CQL requires significantly more time due its reliance Q ensembles and other computationally intensive processes. IQL and DT exhibit similar computa- tional costs to their robust counterparts, RIQL and RDT. Notably, ADG achieves substantially lower computational costs than all baselines, demonstrating its ability to deliver significant performance gains with minimal additional overhead. The total time and per-step time for training the detector and denoiser, as well as for dataset recovery, are also reported.
Table 6: Training time of ADG and offline RL baseline algorithms.
Tasks
CQL
IQL
RIQL
DT
RDT
ADG
Epoch Num
1000
1000
1000
100
100
10
Total Time (h)
9.39
3,90
4.14
1.22
1.27
0.78
Detector Training (h)
0.37 (0.07s/step)
Denoiser Training (h)
0.22 (0.0 (0.04s/step)
Dataset Recovering (h)
0.19
D Additional Experimental Results
We present additional experimental results in this section. The network architecture and hyperpa- rameters remain the same those specified in Table 3. All results are averaged over four seeds.
D.1 Visualization c(#) Predicted by the Detector
To demonstrate the effectiveness of ADG's detection capability, visualize the squared Frobenius norm e(ă) of each sample è across the MuJoCo, Kitchen, and Adroit datasets using trained detec- tors. Consistent with the experiments in Section 5, we use "medium-repla" and "expert" datasets for Adroit. The results are shown in Figure 7 using box plots. From the results, observe clear difference between the clean samples and corrupted samples in the distribution of e(.).
datasets for MuJoCo
D.2 Visualization of the Recovered Trajectories
To demonstrate the effectiveness of our proposed approach, ADG, we visualize partial trajectory of "hopper" in Figure 8, From the results, observe that ADG can effectively detect and recover the
26

---

1.4
Clean
Squared Frobenius Norm
Corrupted
0.8
0.6
0.4
0.2
0.0
ha
ho wa
kit-c
kit-m
kit-p
do
hm
Figure 7: Visualization of the distribution of e(()) for samples in the datasets, including halfcheetah, hopper, walker2d, kitchen-complete, kitchen-mixed, kitchen-partial, door, hammer, and relocate, denoted as "ha", "ho", "wa", "kit-c", "kit-m", "kit-p", "do", "hm", and "re", respectively. We distinguish clean and corrupted samples using different colors.
corrupted samples. The partially noised trajectory has its discontinuity significantly reduced after recovery. Additionally, ADG introduces nearly no further corruption to the clean samples during the recovery process.
(a)
(b)
Figure 8: Visualization of the denoising effect of ADG on the In (a), we present the complete trajectories. while (b), we provide partial view of the trajectories along with their scatter plot representations to highlight the differences in trajectory coherence. In both subfigures, the first row depicts the clean trajectories from the offline dataset, the second row shows the trajectories perturbed with Random State Attack described in Section C.I, and the third row illustrates the trajectories restored using ADG, which have largely corrected the corrupted samples.
dataset.
D.3 Performance Under Gaussian Noise
As presented in Section 5.2, we evaluated the performance of ADG under both random and ad- versarial noise. We further analyze the robustness of ADG against Gaussian noise in the state.
27

---

The implementation simply changes the noise from Uniform I-a, ag in C.I to Gaussian noise (0, a²). We set 17 0.3 and a 1.0, as in the other experiments. The results are presented in Table 7, where ADG consistently improves the performance of the baselines, including IQL. and DT, further highlighting its efficiency.
Table Results under Gaussian noise.
IQL
DT
Attack
Task
Naive
ADG
Naive
ADG
halfcheetah
17.2+4.6
27.9
36.1
36.1
hopper
33.3+9.1
54.942.6
41.8
60.8
walker2d
18.0#5.6
40.3+16.0
50.9
58.2+10.1
State
halfcheetah(109)
13.6#3.5
15.6#3..7
12.8+2.1
24.8
hopper(10%)
25.4
32,6:t4.5
39.0=7.4
44.3+7.7
walker2d(10%)
15.3+5.3
6.723.1
27.1
33.7#4.0
Average
20.5
29.7
34.6
43.0
Improvement
44.88%
24.28%
halfcheetah
10.3+3.9
32.5+2.5
34.1+2.3
44.5+7..3
Full-Element
hopper
28.0+9.9
31.942.6
38.1
49.749.4
walker2d
8.942.7
38.7
59.3=7.2
60.2
halfcheetah(100))
5.941..6
17.8+22.8
11.8+:2.0
21.7++2.8
hopper(109)
15.6+3.8
13.4
25.5=7.9
37.8+12.2
valker2d(10%)
11.745.7
15.8:3.4
24.2=6.3
27.6
Average Improvement f
13.4
25.0
32.2
40.2
86.57%
24.84 %
D.4 Performance Under MuJoCo Dataset with Different Quality Levels
evaluate the robustness ADG under varying dataset quality levels. Specifically, we select the "medium-v2" and "expert-v2" datasets from MuJoCo tasks and downsample each to 2% of their original size, resulting datasets containing 20k samples. For comparison, we include RIQL, DT. and RDT as baselines. We do not include the results of CQL and IQL. they perform poorly on these down-sampled datasets. The results, summarized in Table 8, demonstrate that ADG consistently enhances the performance of these baselines across different dataset quality levels.
Table 8: Results on "medium-v2" and "expert-v2 22" datasets under Random State Attack.
RIQL
DT
RDT
Dataset
Task
Naive
ADG
Naive
ADG
Naive
ADG
halfcheetah(2%)
18.0+
24.842.1
15.7+1.3
31.111.4
1..3+
22.0-3.0
hopper(2%%
47.5+7.3
44.1+2.7
48.6+3.2
51.4++4.0
52.2+-5.7
54.1+4.9
medium
walker2d(2%)
25.4++5.0
26.4++4.0
20.5+6.00
50.0+4.7
28.0+8.0
46.8+3.6
Average
30.3
31.8
28.3
44.2
34.2
41.0
Improvement t
4.95%
56.18 %
19.88%
halfcheetah(2%)
0.5+1.2
1.2+-0.7
2.9+0.8
4.2+1.00
4.4+-02
2.02-006
hopper(2%)
32.0+-4.4
39.7+12.9
38.5+7.4
41.3+4..3
48.6+7.0
42.6+7.0
expert
walker2d(2%)
21.7+4.6
46.6+2.7
40.7+4.8
53.2+3.2
41.6+4.2
55.3++6.1
Average Improvement f
18.1
29.2 61.33%
27.4
32.9 20.07% %
31.5
33.3
5.71 %
D.5 Filtered Datasets vs. Recovered Datasets in MDP-based Algorithms
For MDP-based algorithms such as IQL and RIQI,, which not take sequences as input, filtering out corrupted samples from the dataset using detector, without recovery, is also viable method to
28

---

mitigate the impact of corrupted data. Specifically, we construct the filtered dataset by excluding all for which (z) > Ç 0.20 using ADG. To further investigate this approach, we also evaluate filtering with Ç = 0.10 to assess whether a stricter threshold improves performance. Although ADG demonstrates strong detection capabilities, some corrupted data may remain the filtered dataset. Therefore, we also evaluate the performance of the filtered dataset after removing the remaining corrupted data, which we refer to as the purified dataset. We then perform IQL and RIQL on these datasets. The results, shown in Figure 9, reveal the following insights: (1) The filtered dataset does not consistently improve the performance shows better robustness, to filter the dataset more aggressively does not improve performance, possibly due to the reduced information content in the dataset. These findings highlight that dataset size is crucial factor for RL performance, suggesting that filtering the dataset without recovery does not outperform recovery- based approaches. This emphasizes the importance of recovery in ADGG
the MDP-based algorithms. (2) The recovered dataset when compared the purified dataset. (3) Using lower value of
Normalized Score
40
20
20
Kitchen-complete
(1%)
Kitchen-complete
(ç- 0.10)
(?- (f-0.100 0.10)
[Ç- 0.20)
Purified (ç- 0.20)
ADG
Figure 9: The performance IQL. (left) and RIQL (right) on the noised, filtered, purified, and restored datasets. Performance the noised data is referred to "Naive".
D.6 Evaluation of Performance Across Different Temporal Slice Sizes (H)
In this section, we evaluate ADG with varying temporal slice sizes by setting E 1,,,,,,5,77 As described in Section 4.4, the temporal slice size is defined as 2 H + 1. Notably, H means the diffusion model takes only a single step as input and cannot access sequential information. The results are presented Figure 10. From the results, we observe that the performance of the detector and denoiser in ADG improves significantly as H increases from 0 to 3, and remains robust for H 3, 5, 7. The performance of IQL strongly correlates with that of the detector and denoiser, showing significant improvement for H 3, 5, 7 compared to H 0, 1. In contrast, DT is more resilient to variations in ADG's recovery performance across different values of H, likely due its inherent ability to leverage temporal information and mitigate the effects of corruption. This obser- vation aligns with the findings in RDT [34]. These results highlight the critical role of incorporating temporal slices in enhancing the performance of the detector, denoiser, and overall RL performance on the denoised dataset.
MSF
80
0.6
0.2
80
Normalize
0.3
20
5k
10k
Update Stops
Update Stops
Wilker (IQL)
Walker2d (DT)
--0
H=1
H-3
#=5
H=7
A-O
H
H-3
A-5
H-?
(a)
(b)
Figure 10: The performance of the ADG detector and denoiser under varying temporal slice sizes (H) is shown in (a), while (b) presents the corresponding performance of the baseline algorithms, IQL and DT, on the wwalker2d-mediiu--rplaay (the proportion of corrupted samples incorrectly identified clean). and MSE indicates the mean squared error between the restored dataset and the ground-trum -ttnh dataset.
dataset. FN represents the False Negative ratio
29

---

D.7 Evaluation under Various Dataset Scales
We further examine the robustness of ADG on MuJoCo tasks with dataset sizes of 20% and 50% of the "mediiumrreplay-22" datasets. We investigate corruption types including Random State Attack and Random Full-element Attack as described Section C.I. The comparison results are shown in Figure 11. From the results, observe that the performance of both the algorithms and their robust variants on both corrupted and restored datasets positively correlated with dataset size. Moreover, ADG outperforms the baselines across most dataset sizes, validating the effectiveness and importance dataset recovery.
haltherthh-tate
hospecstate nQ11
hontenatomeet
20000001 Slze Rario
Dutaret Size 1006
Docaget Siee Rade
Dicaset Size Rablo
Dataset Size A00i
D00860t Size Hario
Nave + ADG
Nave
+ ADD
(b)
Ettessaahstttt 1011
Yoppor state (D1)
walk 2e ID13
hallchect eeeement 10
Toguer cle
waller dineett (D))
Dec00st Sloc Raclo
Dutaect Siez 1u0s
Dussect Size Raco
Diciset Sloc Riclo
Dolant Sios 1a4s
Desenet Siee Ravl
- ADG
(d)
Paltcheetah IRIQU
hopperstatee (RIQ.I
wolker2f-store (MOL)
baftchetthh-eeeeen (H10.)
hosper element (41g)
walier ement (4003
Sporra Serte
0.2
03
9-4
10
0.2
9.2
3.2
10 0.1
0.2
2000001 Siac
Dutaeet lali
Dusasct Nale
Dilase Naclo
Jbrascl Ravio
(1
sjate 11271
hospers siate
walkcrrdissute
Ju aheeeerrrrtttt.... (107)
hesper eiom
24 cdccmmmni
10
0.2
033
3,5
0.2
5.2
1.0
0.
0.2
03
1.2
0.2
0.5
3.0
0.2
3.2
5.5
1.0
0.1
0.2
0.5
1.0
- ABG
Figure 11: Performance of ADG under random corruption across different dataset scales.
D.8 Varying Corruption Rates and Scales
We evaluate the robustness of ADG under various corruption rates {0.0, 0.5) and scales (0.0, 1.0, 2.0]. We choose the walker24dmediuu--re replay-v2" dataset downsampled to 10% of its original size to make the results more sensitive corruption rates and scales. As shown Figure 12, increasing corruption rates and scales progressively degrade the performance of baseline algorithms like IQL. and DT due to greater deviations between the corrupted and clean datasets. Nevertheless, ADG consistently enhances the overall performance of these baselines.
30

---

State Elements Attack
Attack
Elemants Attack
10
Homannas
20
10
ouest
(a)
(h)
Figure 12:
Results under various corruption rates (a) and scales (b) on the mealker2d-mediumm
replay-v2" dataset, which is downsampled to 10% of the original size.
D.9 Impact Ç in Dataset Splitting
As outlined in Section 4.4, we introduce threshold hyperparameter < to differentiate between noisy and clean samples. To emphasize the importance of selective sampling during training, we evaluate ADG with trained denoiser by varying over the range {0.02, 0.05, 0.2, 0.5, 1.0). Note that when { = 1, the dataset remains unchanged. The results are presented in Figure 13. A higher Ç incorporates increasing the risk of overfitting in the naive diffusion model. In contrast, a lower results more clean samples being misclassified as noisy, leading unnecessary restoration and loss of original dataset information. There is a trade-off in choosing the threshold Ç, and ( = 0.20 yields the lowest MSE between the restored and clean datasets, well the best D4RL score across all baseline algorithms.
noisy samples into the restored dataset without applying restoration,
1.0
Normalized Score
20
10
0.2
0.0
IQL
8.05
0.20
(=0.02
==0.10
(=0.50
{=0.05
t=
{=1.00
Scalld MSE
(a)
(b)
Figure 13: Results of (a) the detection performance in the dataset recovery process and (b) D4RL score under various Ç on the as positive and clean samples as negative. The False Positive (FP) rate represents the ratio clean samples incorrectly classified as noisy, while the False Negative (FN) rate corresponds to the ratio of noisy samples misclassified result. Scaled MSE is the MSE scaled to [0, 1], which directly reflects the deviation of the restored dataset from the clean dataset.
meetrrdd--miiuu---paa----- dataset. In (a), we label noisy samples
clean. Both are expected to be low for better dataset splitting
D.10 Additional Experiments Comparing the Use of Single vs. Double Diffusion Models
In this section, we provide further details on the implementation of ADG using single diffusion model. We reorganize Algorithm I to merge Step and Step 2 into single loop. In each step, the diffusion model is updated using both Eq. 4 and Eq. 7. We continue to use k, to detect corrupted data in the partially corrupted dataset. The results for medium-replay--2" datasets are presented in Figure 14 complete the analysis. Employing a single diffusion model improves performance in 4 out of 6 tasks compared to the baselines. However, ADG with two independent models still achieves significantly higher performance. These findings align with the conclusions in Section 5.3.
and "hopper-
31

---

Normalized Score
60
40
20
Halfcheetah (IQL)
Halfcheetah (10%, (QL)
Hopper (IQL)
Hopper (10%, IQL)
Normalized Score
40
20
Halfcheetah (DT)
Halfcheetah (10%, DT)
Hopper (DT)
Hopper (10%, DT)
Naive
ADG (w/ SD)
ADG
Figure 14: Comparison results among Naive, ADG with a single diffusion model, and ADG with separate diffusion models for IQL (upper) and DT (lower) under "halfcheetah- wewprettaammmiuurrppplaaa and "hopper-mediuu--repayyya datasets. ADG (w/ SD) denotes ADG using a single diffusion model.
D.11
Missing data
While our main focus is on additive corruption, ADG theoretically applicable to a broader class of noise, including missing data such as dropped elements. We construct a missing-date variant of each dataset by randomly zeroing out 30% the actions. We evaluate baseline offline RL algorithms and ADG under this setting and report the results averaged over four seeds in Table 9. Despite the substantial information loss, ADG consistently improves performance across all environments, recovering significant portion of the original policy quality.
Table 9: Performance under missing data. Results are averaged over 4 random seeds. "Missing Data" refers to standard offline RL applied to the corrupted dataset. "Missing Data (w/ ADG)" denotes applying ADG for recovery.
Environment
Missing Data
Missing Data (w/ ADG)
Clean
halfcheetah-mr
3.8
31.3
38.9
hopper-mr
14.0+E1.0
45.5++16.4 32.7+44.1
86.
walker2d-mr
7.2++0.3
59.9+22.7
These results indicate that ADG not limited to additive noise settings and generalizes well to missing data, further demonstrating its robustness in practical offline RL scenarios where various forms of corruption may co-occur.
D.12 Extending ADG to Recent Offline RL Methods
To evaluate the broader applicability of ADG beyond standard baselines, we further apply it to two recent state-of-the-art offline RL algorithms: A2PR [24] and NUNO [17]. Specifically, test their robustness under the Random State Attack corruption setting and assess the performance gains when combined with ADG. This allows examine whether ADG can consistently enhance policy quality across diverse algorithmic backbones.
As shown in Table 10, ADG significantly improves performance for both A2PR and NUNO in all environments, highlighting its general utility mitigating data corruption across a range of policy learning strategies.
32

---

Table 10: Performance of A2PR and NUNO under Random State Attack, with and without ADG. Results are averaged over four random seeds.
Environment A2PR (noised)
NUNO (noised)
A2PR (w/ ADG)
NUNO (w/ ADG)
halfcheetahh-n
4.4
27.3=7.8
29.2 29.9 25.8 9.3+0.8
13.2
hopper-mi mr walker2d-mi
16.1=5.0 4.6+2.33
12.6=008
11.0=5.55
14.6#4.1
D.13 Recovery under Fully Corrupted Trajectories
In ADG, each trajectory is processed through overlapping temporal slices, where the detector fo- cuses on verifying whether the central element within each slice is corrupted. This design is not limitation but a deliberate choice: it allows ADG to (1) leverage temporal context from both past and future elements for detection, (2) avoid compounding errors when modeling long trajectories, and (3) ensure a fair comparison with prior methods such as RDT [34], which adopt the same middle- step corruption setup.
Importantly, because slices are generated examined the center of some slice, meaning that all time steps eventually verified and potentially recovered during the process, Nevertheless, to further demonstrate ADG's s flexibility and robustness, conduct additional experiments under more challenging corruption patterns.
Midding-window manner, every element the dataset
Specifically, we explore two extended variants beyond the standard setting:
1. All-steps corrupted + middle recovery: all time steps in each trajectory corrupted, but only the middle element each slice is recovered, testing whether local recovery remains effective when global context is degraded.
2. All-steps corrupted + full recovery: all time steps are corrupted and each is recovered sequentially using ADG, assessing whether full-step recovery can further enhance temporal consistency.
shown Table 11, ADG remains effective even when all time steps are only the middle element still yields competitive performance, highlighting the strength of local re- covery guided by contextual information. Furthermore, performing full-step recovery leads to con- sistent improvements across all environments, confirming that ADG can scale to dense corruption and flexibly generalize beyond benchmark configurations.
Table 11: Performance under different recovery strategies. Results are averaged over three random seeds. Environment
Middle-only Recovery
All-steps Corrupted + Middle Recovery
All-steps Corrupted + Full Recovery
halfcheetah-mrr
31.3 + 0.7
26.4 0.9
33.5 0.8
hopper-mr
45.5 + 16.4
40.2 + 3.7
52.8 4.5
walker2d-mr
32.7 + 4.1
27.9 + 3.3
36.1 + 2.9
D.14 Robustness to Non-Additive (Multiplicative) Gaussian Noise
This experiment evaluates the robustness of ADG when the offline dataset corrupted with multi- plicative (non-additive) Gaussian noise. Specifically, each data point z is perturbed as:
i = x (1+c),
E E~ E~~
(36)
where represents Gaussian noise scaled by the original value, unlike standard additive noise.
We conducted experiments on standard offline RL benchmark environments under this corruption.
Table 12 reports the performance of (i) standard offline RI trained directly the corrupted dataset, (ii) ADG applied for recovery, and (iii) the original clean dataset. Results are averaged over random seeds.
33

---

Table 12: Performance under Multiplicative Gaussian Noise. 'Multiplicative Noise" refers to stan- dard offline RL applied to the corrupted dataset. 'Multiplicative Noise (w/ ADG) denotes applying ADG for recovery.
Environment
Multiplicative Noise
Multiplicative Noise (w/ ADG)
Clean
halfcheetah-mr 6.1
29.8=0.6
38.9:0.5
hopper-mr walker2d-mr
12.5=0.9
47.
81.816.9
10.7++0.7
34.9 2.1
59.9+2.7
From the results, we observe that while standard offline RL suffers a severe performance drop under multiplicative noise, ADG substantially mitigates this degradation, achieving performance much closer to the clean-data scenario. This demonstrates that ADG is not only robust to additive noise but also effective against non-additive, multiplicative Gaussian corruption, confirming its practical utility in noisy offline RL datasets.
D.15 Joint Impact of Detection Hyperparameters on Downstream RL Performance
To investigate the joint influence of the detector hyperparameters K₂ and Ç on downstream reinforce- ment learning performance, we conducted additional experiments on the kitchen-complete dataset under the Random State Attack setting. Specifically, ki and were varied over the sets (15, 30, 50) and (0.10, 0.20, 0.50), respectively. For each configuration, both the detector and denoiser were retrained, and evaluated the resulting false negative (FN) and false positive (FP) rates, together with the downstream performance of IQL and DT. All results are averaged over three random seeds.
Table 13: Ablation on the trade-offs between FP/EN and downstream RL performance under Run- dom State Attack. Results are averaged over three random seeds.
ka
C
FN(()
FP(%)
IQL.
DT
0.10
2.7
52.6
22.0
13.1
15
0.20
7.3
12.0
49.9
56.4
0.50
27.2
2.7
34.1
42.6
0.10
1.4
45.5
25.0
12.7
30
0.20
3.8
4.9
45.0
12.4
0.50
25.4
1.7
36.0
42.1
0.10
8.4
55.4
16.5
-0.3
50
0.20
19.5
6.0
37.6
46.9
0.50
29.8
1.8
18.2
31.0
As shown in Table 13, the detector achieves the lowest FN at moderate values of k. and Ç. while both smaller and larger values lead increased FN, reflecting under- and over-filterig, respectively. Increasing C treats larger portion of data as clean, which substantially reduces FP but can also increase FN due to missed corruptions.
For downstream performance, we observe a clear negative correlation between FN and final RL results: higher FN values consistently degrade the performance of both IQL and DT. In particular, when = 0.50, both algorithms exhibit significant drops, likely because excessive data filtering removes crucial sequential dependencies. This effect more pronounced on the relatively small kitchen- complete dataset compared to larger benchmarks such as MuJoCo. Interestingly, lower FP does not necessarily yield better downstream results, corroborating our main findings.
E
Discussion and Limitation
While ADG demonstrates promising results in handling corrupted offline RL datasets, our work has several limitations that warrant discussion. The three-stage diffusion process (detection + denoising) introduces additional training time compared to standard offline RL methods. The performance of
34

---

ADG depends on the choice of key hyperparameters such as the temporal slice size H and corruption threshold ç. The trajectory-basee based recovery mechanism (Section 4.4) relies on temporal consistency. For environments with highly discontinuous dynamics or sparse rewards, the current dice-based approach may miss long-range dependencies.
35


Reviews:
[{"review_id": "THqSnRw1nM", "reviewer": "Reviewer_PmCq", "summary": "The paper develops a new framework to handle the setting where there is noise in the offline dataset. It is a three-stage framework:  \n1. Use ambient loss to train a diffusion model as a detector to distinguish clean vs. noisy data.  \n2. Train a diffusion policy on the filtered clean data.  \n3. Use the learned diffusion policy to denoise the previously noisy data, recovering a clean dataset.  \n\nThe paper provides a solid theoretical analysis and comprehensive experiments to validate the framework.", "questions": "- In Figure 2, does \"10\\% dataset\" refer to sampling 10\\% of transitions or 10\\% of entire trajectories? This should be clarified.\n- The paper sets $\\eta = 0.3$, meaning most data are clean. But when $\\eta$ is high (e.g., 0.7) and the data size is small, the remaining clean data might be insufficient to train a good diffusion policy. Do the authors have thoughts or strategies to address this?\n- In Line 148, the authors claim that even a small amount of noise leads to overfitting. Could the authors provide theoretical or empirical support for this? Paper [1] suggests that models can learn from large amounts of noisy data, which seems contradictory.\n- $k_a$ is a crucial hyperparameter. How should it be selected in practice?\n- In Assumption 4.2, what is the precise mathematical meaning of \"can be effectively learned from samples\"?\n- In Proposition 4.4, the assumption that the noise prediction error for $\\epsilon_\\theta$ follows a Gaussian distribution is strong. Furthermore, Line 199 assumes $\\sigma_k = \\sigma$ for all $k$, implying uniform error across diffusion steps. Shouldn’t we expect larger errors at earlier steps (larger $k$)?\n- Is the threshold $\\xi$ consistent across tasks and methods?\n- After obtaining the recovered dataset, do you retrain the diffusion policy from scratch or continue training from the Stage 2 policy?\n- In Figure 3, intuitively a larger $k_a$ should lead to lower false negatives (FN), but the plot suggests otherwise. Can the authors clarify this behavior?\n- Line 282: What is $t_n$? Is it the same as $k_a$?\n\n[1] Daras, Giannis, et al. *Ambient diffusion: Learning clean distributions from corrupted data*. NeurIPS 2023.", "limitations": "The task is important, and the proposed framework is novel. However, its reliance on strong assumptions in the theoretical analysis limits its generality. Additionally, the three-stage pipeline raises concerns about scalability and error accumulation.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The theoretical analysis appears sound and technically correct; no obvious flaws were observed.\n- The experiments are well-designed, considering different types of noise (random vs. adversarial; state-only vs. full) and demonstrating consistent gains across integration with various methods.\n- The ablation studies are thorough, explaining the effect of key hyperparameters like $k_a$ and $\\xi$.\n\n**Weaknesses:**\n\n- There is still a noticeable gap between the theoretical assumptions and practical settings. For example, the theoretical framework assumes Gaussian noise, while the experiments are based on uniform noise. (See Questions section.)", "was_revised": true, "final_justification": "The authors’ responses have adequately addressed my concerns. I am open to recommending acceptance of the paper.", "rebuttal": "To begin with, we would like to sincerely thank the reviewer for taking the time to provide valuable feedback and for raising insightful questions regarding our work.\n\n>In Figure 2, does \"10% dataset\" refer to sampling 10% of transitions or 10% of entire trajectories? This should be clarified.\n\nIn Figure 2, '10% dataset' refers to using 10% of the entire trajectories, consistent with the setting in RDT [3].\n\n> The paper sets $\\eta=0.3$, meaning most data are clean. But when $\\eta$ is high (e.g., 0.7) and the data size is small, the remaining clean data might be insufficient to train a good diffusion policy. Do the authors have thoughts or strategies to address this?\n\nWe would like to clarify that $\\eta = 0.3$ is the most commonly used setting in corruption-robust RL, as adopted in RIQL [2] and RDT [3]. Indeed, when the proportion of corrupted data is high, it remains an unresolved challenge for the research community. In our setting, we cannot guarantee that a diffusion model trained on limited clean data can effectively recover the corrupted samples. We will seek to address this limitation in our future work.\n\n> In Line 148, the authors claim that even a small amount of noise leads to overfitting. Could the authors provide theoretical or empirical support for this? Paper [1] suggests that models can learn from large amounts of noisy data, which seems contradictory.\n\nIn line 148, we state that\n```\nOn the other hand, if the naive diffusion process is applied to corrupted samples during training, even small amounts of noise in the training set can cause the diffusion model to overfit to these perturbed points, leading to poor performance during subsequent sampling.\n```\nWhen training the noise detector, we observe that using a naive diffusion model to fit the partially corrupted data yields a training MSE loss comparable to that obtained with our ambient DDPM-based noise detector. (Due to rebuttal policies, we are unable to provide the corresponding training curves at this stage.) However, during corrupted sample detection, the performance of the naive diffusion model is extremely poor; it fails to effectively identify noised samples, which subsequently degrades the performance of the downstream denoiser and RL training. We present the relevant ablation study in Figure 3 on page 8 of our main paper.\n\nThe main distinction between our approach and [1] is that we address state-based RL tasks rather than image-based tasks. In image tasks, each pixel is closely related to its neighboring pixels, facilitating effective noise detection. In contrast, in state-based RL, adjacent states may not exhibit strong correlations, and the downstream RL tasks are typically more sensitive to noise present in the training dataset.\n\n> $k_a$ is a crucial hyperparameter. How should it be selected in practice?\n\nThank you for the thoughtful question. The value of $k_a$ controls the magnitude of the added noise and is determined based on the estimated corruption severity in the affected portion of the dataset. The objective is to add enough noise to approximately satisfy the assumption in \\[8]—that the dataset is corrupted by a consistent scale of noise—while avoiding overly large noise that would erase too much useful information. We will clarify this rationale and include additional details in the revised version.\n\n> In Assumption 4.2, what is the precise mathematical meaning of \"can be effectively learned from samples\"?\n\nThank you for pointing out the informal expression in Assumption 4.2. We apologize for any confusion caused. To clarify, let $p_\\theta^{approx} (\\mathbf{x}^{k}\\vert\\mathbf{x}^{k+1})$ denote the distribution of the forward process of DDPM learned from the approximate distribution $\\varrho(\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0)$, and let $p_\\theta (\\mathbf{x}^{k}\\vert\\mathbf{x}^{k+1})$ denote the corresponding distribution learned from the clean distribution $q(\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0)$. \n\nThe revised Assumption 4.2 is as follows:\n\n* There exists a positive constant $c$ such that, for any $k \\geq k_a$, if the Kullback-Leibler (KL) divergence satisfies $D_{KL}[q (\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0) \\Vert \\varrho(\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0)] < c$,  then the ambient DDPM described in Corollary 2.1, when trained on samples from the approximate distribution $\\varrho(\\mathbf{x}^k\\vert\\mathbf{x}^0)$, satisfies $D_{KL}[p_\\theta (\\mathbf{x}^{k}\\vert\\mathbf{x}^{k+1}) \\Vert p_\\theta^{approx} (\\mathbf{x}^k\\|\\mathbf{x} ^{k+1})] < \\varepsilon$ for any $k \\geq k_a$, where $\\varepsilon$ is an arbitrarily small positive value.\n\n\n> In Proposition 4.4, the assumption that the noise prediction error for $\\epsilon_\\theta$ follows a Gaussian distribution is strong. Furthermore, Line 199 assumes $\\sigma_k = \\sigma$ for all $k$, implying uniform error across diffusion steps. Shouldn’t we expect larger errors at earlier steps (larger k)?\n\nIn response to your first point, we believe that assuming the noise prediction error follows a Gaussian distribution is an appropriate choice, supported by the central limit theorem, and it is the most common assumption in measurement theory.\n\nRegarding your second concern, we agree that assuming the error variance is a strictly increasing function of $k$ is more appropriate. However, this does not affect our final conclusion, which is that $\\mathrm{SNR}(k)$ achieves its maximum value at $k=k_a$. The derivation is shown below:\n$$\\frac{\\partial{SNR}(k)}{\\partial{k}}=\\frac{\\partial{SNR}(k)}{\\partial \\bar{\\alpha}_k} \\cdot \\frac{\\partial \\bar{\\alpha}_k}{\\partial k} + \\frac{\\partial{SNR}(k)}{\\partial \\sigma_k} \\cdot \\frac{\\partial \\sigma_k}{\\partial k}$$\n$$= \\frac{\\iota^2\\sigma_k^2}{(1-\\bar{\\alpha}_k)^2\\sigma_k^4} \\cdot \\frac{\\partial \\bar{\\alpha}_k}{\\partial k} + \\frac{2 \\iota^2 \\bar{\\alpha}_k^2 }{(1-\\bar{\\alpha}_k)^2\\sigma_k^3} \\cdot \\frac{\\partial \\sigma_k}{\\partial k}$$\n\nwhere $\\frac{\\partial \\bar{\\alpha}_k}{\\partial k}<0$ and $\\frac{\\partial \\sigma_k}{\\partial k}<0$ by definition. Thus, $\\frac{\\partial{SNR}(k)}{\\partial{k}}$ is also negative, as the coefficients are all positive. This completes the proof.\n\n>Is the threshold $\\zeta$ consistent across tasks and methods?\n\nYes, we use a consistent value of $\\zeta$ across all tasks and methods.\n\n\n> After obtaining the recovered dataset, do you retrain the diffusion policy from scratch or continue training from the Stage 2 policy?\n\nWe retrain the diffusion policy from scratch. While continuing training from the Stage 2 policy is also a viable option, we chose to retrain in order to avoid potential instability introduced.\n\n> In Figure 3, intuitively a larger $k_a$ should lead to lower false negatives (FN), but the plot suggests otherwise. Can the authors clarify this behavior?\n\nThank you for your question — it touches on the core insight of our work. In ADG, we aim to ensure that the assumption in Ambient Diffusion—that the entire dataset is corrupted by a known noise distribution—holds true by explicitly injecting strong noise. In our experiments, we first add a sufficiently large amount of additional noise so that the entire dataset is subjected to a similar noise level. To retain more useful information, we then gradually decrease this value and observe the experimental results. We find that $k_a = 30$ achieves the best FN score, suggesting that it strikes a near-optimal balance in this trade-off.\n\n> Line 282: What is $t_n$? Is it the same as $k_a$?\n\nThank you for pointing this out. Yes, $t_n$ refers to the same quantity as $k_a$. We apologize for the typo.\n\n[1] Daras, Giannis, et al. Ambient diffusion: Learning clean distributions from corrupted data. NeurIPS 2023.\n\n[2] Yang, Rui, et al. Towards Robust Offline Reinforcement Learning under Diverse Data Corruption. ICLR 2024.\n\n[3] Xu, Jiawei, et al. Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling. ICLR 2025."}, {"review_id": "N7meGMXW7O", "reviewer": "Reviewer_zVcz", "summary": "The paper proposes a diffusion-based framework (ADG) to handle data corruption in offline reinforcement learning (RL). The core innovation lies in leveraging diffusion models: the algorithm trains on partially corrupted datasets with theoretical guarantees to distinguish between clean and corrupted data. Experiments on benchmarks (MuJoCo, Kitchen, and Adroit) demonstrate that ADG consistently improves offline RL algorithms under various corruption scenarios.", "questions": "Generalization to Real-World Data: The experiments are conducted on simulated datasets (MuJoCo, Kitchen, Adroit). However, the data robustness is a real world problem *in nature*. How would ADG perform on real-world corrupted datasets with heterogeneous noise sources? Could the authors provide insights or preliminary results on real-data applicability?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: the paper builds upon Daras et al.'s work on Ambient Diffusion Models, which enables diffusion training on consistently corrupted datasets. ADG extends this to partially corrupted datasets (common in real-world RL), addressing cases where noise distribution varies across samples. This is formalized in Theorem 4.3, proving Ambient DDPM can learn from approximated distributions with bounded KL divergence. To summarize, this paper theoretically extends Daras et al. by formalizing diffusion training for partially corrupted data (Theorem 4.3) and algorithmically innovates with a detection-recovery pipeline.\n\nWeakness: while experiments cover simulated benchmarks (random corruption and adversarial corruption), the paper does not address practical challenges like real-world data sets. The paper mentions this but offers no experimental supports.", "was_revised": false, "final_justification": "", "rebuttal": "To begin with, we would like to sincerely thank the reviewer for their positive evaluation of our work and for taking the time to provide valuable feedback and raise insightful questions.\n\n> Generalization to Real-World Data: The experiments are conducted on simulated datasets (MuJoCo, Kitchen, Adroit). However, the data robustness is a real world problem in nature. How would ADG perform on real-world corrupted datasets with heterogeneous noise sources? Could the authors provide insights or preliminary results on real-data applicability?\n\nThank you for the insightful question. In principle, ADG poses no inherent difficulty in generalizing to real-world scenarios with heterogeneous noise sources. The core methodology is designed to handle unknown and complex corruption patterns, making it well-suited for practical applications. Due to limited time and computational resources, we have not yet conducted large-scale real-world experiments. However, we greatly appreciate this suggestion and would be happy to include relevant discussions or preliminary results on real-world applicability in the final version."}, {"review_id": "nkcSggEPOI", "reviewer": "Reviewer_onPe", "summary": "The paper proposes a novel diffusion-based algorithm to tackle the challenge of partially corrupted data in offline reinforcement learning. The approach integrates two diffusion models: an ambient DDPM to identify corrupted samples, and a standard DDPM trained on the filtered, uncorrupted data to reconstruct the corrupted ones. The recovered dataset is then used to train an existing offline RL algorithm. Experimental results on MuJoCo, Kitchen, and Adroit benchmarks demonstrate the robustness of the proposed method compared to a naive baseline.", "questions": "Can you pleas address my points on the weaknesses of your paper?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ The general ideas and purposes of the two diffusion models are clearly described and justified for the most part.\n\n+ The ambient DDPM component is supported by reasonable theoretical analysis.\n\n+ Experimental results across diverse environments demonstrate the robustness of the proposed algorithm compared to the naive baseline.\n\nWeaknesses:\n\n- The term \"effectively learned\" in Assumption 4.2 is vague and should be clarified with more precise language or formal criteria.\n\n- The ambient DDPM relies on a fixed noise injection step k_a, yet the paper does not discuss how this parameter is selected or how sensitive the algorithm’s performance is to this choice. \n\n- In the proposed algorithm, it appears that only a single element—either a state or a triplet—at the middle time step of each trajectory is verified and potentially recovered. The rationale for this design choice is unclear. Intuitively, it seems more appropriate to examine and potentially recover all elements within a trajectory.\n\n− The experimental comparison is limited to a naive baseline. The paper does not show comparison with other existing methods designed to handle data corruption in offline RL, such as the approach introduced in [38]. Including such baselines would provide a more comprehensive evaluation of the proposed method’s effectiveness.", "was_revised": true, "final_justification": "The authors have addressed my concerns. Therefore, I increased my score.", "rebuttal": "To begin with, we would like to thank the reviewer for their careful evaluation of our work and for providing constructive feedback and thoughtful questions. \n\n> The term \"effectively learned\" in Assumption 4.2 is vague and should be clarified with more precise language or formal criteria.\n\nThank you for pointing out the informal expression in Assumption 4.2. We apologize for any confusion caused. To clarify, let $p_\\theta^{approx} (\\mathbf{x}^{k}\\vert\\mathbf{x}^{k+1})$ denote the distribution of the forward process of DDPM learned from the approximate distribution $\\varrho(\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0)$, and let $p_\\theta (\\mathbf{x}^{k}\\vert\\mathbf{x}^{k+1})$ denote the corresponding distribution learned from the clean distribution $q(\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0)$. \n\nThe revised Assumption 4.2 is as follows:\n\n* There exists a positive constant $c$ such that, for any $k \\geq k_a$, if the Kullback-Leibler (KL) divergence satisfies $D_{KL}[q (\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0) \\Vert \\varrho(\\mathbf{x}^{k+1}\\vert\\mathbf{x}^0)] < c$,  then the ambient DDPM described in Corollary 2.1, when trained on samples from the approximate distribution $\\varrho(\\mathbf{x}^k\\vert\\mathbf{x}^0)$, satisfies $D_{KL}[p_\\theta (\\mathbf{x}^{k}\\vert\\mathbf{x}^{k+1}) \\Vert p_\\theta^{approx} (\\mathbf{x}^k\\|\\mathbf{x} ^{k+1})] < \\varepsilon$ for any $k \\geq k_a$, where $\\varepsilon$ is an arbitrarily small positive value.\n\n> The ambient DDPM relies on a fixed noise injection step $k_a$, yet the paper does not discuss how this parameter is selected or how sensitive the algorithm’s performance is to this choice.\n\nThank you for the thoughtful question. In ADG, we aim to ensure that the assumption in Ambient Diffusion—that the entire dataset is corrupted by a known noise distribution—holds true by explicitly injecting strong noise. In our experiments, we first add a sufficiently large amount of additional noise so that the entire dataset is subjected to a similar noise level. To retain more useful information, we then gradually decrease this value and observe the experimental results. We find that $k_a = 30$ achieves the best FN score, suggesting that it strikes a near-optimal balance in this trade-off. We will clarify this point in the revised version.\n\n> In the proposed algorithm, it appears that only a single element—either a state or a triplet—at the middle time step of each trajectory is verified and potentially recovered. The rationale for this design choice is unclear. Intuitively, it seems more appropriate to examine and potentially recover all elements within a trajectory.\n\nTo ensure a fair comparison with prior works such as RDT [1], our main experiments adopt the same setup where only the middle element of each trajectory is corrupted and subsequently recovered. However, ADG is not limited to this setting—in principle, it supports arbitrary corruption patterns and flexible recovery strategies.\n\nTo demonstrate this capability, we conduct an additional experiment with more challenging settings: (1) corrupting all time steps in each trajectory while still recovering only the middle element, and (2) corrupting all time steps and recovering all of them using ADG. Results (shown in Table 1) indicate that while recovery at a single point remains effective, full recovery across the trajectory further improves performance, highlighting the extensibility and robustness of ADG beyond the standard benchmark setup.\n\n**Table 1. Performance under different recovery strategies.**\n\n| Env            | Middle-only Recovery | All-steps Corrupted + Middle Recovery | All-steps Corrupted + Full Recovery |\n| -------------- | -------------------- | ------------------------------------- | ----------------------------------- |\n| halfcheetah-mr | 31.3 ± 0.7           | 26.4 ± 0.9                            | **33.5 ± 0.8**                      |\n| hopper-mr      | 45.5 ± 16.4          | 40.2 ± 3.7                            | **52.8 ± 4.5**                      |\n| walker2d-mr    | 32.7 ± 4.1           | 27.9 ± 3.3                            | **36.1 ± 2.9**                      |\n\n> The experimental comparison is limited to a naive baseline. The paper does not show comparison with other existing methods designed to handle data corruption in offline RL, such as the approach introduced in [38]. Including such baselines would provide a more comprehensive evaluation of the proposed method’s effectiveness.\n\nThank you very much for the valuable suggestion. However, to the best of our knowledge, there are no newer baselines specifically targeting robustness during the training phase in offline RL. The baselines we include in our experiments represent the most relevant and up-to-date methods for this setting. The method referenced in [38], while important, focuses on handling test-time corruption and assumes access to clean data during training. As such, it is not directly comparable to our approach, which addresses corruption present throughout the entire offline dataset.\n\n[1] Xu, Jiawei, et al. Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling. ICLR 2025."}, {"review_id": "xUylHd1bxw", "reviewer": "Reviewer_4o9W", "summary": "In practice, offline-RL datasets contain heterogeneous, trajectory-specific corruptions that are already present during training. Motivated by this challenge, the authors propose an ADG approach that can, firstly, learn a reliable noise predictor from partially corrupted datasets, and then use it to detect corrupted data for training a denoiser using the confidently clean subset. Finally, the corrected data is combined with clean data for subsequential offline RL training. This framework provides an effective way to recover high-quality data in the very setting where existing diffusion models break down.", "questions": "1. Can the current results be extended to bound the policy-value gap or provide a sample complexity guarantee? Although this will not negatively affect my evaluation.\n2. Have the authors explored how to choose the hyperparameters across tasks?\n3. How is ADG robust to non-additive Gaussian noise?\n4. What are the measured false-positive / false-negative rates, and how sensitive is downstream RL to them?", "limitations": "1. Current analysis stops short of guaranteeing final policy quality.\n2. Theoretical results hinge on the current noise model, other forms remain unanalyzed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Ambient DDPM theory is extended to discrete DDPM and linked to an SNR criterion. The theoretical results are sound and well-justified.\n2. ADG consistently outperforms several baseline algorithms while adding <1h data preprocessing and introducing only two intuitive hyper-parameters. \n\nWeaknesses:\n\n1. The theoretical analysis stops at noise prediction, but does not bound the dataset-recovery error or the final policy-value gap, leaving the end-to-end guarantee open.\n2. All theoretical results assume the corruption is formed by adding scaled Gaussian noise to each clean sample. This is what keeps the two forward chains share the same covariance matrix, allowing an analytic, monotonically decreasing KL expression. The missing-data ablation is helpful but limited.\n3. $k_a$ must balance information retention against the KL bound, yet no cross-task rule is given. In addition, $\\zeta$ is fixed at 0.2 for the experiments, risking removal of rare but valuable samples.", "was_revised": true, "final_justification": "I would like to thank the authors for their thorough explanation and additional experiments, which have addressed most of my concerns. I encourage the authors to fold these clarifications into the revised version. Accordingly, I am raising my score by one point.", "rebuttal": "We would like to start by expressing our sincere gratitude to the reviewer for their thorough assessment of our work and for offering valuable feedback and insightful questions.\n\n> The theoretical analysis stops at noise prediction, but does not bound the dataset-recovery error or the final policy-value gap, leaving the end-to-end guarantee open.\n\nTo the best of our knowledge, the error bound for data recovery remains an open problem, even for naive diffusion models. In practice, the performance of diffusion models varies considerably across different tasks and domains. In our work, diffusion is used solely as a noise detector and denoiser. \nRegarding downstream RL algorithms, we employ several methods: some are designed to be robust to data corruption (e.g., RIQL, RDT), while others are not (e.g., CQL, IQL, DT). Consequently, we are currently unable to provide a theoretical analysis on this aspect. We plan to address and improve upon this limitation in future research.\n\n> All theoretical results assume the corruption is formed by adding scaled Gaussian noise to each clean sample. This is what keeps the two forward chains share the same covariance matrix, allowing an analytic, monotonically decreasing KL expression. The missing-data ablation is helpful but limited.\n\nIn response to your first point, we assume that the corruption noise follows a Gaussian distribution, and the diffusion forward process involves adding scaled Gaussian noise to the original data. This assumption is natural, as it aligns with prior work and facilitates tractable theoretical analysis.\n\nRegarding your second point, the missing-data condition is introduced by RDT [1], and we follow their experimental setup. We report results across three MUJOCO tasks, and believe that our experimental findings are sufficiently convincing.\n\n> $k_a$ must balance information retention against the KL bound, yet no cross-task rule is given. In addition, $\\zeta$ is fixed at 0.2 for the experiments, risking removal of rare but valuable samples.\n\nThank you for the thoughtful question. In ADG, we aim to ensure that the assumption in Ambient Diffusion—that the entire dataset is corrupted by a known noise distribution—holds true by explicitly injecting strong noise. In our experiments, we first add a sufficiently large amount of additional noise so that the entire dataset is subjected to a similar noise level. To retain more useful information, we then gradually decrease this value and observe the experimental results. We find that $k_a=30$ achieves the best FN score, suggesting that it strikes a near-optimal balance in this trade-off. We will clarify this point in the revised version.\n\nThe design of $\\zeta$ reflects a trade-off: by discarding a small portion of uncorrupted samples, we aim to improve the overall cleanliness of the dataset. However, if a dataset contains rare but highly valuable samples whose importance outweighs the benefit of increased overall data purity, users may choose to set $\\zeta$ more conservatively.\n\n> Can the current results be extended to bound the policy-value gap or provide a sample complexity guarantee? Although this will not negatively affect my evaluation.\n\nThank you for your suggestion. In our work, diffusion is employed exclusively as a noise detector and denoiser. With respect to downstream RL algorithms, we utilize a variety of approaches: some are value-based methods (e.g., CQL, IQL, RIQL), while others are not (e.g., DT, RDT). Addressing this suggestion is a considerable challenge given the limited time available during the rebuttal period, and we believe it does not constitute the primary contribution of our work.\n\n> Have the authors explored how to choose the hyperparameters across tasks?\n\nWe have explored how to choose the key hyperparameters $k_a$ and $\\zeta$. The guideline for selecting $k_a$ is provided in our response to Weakness 3. For $\\zeta$, we conducted a parameter sweep over the values {0.05, 0.1, 0.2, 0.5}, as shown in Figure 4. However, we did not investigate other hyperparameters, such as the number of layers in the diffusion network. Our focus was on hyperparameters that directly impact robustness to corrupted data.\n\n> How is ADG robust to non-additive Gaussian noise?\n\nAlthough, to the best of our knowledge, \"non-additive Gaussian noise\" has not been widely explored in robust RL, our newly provided supplementary experiments demonstrate that ADG maintains strong performance even when the dataset is corrupted with non-additive (e.g., multiplicative) Gaussian noise:\n\n$$\n\\tilde{x} = x \\cdot (1 + \\epsilon), \\quad \\epsilon \\sim \\mathcal{N}(0, \\sigma^2)\n$$\n\nWe tested ADG under this setting and observed minimal performance degradation compared to the additive case, confirming its robustness to non-additive Gaussian corruption. Detailed results and comparisons are as following:\n\n| Env            | Multiplicative Noise | Multiplicative Noise (w/ ADG) | Clean      |\n| -------------- | -------------------- | ----------------------------- | ---------- |\n| halfcheetah-mr | 6.1 ± 0.4            | **29.8 ± 0.6**                | 38.9 ± 0.5 |\n| hopper-mr      | 12.5 ± 0.9           | **47.2 ± 3.5**                | 81.8 ± 6.9 |\n| walker2d-mr    | 10.7 ± 0.7           | **34.9 ± 2.1**                | 59.9 ± 2.7 |\n\n\n\n> What are the measured false-positive / false-negative rates, and how sensitive is downstream RL to them?\n\nThank you for the question. In our setting, false positives (FP) refer to clean samples that are mistakenly identified as corrupted, while false negatives (FN) refer to truly corrupted samples that are incorrectly retained as clean. Among the two, FN is typically more harmful to downstream RL performance, as corrupted samples used for training can significantly bias the learned policy. In contrast, while FP reduces the amount of usable data, it does not introduce harmful bias. As shown in the paper, we find that ADG achieves a low FN rate under well-chosen hyperparameters (e.g., $k_a = 30$), which is critical for maintaining the quality of the learned policy. We will include more quantitative results on FP/FN trade-offs and their impact on downstream performance in the final version.\n\n> Limitiations:\n1.Current analysis stops short of guaranteeing final policy quality.\n2. Theoretical results hinge on the current noise model, other forms remain unanalyzed.\n\nThanks for your suggestions. We will add the following paragraph to the Limitation Section of our current work:\n\nOne limitation of our current theoretical analysis is that it does not involve the quality of downstream RL policies. Additionally, we restrict our investigation to Gaussian additive noise, following the standard assumption in classical diffusion models.\n\n[1] Xu, Jiawei, et al. Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling. ICLR 2025."}]

Author rebuttal:

Meta-review: This paper proposes the use of diffusion models to address the noisy datasets in offline reinforcement learning. The proposed method achieves favorable results and has a theoretical flavor. The rebuttal was fruitful, addressed reviewers’ concerns and, consequently, three of them increased their scores. At the end, all four reviewers unanimously recommended acceptance. 
Upon further examination, this decision was also supported by the area chairs.
Decision: Accept (poster)