Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learning
Authors: Shijie Liu
Andrew Craig Cullen
Paul Montague
Sarah Monazam Erfani
Benjamin I. P. Rubinstein
Venue: ICLR 2025 Poster (2025)
Keywords: Adversarial Machine Learning
Certified Robustness
Reinforcement Learning
Poisoning Attack
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We strengthen offline RL's certified defense against poisoning attacks in more general RL settings via Differential Privacy, achieving much greater robustness compared to previous methods.

Abstract: Similar to other machine learning frameworks, Offline Reinforcement Learning (RL) is shown to be vulnerable to poisoning attacks, due to its reliance on externally sourced datasets, a vulnerability that is exacerbated by its sequential nature. To mitigate the risks posed by RL poisoning, we extend certified defenses to provide larger guarantees against adversarial manipulation, ensuring robustness for both per-state actions, and the overall expected cumulative reward. Our approach leverages properties of Differential Privacy, in a manner that allows this work to span both continuous and discrete spaces, as well as stochastic and deterministic environments---significantly expanding the scope and applicability of achievable guarantees. Empirical evaluations demonstrate that our approach ensures the performance drops to no more than 50% with up to 7% of the training data poisoned, significantly improving over the 0.008% in prior work (Wu et al., 2022), while producing certified radii that is 5 times larger as well. This highlights the potential of our framework to enhance safety and reliability in offline RL.

Published as a conference paper at ICLR 2025
MULTI-LEVEL CERTIFIED DEFENSE AGAINST POISONING ATTACKS IN OFFLINE REINFORCEMENT LEARNING
Shijie Liu't, Andrew C. Cullen', Paul Montague", Sarah Erfani', Benjamin I. P. Rubinstein' 'School of Computing and Information Systems, University of Melbourne, Melbourne, Australia Defence Science and Technology Group, Adelaide, Australia "shi e iunnimmieleemmmm lb. edu.au
ABSTRACT
Similar to other machine learning frameworks, Offline Reinforcement Learning (RL) is shown to be vulnerable poisoning attacks, due to its reliance on exter- nally sourced datasets, a vulnerability that is exacerbated by its sequential nature. To mitigate the risks posed by RL poisoning, we extend certified defenses to pro- vide larger guarantees against adversarial manipulation, ensuring robustness for both per-state actions, and the overall expected cumulative reward. Our approach leverages properties of Differential Privacy, in a manner that allows this work to span both continuous and discrete spaces, as well as stochastic and deterministic Signifiaann expanding the scope and applicability of achievable guarantees. Empirical evaluations demonstrate that our approach ensures the per- formance drops to more than 50% with up to 7% of the training data poisoned, significantly improving over the 0.008% in prior work (Wu et al. 2022). while producing certified radii that is 5 times larger well. This highlights the poten- tial of our framework to enhance safety and reliability in offline RI.
INTRODUCTION
Offline Reinforcement Learning (RL., also known as batch RI, involves training policies entirely from pre-collected datasets. Doing so is particularly advantageous in scenarios where directly in-
teracting with the environment is costly, risky, or infeasible, such as healthcare (Wang et al. 2018
autonomous driving (Pan et al. 2017), and robotics (Gürtler al. 2023). Due to this, offinne RL mechanistically shares the same vulnerability to data poisoning attacks (Kiourti et al. 2020 Wang et al.) 2021) as traditional classifiers, in which adversarial manipulation of the training data can lead to suboptimal or harmful decisions. Such vulnerability is further intensified by the dependence on external datasets collected by unknown behavioral agents and the dynamic, sequential decision- making process of RL. Across industrial users of RL, poisoning attacks are broadly considered to pose the most pressing security risk (Kumar al. 2020), with offline settings being of particular concern (Zhang et al. 2021a). These intrinsic risks highlight the need for specialized defensive strategies be developed, order to support RL deployments.
Defenses in RL share many of the same risks as defenses deployed for other Machine Learning paradigms, in that they can be cireumvented by a motivated attacker. By contrast, certified defenses offer theoretical guarantees of robustness against worst-case adversarial manipulations. Such ro- bustness guarantees are particularly desirable safety-critical domains and have been extensively explored in classification tasks (Lecuyer et al. [019 Salman et al., 2019 Cullen et al. 2022). How- ever the direct applicability of these techniques to RL is made more challenging due to the complex sequential dependency and interactive nature of RL. (Kiourti et al. 2020).
While some works Ye et al. 2023 Zhang et al. 202 Lykouris et al., 2023) have established robustness bounds for RL from theoretical perspective, they typically rely significant sim- plifications of the problem setting that do not reflect the complexities of real-world RL scenarios, limiting their applicability. Furthermore, these robustness bounds are typically expressed in terms

---

Published as a conference paper at ICLR 2025
of the optimality gap to the practically unattainable Bellman optimal policy, offering qualitative insights rather than quantitative certification value of the robustness.
To circumvent these limitations, recent research has begun to consider how practical certifications can be constructed, resulting in the COPA approach (Wu et al. 2022). This approach provides com- putable lower bounds cumulative reward and certified radii, ensure policies
robust against poisoning attacks. However, despite its utility, COPA is fundamentally limited to discrete action spaces and deterministic settings, which constrains it small subset of potential RL environ- ments. Additionally, it certifies only individual trajectories without offering robustness guarantees for the overall performance of the learned policy.
To resolve these limitations, within this work, we propose the Multi-level Certified Defenses (MuCD) against poisoning attacks in general offline RL settings, offering multi-level robustness guarantees across different levels of poisoning. To assist in this, we distinguish between adversar- ial attacks against RL that involve trajectory-level poisoning, which occur during the data collec- tion process; and fransition-level poisoning, which occurs after the training dataset is collected. In response to the existence these threat models, we propose employing certifications that employ action-level robustness (expanding upon Wu et al. 2022) to ensure that critical states are safeguarded against being entered; and policy-level robustness, which provides a lower bound on the expected cumulative reward. This latter framework naturally aligns with RL policy's ' primary goal (Prudencio et al. 2024). To achieve these certifications, our framework comprises two stages: a Differential Privacy (DP) based randomized training process and robustness certification methods. These certi- fications are broadly applicable to RL, covering both discrete and continuous action spaces, as well as deterministic and stochastic environments.
Our contributions on both a theoretical and empirical level are:
Formulating both multi-level attacks and certifications for poisoning attacks in offline RI., abling comprehensive analysis of the robustness of the offline RL training process.
Proposing the first practical certified defense framework that provides computable robustness cer- tification in terms of both per-state action stability and expected cumulative reward in general offline RL settings.
Experimentally demonstrating significant improvements over past certification frameworks across varying environments and RL algorithms.
2 RELATED WORKS
Offline and Online RL. RL approaches can be broadly categorized as online or offline learning. Of these, online learning algorithms involve agents learning by interacting with the environment in real-time, driving advances in a range of fields (Silver et al. 2017 Schulman et al., 2017 Kendall 2019). While frameworks like Policy Gradient (Sutton et 1999) and Actor- Critic (Mnih) 2016 can effectively learn policies for online RL, their trial-and-error exploration may lead to un- intended harmful outcomes and unsafe decisions during the learning process, which of particular concern to safety-critical areas such healthcare (Yu et al. (222I) and finance (Nevmy et al. 2006). By contrast, offline (or batch) RL is considered safer, as it learns to emulate pre-collected data to create optimal policies without interacting with the environment (Lange et al. 2012). Algo- rithms such as Deep Q-Network (DQN) (Mnih et al.) 2013)) Implicit Q-Learning (IQL) (Kostrikov let al. 2021) and C51 (Bellemare al. 2017) have demonstrated effective in leveraging historical data to optimize decision-making without further exploration.
al.
Poisoning Attacks in Offline RL. Adversarial attacks are well-documented threat to machine learning, where motivated adversaries manipulate models to induce unexpected behaviors. Among these, poisoning attacks (Barreno et al., (2066; Biggio et al. 2013)- deliberately corrupt the training data to degrade the performance of learned models are particularly concerning for offline RL, due to its reliance pre-collected datasets and the complex dynamics of RL frameworks (Ku- mar et al. 2020 Kiourti et al. (220)) Adversaries can target specific components of the data, such as in reward poisoning attacks (Wu et al. 2023), or broadly corrupt the entire dataset in general poi- soning attacks (Wang et al. 2021). Corruption can occur after data has been collected cleanly (Zhang
et al. 2021a) during data collection (Ye et al. 2023
Gong et al. 2024
2

---

Published as a conference paper at ICLR 2025
Certified Defenses. In response to these attacks, a range of defensive mechanisms have been pro- posed. Of these, certified defenses have drawn particular interest, due to their ability to provide ro- bustness guarantees against attack existence for classification tasks (Peri et al. 2020 Lecuyer al.) 2019 Liu et al. 2021 Cullen et al. 2024b), however applying these methods to RI. directly has proven challenging due to RL's sequential dependency (Kiourti al.] [2020). While some research has addressed certified robust RL in the context of reward poisoning (Banihashem et al. 2021 Nika ct al. 2023), extensions to general poisoning attacks have been more limited, with results primarily restricted to theoretical analyses of simplified variants (using linear MDPs or assuming bounded distances to Bellman optimality) of offline RL under attack occurring during data collection (Ye) et al. (023) or afterward (Yang et al., 2024 Zhang et al.) 2021a). Crucially, these robustness ap- proaches typically produce bounds expressed asymptotic measures of the optimality gap between the learned policy and the theoretical optimal policy, which have limited practical applicability.
By contrast, COPA (Wu al. 2022) recently demonstrated that per-state certification for RL, could be computed by adapting the Deep Partition Aggregation (DPA) (Levine & Feizi| (021) method from classification tasks. Based on that, they proposed a tree-search approach that exhaustively explores all possible trajectories to compute lower bound the cumulative reward. However, such an approach intrinsically limits it to discrete action spaces and deterministic environments. Consequently, their certification framework only applies to specifie, repeatable trajectories and fails to provide robustness guarantees for the reward or policy in more general scenarios.
3 PRELIMINARIES
In this section, we formulate the offiine RL framework episodic finite-horizon Markov De- cision Process (MDP), establishing the foundation for our discussion. We then outline the dataset construction process and introduce a comprehensive multi-level poisoning attack model address potential risks in offline RL training, along with the objectives for certified defense. Finally, we highlight key concepts from Differential Privacy (DP) that underpin our approach.
3.1 MULTI-LEVEL POISONING
Framework. The RI. framework is modeled as an episodic finite-horizon MDP, represented by the tuple (S, A, P, R. H,)), where S is the state space, A is the action space, P: S x A - (S) is the stochastic transition function with A(-) defining the set of probability measures, R S x A R is the bounded reward function, is the time horizon, and y E R is the discount factor.
At time step I, an RL agent in state 8, S selects an action d, T(x)) according to its policy T E :I: + A. Upon executing a,, the agent transitions to the subsequent state $t41 ~ P(s at) and receives a reward Tt R(st, a.). The tuple (st, ag $t+1,7() rr) is referred to as transition, and over one episode constitutes trajectory
the sequence of transitions {(st, at, 8141, re) r+) H
Offline RL Datasets. Offline RL employs dataset D = 1 consisting of M trajectories, or equivalently N transitions D {s); 0i 8/+ 1141, H collected by an unknown behavioral policy TH. The agent learns its policy from this dataset without further interaction with the environment, which provides opportunities for an adversary to poison the training data during the behavioral policy execution or after the collection process.
Trajectory-level Poisoning. Trajectory-level poisoning occurs when the adversary corrupts the data collection process. The adversary, with full knowledge of the MDP, can observe all the histor- ical transitions and alter transitions {(st, ag s(+), at arbitrarily many time steps by replacing them with {(5t. âz, Pt)}, As any alteration can affect subsequent transitions and propagate through gameplay, the following definition quantifies corruption by the number modified trajec- tories, following the adversarial models in robust statistics (Diakonikolas et al. 2019) and robust RL (Zhang et al. 2021b Wu et al. 2022) settings.
Definition 3.1 (Trajectory-level poisoning). Assume an adversary can make up to changes, in- cluding additions, deletions, or alterations, to the trajectories within a clean datasel = {T; {
Then the set of all possible poisoned datasets is Biej (D,r) == D ID Stri D < r). where ID Stra D) measures the minimum number of changes to the trajectories required map D to D.
3

---

Published as a conference paper at ICLR 2025
Transition-level Poisoning. In transition-level poisoning, the adversary modifies transitions after the data is collected. Therefore, alterations are limited to the specific transitions {(5;, ae, 1, F=) that were directly modified, without impacting any subsequent transitions across the trajectory. Hence, we quantify the corruption by the total number of modified transitions through the following definition.
Definition 3.2 (Transition-level poisoning). Assume adversary can make up r changes, including additions, deletions, or alterations, to the transitions within a clean dataset D Then the set of all possible poisoned datasets is Bera = {D
{(ss, 04 F::):::))
ID Stra DI S r}. where ID Stra DI measures the minimum number of changes to the transitions required to map D to D.
3.2 MULTI-LEVEL ROBUSTNESS CERTIFICATION
To ensure robustness against data poisoning in RL, we aim certify test-time performance of policy т = MMD) trained on a clean dataset D with the training algorithm M. In doing so, we will bound the difference between π and the equivalent म = MMDD trained upon a poisoned dataset ń, subject to constraint on the difference between D and D.
Policy-level Robustness. Offline RL. algorithms aim to find an optimal policy that maximizes the expected cumulative reward for all trajectories induced by the policy (Prudencio et al. 2024). Thus we first aim to construct certifications regarding the expected cumulative reward. The expected cumulative reward is denoted as ( E., E'r"r x]. where & represents the randomness of the environment and a represents the randomness introduced by the training algorithm. The following definition demonstrates how policy-level robustness certifications can construct lower bound on the expected cumulative reward, henceforth labelled as J,, under poisoning attack of size r.
Definition 3.3 (Policy-level robustness certification). Given clean dataset D, a policy-level certifi- cation ensures that a policy M(D) trained on any poisoned dataset D E B(D,r) will produce an expected cumulative reward J ( 2 J. with probability at least 1 - 8.
Action-level Robustness. Beyond ensuring generalised robustness, it is crucial to be able to guar- antee the safety of the agent by ensuring avoids catastrophic outcomes and entering undesirable also aim to certify the stability of the agent's 's actions on
states (Gu et al. 2024). Therefore, per-state basis during testing. The action-level robustness certification in discrete action space at state s, under poisoning attack of size r is defined in the following definition.
Definition 3.4 (Action-level robustness certification). Given a clean dataset D and state su the action-level robustness certification states that for any poisoned dataset B(D,r) r). the clean and poisoned policies produce the same action π (Tx) = (ss) where ग = MM(D and ã = M(D), with probability at least 5.
3.3 DIFFERENTIAL PRIVACY
DP quantifies privacy loss when releasing aggregate statistics or trained models on sensitive data (Dwork et al.) 2006] Abadi et al.) 2016); Friedman & Schuster 2010). As DP can be used to measure the sensitivity outputs to input perturbations, it well aligned use in certifications, leading to it being employed in multiple works (Lecuyer et al. 2019 Ma et al. 2019 Cullen et al.) 2024a). The remainder of this section will introduce key properties of DP as employed by our work, with more detailed explanations of the Approximate-DD DP (ADP) and Rényi-DP (RDP) mechanisms deferred to Appendix I
Our work relies upon two key principles of DP-the post-processing property, that any computation applied to the output of a DP algorithm preserves the same DP guarantee (Dwork et al. 2006); and the outcomes guarantee (Liu et al. 2023 Mironov] 2017), explained in the following definition
specifically for ADP and RDP.
Definition 3.5 (Outcomes guarantee for ADP and RDP). A randomised function M is said to pre- serve (KC, ()C outcomes guarantee if for any function K E K. such that for all datasets Dr and
4

---

Published as a conference paper at ICLR 2025
D2 B(D1,r), and for all measurable output sets S S Range(M) if Pr[M((D)) SI ((P)(M((2)) S()
(1)
In ADP, the function family K. is parameterized by (e,((x) E¹ 8, while in RDP K. is a=1
parameterized by c, a as (x)) = (e
APPROACH
Our novel certified defense employs DP-based randomized training process and provides two unique certification methods to construct both action-level and policy-level robustness certification against transition and trajectory level poisoning attacks.
4.1 RANDOMIZED TRAINING PROCESS
Our certification requires the training algorithm M to ensure the DP guarantee of its output policy π with respect to the training dataset D. DP mechanisms introduce randomness into the training process by adding calibrated noise in updating the parameters, producing randomized policy л. Empirically, this can be represented as a set of policy instances (f1.... Rp). As each instance undergoes the same training process, this be easily parallelized for efficiency. For larger datasets,
further efficiency gains can be achieved by training each instance on a subset Denb C D.
Our specific approach employs the Sampled Gaussian Mechanism (SGM) (Mironov al. 2019 to ensure DP guarantee at the transition-level Bira. and adapts the DP-FEDAVG (McMahan et al., 2017) for the trajectory-level Birj DP guarantee. Details of the training algorithms deferred to Appendix A.2 For the remainder of this paper, B will represent either Br or Btrj, depending on whether the applied DP training algorithm provides transition- or trajectory-level guarantees.
4.2 POLICY-LEVEL ROBUSTNESS CERTIFICATION
Consider DP training algorithm M (as described in Section 4.1 that preserves (K,, outcomess guarantee for the clean dataset D, producing the clean policy . = M(D). When the dataset poisoned as D, the resulting policy is denoted as = M(D). To certify the policy-level robustness as in Definition 3.3 ji terms of the lower bound of expected cumulative reward, denote the testing time expected cumulative reward of a policy æ as expressed by
(TT E [C(=]] where C(τ)
= E E
Σ
r
that represents the training randomness, and represents the environment randomness.
To provide bounds over the expected output of the DP mechanisms, we propose the following lemma that extends the outcomes guarantee from probability to the expected value.
Lemma 4.1 (Expected Outcomes Guarantee for ADP and RDP). If an M that produces bounded outputs in [0, b), E R+ satisfies (I., -ouccomes guarantee, then for any H E B(D,y) r) the expected value the outputs of the M must satisfy: denotes the function family of ADP K.e,b.
(B(M((D) bō) s B[M(D)) S <EE[M()) b5
(3)
Similarly, if K. denotes the function family RDP K...o.
e ( <(<bV\/
E[M( < ( ((E((M(( (a-
(4)
where the expectation is taken over the randommess in M.
Proof. While full proof is contained within Appendix A.3 here we present an informative sketch. The upper bound of the expected value can be obtained by integrating over the right-tail distribution function of the probabilities in Equation T by Fubini's Theorem (Fubini 1907). The integral results of ADP and RDP be derived by respectively employing Lecuyer et al. (2019) and Hölder's Inequality. The lower bound follows by the symmetry in the roles of D,, D, in DP, and by K being strictly monotonic.
5

---

Published as a conference paper at ICLR 2025
With this preliminary result, now turn to the main result of this section, which is establish that DP learning algorithms ensure pele robustness against poisoning attacks up to size r, with an extension to real-valued cumulative rewards deferred to Appendix A.4
Theorem 4.2 (Policy-level robustness by outcomes guarantee). Consider an RL environment with bounded cumulative reward in the range [0, b), E Rt, as well as a randomized offline RL policy M(D) constructed by the learning algorithm M using training dataset D. If M preserves ADP (K. -ootcomms guarantee, then each E KKe.5 with corresponding c, satisfies the policy-
level robustness of size for any poisoned dataset D E rr) as
2 ( (J(x) - b5)
(5)
If M preserves RDP (K, outtcomes guarantee, then each K E Ke with corresponding a satisfies the policy-level robustness of size r as
J( 2 (R) = -b
(6)
Proof. This result is direct consequence of the (K,, outcomess guarantee and the post-processing property, as (( = (D) a post-computation applied to the output of the DP mechanism M. hence it satisfies the same (K,, -o-utcomsssss guarantee by the post-processing property. By Lemma the expected value J(ñ) [*] and J(T) B[C(=)] satisfy the inequality in Equation and Equation (6) for ADP and RDP respectively.
To compute the policy-level robustness using Theorem 4. we need to obtain the lower bound of (Tx) to substitute into the Equation (5) and Equation 6 Consider the cumulative reward of the
policy и random variable X Et r'Tr where J ((T)) Eo,g [X]. The estimations of X can be obtained by playing the games m times using the trained policy instances (#1, Ap). and obtain its empirical Cumulative Distribution Function (CDF) Fx(z). By the Dvoretzky-Kal Kieyy-KKeffr-ooffooe inequality (Dvoretzky et al. 1956), the true CDF Fx Fx(x) must be bounded by an empirical CDF Fx (I)) a finite sample size m with probability at least as
$ Fx where ==
2m
The expected value of the random variable X in the bounded range [0, b] can be expressed by the true CDF (x) and bounded by the empirical CDF Fx) as
(Fx(()) - ((=) = E = 1 (1- de > ( (1
(8)
of
and thus allows us to construct the lower bound (T),, as required for policy-level certification.
4.3 ACTION-LEVEL ROBUSTNESS CERTIFICATION
In this section, we propose a method for certifying action-level robustness as in Definition 3.4 in terms of the stability of output actions. To achieve this, we begin by considering the decision- making process of policy π given state 8, in a discrete action space each instance मं; of the policy, the action az is selected based on the highest action-value 04,i
{AI..... Ar).. For
arg maX Q+, (st,
Ex, Σt''' 7'r, so = st, dg a.) Without loss of generality, we denote the action a; chosen by the randomized policy æ as the one with the highest inferred scores IA (st, R), where EAEA Lai (st, x) = 1 and LA (s., E [0, 1). The inferred score function of the randomized policy x takes the form
(s m) Pr(arg max Qu (st, a.) Ai]
a,
indicating that the action is induced the most likely one, which can be estimated unbiasedly by using a majority vote among all policy instances. We then propose the following lemma, which extends the outcome guarantees inferred scores.

---

Published as a conference paper at ICLR 2025
Lemma 4.3 (Inferred scores outcomes guarantee). If M preserves a (K, ( outcomes guarantee
for a dataset D, and there exist an that maps the learned policy т = M(D) and a state š to an inferred score, then for any K K, it must hold that for any action A E A and any policy f = MM()) trained with dataset D E B(D,r):
K (8A, m)) S LA, (8,, S (STA
(10)
The composition o .M preserves the same outcomes guaranteed by the post-processing property. The reverse inequality is derived from the symmetry in the roles of the datasets in DP, with K' being strictly monotonic. The outcomes guarantee can be directly converted Equation (T)) by defining S = { arg max,, Qx (8,, 04) = Ar)-
Proof.
With the bounds over inferred scores by outcomes guarantee, present the theorem that speci- fies the conditions under which a DP learning algorithm maintains action-level robustness against poisoning attacks of size in any arbitrary state 8t.
Theorem 4.4 (Action-level robustness by outcomes guarantee). Consider an offtine RL training dataset D, a randomized learning algorithm M that satisfies a (K., -outcomess guarantee and oulputs a policy time input state with the corresponding output action at = arg max La (8t, T). ). If there exist K1, K2 E K. such that:
M(D). Let be the inferred score function, and 8t be an arbitrary test-
(sr,
max (.EA((a)
(st,
(1I)
then the algorithm preserves action-level robustness state 8t under a poisoning attack of size
Proof. As the policy selects the action that maximises the inferred score, the objective of certifying action-level robustness equivalent proving that the inferred score of ag larger than the inferred score of any other actions aj E AAY {ar} for any poisoned dataset D B(D,r).
VD B(D,r)
(12)
Ia. (s,, M(D)) > max Ia, (s,,, ((.,)(()) ((E))((a))
Given M preserves a (K,, outcomes guarantee, then for any K1, K2 E K. the following inequali- ties can be derived by Lemma 4.3
(s, M(D)) (s,, max Ia, (st, M(D)) a.EAi{ar)
(13)
max a.EAA
Therefore, if there exists K2 and K₂ that satisfy the condition in Equation 11 the transitive prop- erty of inequalities ensures that the condition in Equation (12) is also satisfied.
The maximum tolerable poisoning size can be calculated while maintaining action-level robust- ness by way of the policy instances (# , #p)) and the condition in Theorem 4.4 each time, the selected action d, is that with the highest inferred score across the policy instances, with upper and lower bounds estimated simultancously through sampling the outputs the policy instances to confidence level of at least 1 - å by the SIMUEM method (Jia et al. 2020). We substitute the lower bound of Ja, (St, π) and the upper bound of max, EAV (a)(()) Ja., (8c, T into the condition of Equation [T]. By Theorem 4.4 we can then certify whether action-level robustness is achieved state under poisoning size r. The maximum tolerable poisoning size rı is determined through
binary search over the domain of K. find the Ki and Ka that satisfies the condition for maximising r. We defer the details of this process to Appendix A.5
5 EXPERIMENTS
In this section, we evaluate our proposed certified defenses under scenarios of either transition- or trajectory-level poisoning (Section 3.1 for policy-level and action-level robustness (Section 3.2 To
7

---

Published as a conference paper at ICLR 2025
facilitate these, conducted evaluations using Farama Gymnasium Towers et al.
2023) discrete
Atari games Freeway and Breakout, as well as the continuous action space Mujoco game Half Chee- tah. We also employed the D4RL (Fu et al. (020)) dataset and the Opacus (Yousefpour et al. 2021
DP framework. Our environments were trained using DQN (Mnih et al., 2013), IQL (Kostrikov al. 202T) and C51 (Bellemare et al. 2017), implemented with Convolutional Neural Networks
(CNN) in PyTorch on a NVIDIA 80GB A100 GPU. Further results considering the robustness of these defenses to empirical attacks are presented within Appendix A.7
Our offline RL datasets consist of 2 million transitions for each game, with corresponding trajectory counts of 976 for Freeway, 3, 648 for Breakout, and 2, 000 for Half Cheetah. In all experiments, the sample rates in the DP training algorithms were adjusted to achieve batch size of 32, with varying the results. Uncertainties were estimated within confidence
noise multipliers as detailed interval suitable for 8 = 0.001. For each game, the number of policy instances as described in Section 4.1 set to 50. The number of estimations of expected cumulative reward m is set 500, with 10 estimations per policy instance, detailed in Section
5.1
ACTION-LEVEL, ROBUSTNESS RESULTS
Freeway (Transition)
Freeway (Trajectory)
Breakout (Transition)
Breakout (Trajectory)
1.00
1.00
1.00
1.00
Stability Ratio
DQN
0.50
0,50
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.00
100
200
150
150
Stabilty Ratio
1.000
1.00
0.75
0.75
0.73
0.75
CS1
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.00
200
150
150
50
100
150
Threshold
Threshold
Threshold
Threshold
Figure 1: Stability ratio against the tolerable poisoning threshold for action-level robustness using DQN and C51 for the Freeway and Breakout environments under transition- or trajectory-level poi- soning attacks. Blue, Green and Red lines represent different noise levels during the randomized training process as a {1, 2, 3} for Freeway and {1, 1.5, 2} for Breakout, while the yellow dashed line denotes COPA, which can only be calculated for trajectory- poisoning.
Avg. Cumulative Reward
Action-level Mean Radii
Environment
Method
Noise
DQN
DQN
C51
Transition
Trajectory
Transition
Trajectory
0.0
20.1
21.3
N/A
N/A
N/A
16.9
16.1
128.1
32.6
111.6
22.1
Proposed (RDP)
Freeway
2.0
16.6
15.3
145.5
58.7
119.3
37.8
3.0
16.0
15.1
160.0
102.4
134.5
49.7
COPA
N/A
16.4
16.4
N/A
10.1
N/A
9.7
0.0
385,4
389.3
N/A
N/A
N/A
N/A
1.0
366.6
369.0
3.4
3.2
4.0
3,9
Proposed (RDP)
Breakout
1.5
320.8
270.4
7.9
7.6
9.5
9.3
2.0
268.4
102.7
17.7
16.9
22.0
21.7
COPA
N/A
325.7
330.1
N/A
6.6
N/A
Table 1: Testing time average cumulative reward in a clean environment and the mean of maximum tolerable poisoning size r; of the action-level robustness.
We will evaluate action-level robustness across varying RL algorithms and environments for RDP, with additional ADP based experiments and statistics provided in Appendix AA6]] This analysis considers the mean and maximum value of rt across set of evaluated trajectories well the
8

---

Published as a conference paper at ICLR 2025
stability ratio (Wu al. 2022) This latter metric represents the proportion of time steps in trajectory where the maximum tolerable poisoning size rt of action-level robustness is maintained under poisoning attack of size up to given threshold F for trajectory length H by way of
1 Σ 1m
Stability Ratio
(14)
H
To interpret our results, it is important to emphasise that a higher stability ratio at larger thresholds signifies better certified robustness. Models trained with higher noise achieve stronger certified robustness, at the cost of clean performance decreasing in terms of the average cumulative rewards, as shown in Table [] In concert with Figure i it clear that DQN produces a consistently higher certifications than C51 in Freeway, while matching performance in Breakout. These differences arise from DQNs ability to adapt to noisey training process, allowing it to ameliorate the impact of these perturbations without significant drop in performance. It is also important to note that the Freeway consistently shows better action-level robustness than Breakout, suggesting that Freeway supports more stable and robust policies.
Given the effectively interchangeable action-level robustness of the variants as reported in COPA (Wu et al. 2022), our comparisons constructed against the basic PARL. variant in the same setting, with the number of partitions set to 50. For a fair comparison, the models trained at a noise level represented by the green line in each game achieve comparable performance to COPA in terms of Avg. Cumulative Reward, as shown in Table | Our technique's 's maximum tolerable poi- soning size indicated by the x-axis intersection Figure is approximately 5 times larger than other approaches, which confirms that our approach produces stronger robustness in states where certain actions are highly preferable, typically during critical moments. Additionally, our method achieves higher mean value of the tolerable poisoning size across all steps as shown in Table 1 demonstrating better certification for most states.
5.2 POLICY-LEVEL ROBUSTNESS RESULTS
Freeway (Transition)
Freeway (Trajectory)
Breakout (Transition)
Breakout (Trajectory)
15
300
DON
10
200
200
5
100
100
0
500
1000
50
100
150
500
1000
1500
100
15
300
300
10
200
200
100
500
1000
1500
50
100
150
500
1000
1500
Poisoning sizer
Poisoning size r
Poisoning size
Poisoning size
Figure 2: Policy-level robustness certifications, capturing the lower bound of the expected cumula- tive reward J, against poisoning size r for Atari games. Solid and dashed lines represent RDP and ADP derived guarantees respectively, with colors indicating noise levels as per Figure
To assess policy-level robustness, we turn to the measure (T), which directly reflects the policy- level robustness of the learned policy against a poisoning attack of size (as in Definition 3.3 with Figure [ showing this for the discrete games Freeway and Breakout. We also consider the continuous game Half Cheetah as shown in Figure 3 where the benign training yielded an average cumulative reward of 96.47 and randomized training with noise levels set at 3.0 yielded average cumulative rewards of 90.5, 87.0, and 83.4, respectively. The results can be compared from different aspects. In terms of the poisoning level, for the same policy certification certification is about 10 times larger than in
1.0, 2.0, and
J, the tolerable poisoning size r in transition- trajectory-level. Given that each trajectory consists of approximately 1, 000 transitions, the total

---

Published as a conference paper at ICLR 2025
HalfChectah (Transition)
HalfCheetah (Trajectory)
75
75
50
50
101
--
m000
1000
1500
1000
1500
Poisoning size
Poisoning size r
Figure 3: Policy-level robustness certification for the continuous action game Mujoco Half Cheetah, using RI. algorithm IQI.. The plot is formulated in the same way as Figure 2
number of transitions that can be altered in trajectory-level certification is actually higher than in transition-level. This aligns with the nature of the threat models: in trajectory-leve poisoning, not all transition changes are assumed adversarial, whereas in transition- poisoning, the adversary can more precisely target key transitions to minimize the number of modifications.
In analyzing the influence of RI. algorithms and certification methods, DQN consistently demon- strates higher robustness certification than C51, which is consistent with the analysis from the action- level certification. RDP's tight quantification of privacy loss, particularly in handling iterative func- tion composition in deep networks, provides significant advantage over ADP in all settings.
Lastly, our approach accommodates more general RL settings, that it can be applied to both discrete and continuous action spaces, well as deterministic and stochastic environments, and significantly improves upon the performance of extant techniques, namely COPA. As COPA cer- tifies the cumulative reward for specific trajectories rather than the expected cumulative reward of the policy, our comparison with COPA is often implicit. The limited applicable scenarios of COPA stem from its reliance upon exhaustive tree search in certifying cumulative reward, which is funda- mentally incompatible with environments involving randomness or continuous action spaces, and limiting the trajectory length to 400 in Freeway and 75 in Breakout due the exponential growth of the tree size, while the default trajectory lengths are 2, 000 and 600, respectively. As a result, the maximum cumulative reward COPA can certify is restricted to 5 in Freeway and 2 in Breakout. By contrast, our approach has no such limitations regarding environment settings or trajectory length. Furthermore, for Freeway, COPA only allows 0.008% of trajectories in the training dataset to be poisoned while certifying less than a 50% performance drop, whereas our method achieves much higher ratio of 7. 7.17%. We observe similar delta relative performance within the Breakout games, where COPA's ratio of 0.0075% is significantly smaller than 2.05% observed for our approach.
6 CONCLUSIONS
This work explored how certified defenses against poisoning attacks can be both constructed and enhanced in offline RL. To do this, we introduced a novel framework that leverages Differential Privacy mechanisms provide the first practical certified defense in general offline RL setting. While past works have only considered theoretical robustness bounds or are limited to specific RL settings, framework is able to offer both action-level stability and policy-level lower bounds with respect to the expected cumulative reward of the learned policy. Furthermore, our experiments across a wide range of RL environments and algorithms demonstrate robust certifications in practical applications, significantly outperforming other state-of-the- defense frameworks.
Our work suggests several potential directions for future research. First, developing unified DP training algorithm that simultaneously supports both transition- and trajectory- certifidd de- fenses could significantly enhance robustness. Additionally, defense performance may be further improved by designing more sophisticated noise injection mechanism that adapts noise levels dy- namically, rather than uniform noise throughout the entire training process.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This research was supported by The University of Melbourne's Research Computing Services and the Petascale Campus Initiative. This work was also supported part by the Australian Department of Defence through the Defence Science and Technology Group (DSTG). Sarah Erfani is in part supported by the Australian Research Council (ARC) Discovery Early Career Researcher Award (DECRA) DE220100680.
REFERENCES
Martín Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Tal- war, and Li Zhang. Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 308-318, October 2016. doi: 10. 1,114/22976749.297888888 URL 1607.00133.
1607 0000///133000033333333 arXiv:
Borja Balle, Maziar Gomrokchi, and Doina Precup. Differentially private policy evaluation. In International Conference on Machine Learning, pp. 2130-210111 2138.. PMLR, 2016.
Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis Testing Interpretations and Renyi Differential Privacy, arXiv:I905, 09982 (cs, stat], October 2019. URL
http://arxiv. org/ abs 09982
arXiv: 1905.09982.
Kiarash Banihashem, Adish Singla, and Goran Radanovic. Defense Against Reward Poisoning Attacks in Reinforcement Learning, June 2021. URL http::/aax.v..rr/abbb/////
05776 arXiv:2102.05776 [cs].
Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and Doug Tygar. Can Machine Learning be Secure? In Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security, pp. 16-25, 2006.
Marc G Bellemare, Will Dabney, and Rémi Munos. A Distributional Perspective on Reinforcement Learning. In International Conference on Machine Learning, pp. 449-458. PMLR, 2017.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Ma- chines. arXiv:/206. /206.639066 [cs, stat], March 2013. URL http:::/aax.w..rr//abb/////// 6389
arXiv: 1206.6389.
Andrew Cullen, Paul Montague, Shijie Liu, Sarah Erfani, and Benjamin Rubinstein. Double Bubble, Toil and Trouble: Enhaneing Certified Robustness Through Transitivity. Advances in Neural Information Processing Systems, 15:19999-19112 2022.
Andrew C. Cullen, Paul Montague, Shijie Liu, Sarah M. Erfani, and Benjamin I. P. Rubinstein. It's Simplex! Disaggregating Measures to Improve Certified Robustness. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 2886-2900, 2024a. doi: 10. 1099//9/99 20..0000055..
Andrew Craig Cullen, Shijie Liu, Paul Montague, Sarah Monazam Erfani, and Benjamin IP Ru- binstein. Et Tu Certifications: Robustness Certificates Yield Better Adversarial Examples. In International Conference on Machine Learning, 2024b.
Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jemyy Li, Ankur Moitra, and Alistair Stewart. Robust Estimators in High Dimensions without the Computational Intractability, March 2019.
URL http:htt:///axxt org. haagg/abs/1ie 604. 06443) arXiv: 1604.06443 [cs, math, stat].
Aryeh Dvoretzky, Jack Kiefer, and Jacob Wolfowitz. Asymptotic Minimax Character of the Sample Distribution Function and the Classical Multinomial Estimator. The Annals of Mathematical Statisties, 27(3):642 1956.

---

Published as a conference paper at ICLR 2025
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Theory of cryptography conference, pp. 265 284.. Springer, 2006.
Arik Friedman and Assaf Schuster. Data Mining with Differential Privacy. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 493- 502, 2010.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, 2020.
Guido Fubini. Sugli integrali multipli. Rendiconti del Circolo Matematico di Palermo, 24(1):97- 155, 1907. doi: 10.1007/BF0301432)
Evrard Garcelon, Vianney Perchet, Ciara Pike-Burke, and Matteo Pirotta. Local Dif- ferential Privacy for Regret Minimization in Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 34, pp. 10561-10573. Curran Asso- neurips wtt.cc//ppppeee cc//paper/222//hast/ ciates, Inc., 2021. URL. -ttpeeeea.etllcbbbbboottttattttd Abstract.htm. html
Quan Geng and Pramod Viswanath. The optimal noise-adding mechanism in differential privacy. IEEE Transactions on Information Theory, 62(2):925-955 2015.
Chen Gong, Zhou Yang, Yumpeng Bai, Junda He, Jieke Shi, Kecen Li, Arunesh Sinha, Bowen Xu, Xinwen Hou, David Lo, et al. Baffle: Hiding Backdoors in Offline Reinforcement Learning Datasets. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 2086-2104. IEEE, 2024.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems, 34: 16411663111142, 2021.
Shangding Gu, Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A Review of Safe Reinforcement Learning: Methods, Theory and Applications, May 2024. URL
http://arxiv. wwt.:oog//aaaaaoooooooooooe http:://arxxv..rrg/abs//222. 10330 arXiv:2205. 103302205.03300 [cs].
Nico Gürtler, Sebastian Blaes, Pavel Kolev, Felix Widmaier. Manuel Wüthrich, Stefan Bauer, Bern- hard Schölkopf, and Georg Martius. Benchmarking Offline Reinforcement Learning on Real- Robot Hardware. arXiv preprint arXiv:2307. /5690, 2023.
Jinyuan Jia, Xiaoyu Cao, and Neil Zhenqiang Gong. Intrinsic Certified Robustness of Bag- ging against Data Poisoning Attacks. arXiv:2008. 04495 [cs], December 2020. URL http:
04495
arXiv: 04080.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning Drive in Day. In 2019 international conference on robotics and automation (ICRA), pp. 8248-8254. IEEE, 2019.
Panagiota Kiourti, Kacper Wardega, Susmit Jha, and Wenchao Li. TrojDRL: Evaluation of Back- door Attacks on Deep Reinforcement Learning. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pp. 1-6, San Francisco, CA, USA, July 2020, IEEE. ISBN 978-1-72811- 085-1. doi: 10. 1111///AC187..2000022222221 |documett//// / 92 99218663711111
URL htt.attss//eeexllreeeeeeet wew.. /
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-learning. arXiv preprint arXiv:2110.06169, 2021.
Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial Machine Learning-Indee Per- spectives. In 2020 IEEE security and privacy workshops (SPW), pp. 69-75. IEEE, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch Reinforcement Learning. In Reinforce- ment learning: State-of-the-ar pp. 45-73. Springer, 2012.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified Robustness to Adversarial Examples with Differential Privacy. 0rXiv:/802.0344 2019. URL http: /tet:://arxie httt::::/axxii.orr/aaabaa /8002 03471
004.0 [cs, stat], May arXiv: 1802.03471.
12

---

Published as a conference paper at ICLR 2025
Alexander Levine and Soheil Feizi. Deep Partition Aggregation: Provable Defense against General Poisoning Attacks. arXiv:2006. 14768 [cs, stat]. March 2021. URL http::/arxiv. orq/
abs/2006.1477 14768 arXiv: 2006. 14768.
Chizhou Liu, Yunzhen Feng, Ranran Wang, and Bin Dong. Enhancing Certified Robustness via Smoothed Weighted Ensembling. NxXi:2205.09333 [cs, stat], February 2021. h //arx/v.org//bs//20000 09363
arXiv: 2005.09363.
Shijie Liu, Andrew C Cullee,, Paul Montague, Sarah M Erfani, and Benjamin IP Rubinstein. Enhanc- ing the Antidote: Improved Pointwise Certifications Against Poisoning Attacks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 8861-8869, 2023.
Thodoris Lykouris, Max Simchowitz, Alcksandrs Slivkins, and Wen Sun. Coruption-robust Explo- ration in Episodic Reinforcement Learning, October 2023. URL http://azziv.or///a.
1911. arXiv:1911.0860 [cs, stat].
Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data Poisoning against Differentially-Privaat Learners: Attacks and Defenses. arXiv: 1903.09860 [cs], July 2019. URL http:://rxxv..rrg// arXiv: 1903.09860.
1903. 09860)
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning Differentially Private Recurrent Language Models. arXiv preprint
2017.
Ilya Mironov. Renyi Differential Privacy. 2017 IEEE 30th Computer Security Foundations Sym- posium (CSF), pp. 263-275, August 2017. doi: 10. 10.110./15......11.. URL http::/arxix. arXiv: 1702.07476.
org/abs 0774
Ilya Mironov, Kunal Talwar, and Li Zhang. Renyi Differential Privacy of the Sampled Gaussian Mechanism. arXiv:1908. 10530 [cs, stat], August 2019. URL attp::/arxixs www.xiv.org/bb... / 19C8. 1 10530 arXiv: 1908.10530
Volodymyr Mnih. Asynchronous Methods for Deep Reinforcement Learning. arXiv preprint 1rXin:1602.01783, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning, December 2013. URL http::ttta/tiiitee org/ abs 1312. 5602, arXiv:1312.5602 [cs].
Yuriy Nevmyvaka, Yi Feng, and Michael Kearns. Reinforcement Learning for Optimized Trade Execution. In International Conference on Machine Learning, pp. 673-680, 2006.
Andi Nika, Adish Singla, and Goran Radanovic. Online Defense Strategies for Reinforcement Learning Against Adaptive Reward Poisoning. In 26th International Conference on Artificial Intelligence and Statistics, pp. 335-358. PMRL, 2023.
Hajime Ono and Tsubasa Takahashi. Locally private distributed reinforcement learning. arXiv
preprint arXiv:2001.11718, 2020.
Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntaek Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile Autonomous Driving Using End-to-end Deep Imitation Learning. arXiv preprint 1000v:1100007000 2017.
Nechar Peri, Neal Gupta, W Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi, Tom Goldstein, and John Dickerson. Deep K-NN Defense Against Clean-label Data Poisoning Attacks. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part 16. pp. 55-70. Springer, 2020.
Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Trans- actions Neural Networks and Learning Systems, pp. 1-0, 2024. ISSN 2162-237X, 2162- 2388. doi: 10. 1001100TNNN.SS20333550000 URL httts:://eeexplorr leee org/ #ocument/10078.0
13

---

Published as a conference paper at ICLR 2025
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers. Advances in neural information processing systems, 32, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv:1707.06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the Game of Go Without Human Knowledge. nature, 550(7676)3354-3333 2017.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Meth- ods for Reinforcement Learning with Function Approximation. Advances in neural information processing systems, 12, 1999.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez-Vicente, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymna- http:::ss///eeooo..o sium, March 2023. http::p::///eeoo.ooo//eoooooo wwtp::oor./ggooo.oooooooooo.
Giuseppe Vietri, Borja Balle, Akshay Krishnamurthy, and Steven Wu. Private reinforcement learn- ing with pac and regret guarantees. In International Conference on Machine Learning, pp. 9754- 9764. PMLR, 2020.
Lu Wang, Wei Zhang, Xiaofeng He, and Hongyuan Zha. Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendation. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2447-2456, 2018.
Yue Wang. Esha Sarkar, Wenqing Li, Michail Maniatakos, and Saif Eddin Jabari. Stop-and-Gop Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems. IEEE Transactions on Information Forensics and Security, 16:4772 4787, 2021. ISSN 1556-6013, 1556-6021. doi: 10.1 109/TIFS.2021.1112 2003. 07859
#XXiv:2003.0885 [physics, stat].
Fan Wu, Linyi Li, Chejian Xu, Huan Zhang, Bhavya Kailkhura, Krishnaram Kenthapadi, Ding Zhao, and Bo Li. COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks, March 2022. URL http:///axxxv.rrr//b/////00 arXiv:2203.0839 08899 [cs].
Young Wu, Jeremy McMahan, Xiaojin Zhu, and Qiaomin Xie. Reward Poisoning Attacks on Of- fline Multi-agent Reinforcement Learning. In Proceedings of the aaai conference on artificial intelligence, volume 37, pp. 10426-10434, 2023.
Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. To- wards Robust Offline Reinforcement Learning under Diverse Data Corruption, March 2024. URL http://arxe wttp:/b.//lbb....o.. 12955 1rXiv:2310.12900 [cs].
Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-Robut Of- fline Reinforcement Learning with General Function Approximation. Advances in Neural Information Processing Systems, 36:36208-36221, December 2023. URL
fttps://proccedins..nneil htt.://wwrppp....pp..ppaaaaaaaap
html
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj. Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly Differential Privacy Library PyTorch. arXiv preprint
2.Xi::2109.12298, 2021.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement Learning in Healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
14

---

Published as a conference paper at ICLR 2025
Xuezhou Zhang, Yiding Chen, Jerry Zhu, and Wen Sun. Corruption-Robut Robust Offline Reinforcement Learning, June 2021a. URL http:://arxx...orc//aa////////// [es].
arXiv:2106.06630
Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Robust Policy Gradient against
Strong Data Corruption, June 2021b. URL http://aaax...rrr//////0/...... 05800 arXiv:2102.05800 [es].
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADP AND RDP DEFINITIONS
Definition A.1 (Approximate-DPr A randomised function . is said to be (e, -)ppprxximat DP (ADP) if for all datasets D1 and D2 for which D2 E (.(1..) 1), and for all measurable output sets
S RM)ge(M):
Pr(M(DI) E S Pr(M((22 E S
(15)
where c > and 8 E [0, 1) are chosen parameters.
ADP with the privacy guarantee as expressed in Equation (15) is the most commonly used format in DP research. Smaller values of the privacy budget restrict the (multiplicative) influence of a participant joining dataset D2 to form D,, thereby limiting the probability of any subsequent privacy breach. The confidence parameter 8 relaxes this guarantee by allowing for the possibility that no bound is provided on privacy loss in the case of low-probability events.
To bound the residual risk from ADP, Rényi-DP (RDP) was introduced by Mironov 2017). Rényi- DP provide tighter quantification of privacy through sequences of function composition, as required when iteratively training deep net sensitive data, which leads to improved certifications in practice.
Definition A.2 (Rényi-DP), A randomised function M preserves (a, -éRényi-DD with a > 1,e O, for all datasets Dj and D2 E B(D1, 1); Da M S
(16)
where Da represents the Rényi divergence of finite order a + between two distributions P and Q defined over the same probability space X with densities p and q as
1 In / ((x) x
p(x)
(a(l))
dr
(17)
q(x)
The generalization to Definition 3.5 incorporates group privacy (Dwork et al. 2006 to extend DP
to adjacent datasets pairs datasets that differ in up data points B(D,,r), r)). The ADP's function family K. is derived directly from the Definition AA. while the RDP's family is obtained by applying Hölder's inequality to the integral of the density function in the Rényi divergence (Mironov 2017).
A.2 DP TRAINING ALGORITTMS
Several works have extended differential privacy to RL. by developing privacy-preserving algorithms that balance the trade-off between model performance and privacy guarantees. To tackle the distinct challenges in RL, such as the sequential dependency, multi-sourced data, notable contributions have been made, including techniques for regret minimization RL with privacy guarantee (Vietri et al.)
2020 Dann al. 2017 Garcelon et al. 2021), off- off-policy evaluation (Balle al. 2016), and distri- butional RL (Ono& & Takahashi 2020).
The proposed methods require the training algorithm M to preserve the DP guarantec of its output policy π regarding the training dataset D. Depending on the DP training mechanisms, the trained private policy in the context of offline RL can achieve either transition- trajectory-level DP guar- antees, meaning the B used in the aforementioned DP definitions can be Bi or Birj respectively.
While numerous differential privacy (DP) mecha- nisms have been proposed and extensively studied in machine learning (Abadi et al.) 2016 Mironov al. 22019) most rely on adding noise directly to the training samples. Instead, the Sampled Gaus- sian Mechanism (SGM) (Mironov et al. 2019) introduces randomness through both noise injection and sub-sampling, providing a better privacy cost. In SGM, each element of the training batch is sampled without replacement with uniform probability from the training dataset. Additionally, Gaussian noise is added to the gradients during each weight update step. The training algorithm is illustrated in Algorithm When applied to model M, SGM preserves ea,P) where is determined by the parameters (a, M. o)). This RDP guarantee can be further transformed into (c, -DP using the conversion method described by Balle et al. (2019).
Transition-level DP Training Method SGM.
16

---

Published as a conference paper at ICLR 2025
Algorithm Sampled Gaussian Mechanism (SGM) for Model M using Dataset D Require: Dataset D with n samples, sampling ratio 9, noise multiplier ø, number of iterations T,
learning rate
Ensure: Private model .M
1: Initialize model parameters 8g
2: for t= ...... do
3: Sample a mini-batch B, C D by selecting each element of D with probability q without
replacement
4: Compute gradients VL(0;-1; B,) with respect to the mini-batch
5: Clip gradients: VC max(1, "
VC
where C is the clipping threshold
6: Add Gaussian noise: V = VL + (0,
7: Update model parameters: 0, 8,-1 V
8: end for
9: return Differentially private model M with parameters 0r
Trajectory-level DP Training Method DP-FEDAVG. As demonstrated in the SGM, clipping per-sample gradients makes it unsuitable for trajectory-level DP, where the privacy cost needs to be accounted for on a per-trajectory basis. To address this, we utilize DP-FEDAVG (McMahan al., 2017), initially designed for client-level privacy in federated learning, which can be adapted to scenarios where training data is naturally segmented. such as trajectory data in offline RL. The subset B, of trajectories sampled
core idea of DP-FEDAVG is as follows: each iteration from the dataset D with probability without replacement. A single gradient (L(((-1:7)) is then computed and clipped with constant for each trajectory. An unbiased estimator of the average gradient of the subset is then calculated, with sensitivity bounded by the C' divided by the batch size. Finally, the Gaussian mechanism is applied with noise magnitude a, and the model updated using the noisy gradient. The details of the training algorithm shown in Algorithm 2
Algorithm Model Training with DP-FEDAVG
Require: Dataset D, sampling ratio q (0, 1), noise multiplier a, clipping norm C, local epochs E. batch size B, learning rate 1)
Ensure: Private model M
1: Initialize model parameters 0g
2: for each iteration E -- do
3: U, + (sample with replacement trajectories from D with probability 4)
4: for each trajectory ты: U, do
5:
Clone current model 0 +- 0, for each local epoch E E] do
6:
7:
1 +- (split r's data into size B batches)
8:
for each batch E B do
9:
0
(00;b)
10:
0 + Start + PerLayerClip(# stari C)
end for
12:
end for elipped A L
A 8stat
end for
ENCU, Allrre
15:
avg
gk
16:
avg
A. + +N (0,
CC ( qk
2
17:
0 t 82 + avg
18: end for
In addition to the DP training algorithms discussed above, it is worth highlighting the existence of more advanced DP algorithms capable of offering tighter privacy guarantees, enhanced computa- tional efficiency, or optimality analysis (Gopi et al.] 202T Geng Viswanath 2013). These ad- vancements enable more precise privacy bounds and contribute further reinforeing the robustness
17

---

Published as a conference paper at ICLR 2025
of the certification for our proposed defense mechanism. However, it is crucial to emphasize that the primary focus of our work is on establishing a general framework for integrating DP into certified defenses for offline RL. This framework is designed to be adaptable, allowing for the incorporation of more advanced DP mechanisms in future developments.
A.3 PROOF OF THE EXPECTED OUTCOMES GUARANTEE
Lemma A.1 (Expected Outcomes Guarantee for ADP and RDP). Suppose a randomized function M, with bounded output [0, bl.b b E Rt, satisfies -outcomes guarantee. Then for any D
[ B(D,r), r), if IC denotes the function family of ADP Ke,,o, the expected value of its outputs satisfies:
(BIM((D) b9) S B[M(D)) S (EE[M(D)) + bō
(18)
if K. denotes the function of RDP K...o the expected value of its outputs satisfies:
(((b-1//[MMMD))) S [[M(D)) & 12/0 (cT(MM()) (a-)/a
(19)
where the expectation taken over the randomness in M.
Proof. The expected value can be obtained by integrating over the right-tai -ail distribution function of the probabilities in Equation T by Fubini's Theorem (Fubini 1907) as
B[M(D)) = f-'' PH[M(D) > di
(20)
In the case of ADP, for any K E Ke, that parameterized by 6, we have
BIM(Di S S." ('PPP[M(D) + ELM((D) 68
(21)
In the case of RDP, for any K E Kr that parameterized by a, we have
BLM(D) J (e'PP(((D)) ()(a-1)/a dt
(22)
Recall Hölder's 's Inequality, which states that for real-valued functions and g. and real p,g > 1, such that 1/p + 1/q 1,
fflll - Ilflll
(23)
By Hölder's 's Inequality setting p and (0) (P( 1 1//a allows for us to state that
1)/a
B(M(D)) cda-1)/a / /a ("PH(((D)) fjdt)(a-1)/a
(24)
(((D))
= ((M())) 11///a
The alternative inequality follows by both K being strictly monotonic and symmetry in the roles of D1, D2 for DP.
A.4 POLICY-LEVEL ROBUSTNESS CERTIHCATION FOR REAL-VALUEE REWARD
To certify policy-level robustness in real-valued cumulative reward, we first extend the Lemma 4.1 to the expected value in real number.
Lemma A.2 (Real-valued Expected Outcomes Guarantee for ADP and RDP). Suppose a random- ized function M, with bounded output (a, b), a E R- R R+, satisfies (K,r)-outcomss guarantee. Then for any D E B(D,r), if IC denotes the family of ADP Ke,5, the expected value of its outputs satisfies:
E[M(D) 2 e (B[M([])] 68) - -(E ad) [[M(D) < 55 (B(M(D) 1EM a.)
(25)
(26)
18

---

Published as a conference paper at ICLR 2025
if K. denotes the function famity of RDP Ke,,s the expected value of its outputs satisfies:
B[M(D)) 60 - BLM(D)] S (
-))
(27)
(28)
where the expectation is taken over the randomness in M. (D)+ represents the expected value of all non-negative M(D), B[M(D)= represents the expected value of all negative M(D).
Proof. We extend the Fubini's Theorem from non-negative values to real values as:
[[ /
Pr[X 2 df
Pr[x S t dt
(29)
which can be derived as
X
X
Let X+
0
otherwise
(30)
-X if < 0
X =
otherwise
Then, have
X X + -| E[ - [[X
Pr[x+ 2 2dd0 ! Pr(x 2 fjdt
E[X+]=
(31)
EX | Pr[X- 2 t]dt =
000
Pr[X < -tldt = f Pr[[ s tldt
The expected value B[M(D)) in range [a with M satisfies routcomms guarantee, can be written as
B[M(D)] = / Pr[M(D) 2 tjdt- 1. Pr[M(D] u)du
/ Pr[M(D] E T)di [- Pr[M[D] Uddu
(32)
K K- E TT)dt -f. [((PP[M(D) e Ul)du
where K E K. For the cases of ADP and RDP, replace the K with corresponding function as outlined in Definition 3.5
Then the Theorem 4.2 can be extended to the case of real-valued expected cumulative reward as,
Theorem A.3 (Policy-level robustness by outcomes guarantee in real-value range). Consider an RL environment with bounded cumulative reward in the range [a, b), T - R+, offline RL training dataset D, and a learning algorithm M that takes the training dataset D and outputs the randomized policy x M(D). If M preserves (K, -outcomes guarantee in ADP, then for each K E Ke,s with corresponding 8 satisfies the policy-level robustness of size r for any poisoned dataset D E B(D.r) as
J( e b6)
(e' - a5)
(33)
If M preserves a (K, r -outcomes guarantee in RDP, then for each K E Ke with corresponding e, a satisfies the policy-level robustness of size as
J(
(34)
where J(x)+ denotes the expected value of all non-negative cumulative rewards, J(x)- denotes the expected value of all negative cumulative rewards.
Proof. The proof is similar to the proof of Theorem 4.2 instead replace the usage of Lemma Lemma A.2 for the real-valued expected outcomes guarantee.
19

---

Published as a conference paper at ICLR 2025
A.5 ADDITIONAL DETAILS ACTION-LEVEL CERTIFICATION
Here we provide additional details of the action-leved certification process. As discussed in Sec- tion 4.3 the inferred scores are estimated by the sampling from the policy instances (#1,... mp). Due to uncertainty, we obtain the upper and lower bounds of the inferred score with confidence interval of at least a via the SIMUEM method (Jia et al. 2020) based on the Clopper- Pearson method. Specifically, the SIMUEM directly estimates the upper and lower bounds of the inferred scores LA, (5c, T) Pr[arg maxo, (sc: a.) = Ai] based on the frequencies (n4, ,,g, of each Ai produced by the policy instances as
0 LA, Beta
nt, p- T4 n++1 +1)
L
7 A: Bela
L Iin, 1, 1.p-n)). Vi Vi.
To determine the maximum tolerable poisoning size r, for a given state $1, a binary search is per- formed the domain of K. As described Section 3.3 and Appendix [A.2] X. represents the of (6,e) or (a,e) pairs that satisfy the privacy guarantees of the respective DP training algorithm. The binary search operates within predefined range, such (0,500) (00). aiming identify the largest radius that meets the condition specified in Theorem 4.4 provided there exist K₁ and K2 within the domain of K.
A.6 ADDITIONAL RESULTS OF ACTION-LEVEL CERTIFICATION
The Figure and Table show additional experimental results of action-level robustness certification and training statics.
Freeway (Transition)
Freeway (Trujectory)
Breakout (Transition)
Breakout (Trujectory)
1.00
1.00
1.50
DQN
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
1.00
1.000
0.75
C51
0.50
0.50
0.50
0.50
0.25
0.00
0.00
0.00
0.00
10
10
20
Threshold
Threshold F
Threshold
Threshold
Figure 4: Stability ratio versus the tolerable poisoning threshold F for action-level robustness with ADP. Results are presented for two Atari games, Freeway and Breakout with RL algorithms DQN and C51 under transition- and trajectory-level poisoning. The blue, green, and red lines represent our proposed certified defense.
A.7 ADDITIONAL RESULTS AGAINST EMPIRICAL ATTACKS
To assess the performance of certifications relative to trajectory-level attacks, we implemented two attacks against HalfCheetah when defended using RDP at 2.0, with the results presented below. These attacks include one in which the rewards in subset of trajectories are replaced with r ~ Uniform[- -1 1] (Random Reward); and one where they are replaced by r = -r; (Adversarial Reward), following the methodology of Ye et al. (2023). For each attack, our experiments were conducted over 150 runs to provide the estimated expected cumulative reward (Est. ECR) with 95% confidence intervals, and the minimum cumulative reward among all the runs. The corresponding policy-level robustness certification (certified lower bound on ECR) J, for each poisoning size 10% (r 200) and 20% 400) shown as the same in Figure 3 in the paper.
20

---

Published as a conference paper at ICLR 2025
Action-level Max Radii
Enviromment
Method
Noise
DQN
Transition
Trajectory
Transition
Trajectory
0.0
N/A
N/A
N/A
N/A
144
Proposed (RDP)
Freeway
2.0
186
66
186
59
3.0
200
136
200
83
COPA
N/A
N/A
N/A
0.0
N/A
N/A
N/A
N/A
99
98
100
100
Proposed (RDP)
Breaknut
1.5
99
99
100
100
2.0
118
117
120
120
COPA
Our proposed method with RDP. The maximum value of maximum tolerable poisoning size r, of the action-level robustness for the evaluated environments, certified methods, noise levels, and RL algorithms.
Table 2:
Action-level Mean Radii
Environment
Method
Noise
DQN
C51
Transition
Trajectory
Transition
Trajectory
0.0
N/A
N/A
N/A
N/A
Freeway
Proposed (ADP)
0,3
2.0
18.0
0,7
11.3
1.7
20.5
20.0
9.6
2.1
Breakout
Proposed (ADP)
1.0
0.1
0.07
0.1
0.04
0.09
0.03
0.18
0.08
0.3
0.10
Table 3: Our proposed method with ADP. The mean value of maximum tolerable poisoning size rt of the action-level robustness for the evaluated environments, certified methods, noise levels, and RI. algorithms.
Action-leved Max Radii
Environment
Method
Noise
DQN
CS1
Transition
Trnjectory
Transition
Trajectory
0.0
N/A
N/A
N/A
N/A
Freeway
Proposed (ADP)
2.0
23
3
18
7
N/A
N/A
N/A
N/A
Breakout
Proposed (ADP)
1.0
10
11
5
2.0
10
11
Table 4: Our proposed method with ADP. The max value of maximum tolerable poisoning size of the action-level robustness for the evaluated environments, certified methods, noise levels, and RI. algorithms.
The results demonstrate that the empirical performance of our certified defense significantly exceeds the certified lower bound. This observation aligns with the theoretical framework, which defines the certified lower bound as a guarantee for the worst-case scenario, and provides conservative measurement of the robustness against attack.
Attack (Trajoctory-levee)
Poisoning Proportion
Est. ECR
Min Cumulative Reward
Certified Lower Bound ECR (J.)
Random Reward Random Reward
10%
79.73 0.44
76.52
48.76
20%
75.94 0.74
71.96
23,17
Adversarial Reward
68.49 0.93
61.33
48.76
Adversarial Reward
20%
60.97 0.58
56.39
23.17
Table Trajectory-level defence with RDP and noise o = 2.0 against empirical attacks in the game Halfcheetha.
21


Reviews:
[{"review_id": "akPKpfzbqN", "reviewer": "Reviewer_pBsV", "summary": "The authors investigate certified poisoning defenses in offline RL. The authors employ DP approaches to achieve both action and policy level robustness guarantees.", "questions": "I'm curious to know how much of a threat existing attacks against offline RL systems actually pose. I know this isn't the focus of this work, but it would be nice to have a bit of explanation about attack specifics/provide more grounding as to why practitioners should be concerned beyond the cited Kumar et al. survey (which I'm not sure is specific to RL poisoning.)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is very well presented. As someone who is not very familiar with RL poisoning, or certified defenses to poisoning, I think the exposition and organization in the paper was great. The flow was logical and the writing, definitions, and lemmas/theorems were all clear. \n* The experimentation is thorough and provides reasonable empirical justification for the proposed method.", "weaknesses": "* Although the proposed method does offer transition-level certified robustness, it seems like the trajectory level results presented in Table 1 do not always show an improvement from the proposed method. I'm specifically looking at the Breakout results. I do acknowledge that the proposed method also works for continuous action spaces, and has other flexibility advantages, so I do not consider the previously mentioned results to be of great concern. \n* The need to train multiple policies could prove cumbersome in certain settings, and may dissuade practitioners from adopting this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4HYpxLqeSy", "reviewer": "Reviewer_UiiB", "summary": "This paper studies the problem of robust DRL algorithms against data poisoning attacks. Based on the idea from differential privacy, it provides a provably robust framework such that by combining an RL algorithm with the framework, one can get a new learning algorithm with a high-probability guarantee on the lower bound of the performance of the learned policy under the attack with a certain level. Compared to the previous work COPA, the performance of the method in this work increases significantly: the performance drops to no more than 50%, with up to 7% of the training data poisoned. At the same time, the percentage COPA can guarantee is only 0.008%. In some sense, this paper develops the first practical and provably robust DRL framework.", "questions": "1. The legends in the figures are missing. They are only explained in the caption.\n\n2. It is important to give a good method a good name. However, it is unclear to me what we should call the method in this work. Could you consider an official name for your method?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. This paper studies an important problem. The threat of poisoning attacks in the real world should not be underestimated. It is crucial to develop methods that are robust against such attacks.\n\n2. This paper provides a framework that can make a class of RL algorithms robust, which is much more powerful than providing a single robust algorithm\n\n3. This paper provides strong theoretical guarantees on its learning framework instead of empirical evaluation, which is especially important for developing trustworthy algorithms. \n\n4. It is interesting to see how ideas from differential privacy help develop a provably robust algorithm in the DRL setting.\n\n5. This paper provides the first practical DRL algorithm/framework against poisoning attacks. The previous work, COPA, had poor performance, which is hard to be realistic about. The experiments result show that the method in this work has a much better performance.", "weaknesses": "1. This work does not investigate the lower bound of the guarantee one can get for a robust DRL algorithm under the poisoning attack. So, it is unknown if the method in this work is optimal and the gap between them.\n\n2. The attack's power is too great and unrealistic. The learning algorithm assumes that the attacker can modify some trajectories arbitrarily without any constraint, which may not be realistic for the attack. Currently, the method expects a 50% drop in performance for robustness against 7% of data corruption, which costs too much and can be a consequence of the attack being unrealistic.\n\n3. It is unclear exactly what will happen to the robust algorithm under some practical attacks. The bound provided in the theoretical analysis might not be tight, especially for the practical attacks that do not lead to the worst case. It will be great to see what is the performance and behavior of your robust algorithms under actual attacks.\n\n4. The discussion on the application of differential privacy in RL is missing in the related works", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Od5j3sKL1o", "reviewer": "Reviewer_joBX", "summary": "The paper proposes a certified defense framework for certifying offline reinforcement learning algorithms against two levels of poisoning attacks, transition-level and trajectory-level. The proposed algorithm leverages differential privacy (DP) techniques; the derived theoretical guarantees regarding per-state action and expected cumulative reward arises from the properties of DP. Compared to prior works, the proposed work can handle continuous action space and stochastic transition. Empirical evaluations demonstrated the effectiveness of the proposed approach.", "questions": "I have listed in the \"Weaknesses\" section some major questions re. clarifications of algorithmic details and comparisons with prior works on specific aspects. Below are some additional questions.\n\n1. Why did the authors choose to consider ADP and RDP? How do different versions of DP impact your methods? Is it through the DP accountant used in training the model + the certification results? -- this needs clarification. There exist other advanced and tighter DP accountants, e.g., Gopi et al., 2021, Doroshenko et al., 2022. Can the authors discuss the applicability of these techniques in your work, and whether this will likely strengthen your results? \n2. From the experimental results, it seems that RDP almost dominates ADP everywhere. Can the authors provide some insights re. why this happens? \n\n**References**\n\nGopi, Sivakanth, Yin Tat Lee, and Lukas Wutschitz. \"Numerical composition of differential privacy.\" *Advances in Neural Information Processing Systems* 34 (2021): 11631-11642.\n\nDoroshenko, Vadym, et al. \"Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions.\" Proceedings on Privacy Enhancing Technologies (2022).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The proposed framework extends beyond the previous work and can handle several important variations including continuous action space, stochastic transition, as well as a transition-level poisoning adversary.\n2. The authors conducted extensive evaluations and offered concrete discussions and comparisons.", "weaknesses": "1) One major issue with the current status of the draft is the lack of clarity. Important technical details are missing, e.g., the authors didn't clarify how DP is used in the framework; I assume it is through applying DP-SGD when training each sub-policy to ensure each of the obtained set of policies is DP, but this is definitely important technical detail that shouldn't be omitted. Moreover, it's confusing when the authors mentioned in line 240 \"depending on whether the DP training algorithm provides transition- or trajectory-level guarantees\" while not providing any detail re. what exact algorithm offers what guarantee. An outline of the training/aggregation/certification algorithm will significantly improve the readability. \n2) Discussion and comparison with prior works are insufficient, which makes it hard to assess the contribution of the proposed work. The paper didn't make it clear re. what's new in this work & what's different from prior works. 1) The notion of policy-level robustness and action-level robustness was already proposed in Wu et al., 2022, then what is new in Sec 3.2; why the highlight of \"multi-level\" if it's already covered in the prior work? 2) The authors pointed out the limitation of the prior work being restricted to deterministic environment and discrete action space. Algorithmically, what difference in the proposed approach enables it to address these challenges? The comparisons are particularly important but are missing in the draft. \n3) In the experiments, the authors compared with the PARL algorithm in Wu et al., 2022, but the prior work proposed two other algorithms TPARL and DPARL which were reported to achieve better results than PARL on some datasets. It is unclear how the proposed algorithm compares with the two stronger algorithms in the prior work. \n\nMinors:\n\n- Lack of mathematical rigor & typos in various places.\n- Line 317: a value network wasn't introduced in the previous context.\n- Line 373: a brief description of SimuEM method would help.\n- Line 377: it's unclear how \"a binary search over the domain of $\\mathcal{K}$\" Is conducted.\n\n**References**\n\nWu, Fan, et al. \"COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks.\" *International Conference on Learning Representations* (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wOThMJzdjq", "reviewer": "Reviewer_4YPC", "summary": "The authors propose a new certified defense technique against poisoning attacks in offline reinforcement learning using differential privacy. The goal of their defense is to certifiably minimize the trained agent's drop in episodic return (policy level robustness), and minimize the probability that one of the agent's actions are changed (action level robustness) when trained on a poisoned dataset $\\tilde{\\mathcal{D}}$. They further want to extend these certifications to both attackers that modify whole trajectories (trajectory-level poisoning) or individual transitions (transition-level poisoning). They model the strength of the attacker as the total number of alterations they make to the dataset.\n\nTheir method, in line with common practice in differential privacy, adds additional random noise to the training of an RL agent given the (potentially poisoned) training data $\\tilde{\\mathcal{D}}$. They show how their particular construction, given established properties of differential privacy, is able to put a lower bound on the agent's drop in episodic return and an upper bound on how many of the agent's actions are changed. The paper's main contributions are theoretical.\n\nThey evaluate their defense against trajectory and transition level attackers of varying strength given discrete and continuous offline datasets. They compare their results against the prior COPA defense, displaying increased episodic return and a larger certified radius.", "questions": "* How do you measure attack performance with respect to $\\bar{r}$ do you directly poison datasets with some attack and evaluate the relevant metrics or is it something else? If you are performing an attack, which attack are you using or how is it formulated?\n* How do you think your defense would work in the presence of backdoor attacks? Do these attacks fit within your certifications?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Defense is well motivated within the scope of prior work.\n* Theoretical results are strong and proofs provided in the appendix are convincing.\n* The proposed defense improves over prior defenses.\n* The writing and presentation is organized and easy to follow.", "weaknesses": "My main complaint with this paper is the lack of clarity in the experimental results section. The authors evaluate each defense against different thresholds of $\\bar{r}$, but it's very unclear to me what this means. Under the authors definitions of $\\mathcal{B}$ the value of $\\bar{r}$ is the number of \"changes\" to trajectories or transitions. Given this definition I could have $\\bar{r} = |\\mathcal{D}|$ by simply shaping the rewards with some function $F(s,a,s') = \\gamma \\Phi(s') - \\Phi(s)$, which does not alter the optimal policy in an MDP [1]. This would be a \"full strength\" attack, which manipulates every timestep in the dataset, but it would not actually impact the optimal policy. On the other hand I could invert each reward as $\\hat{r}_t = -r_t$, which would have the same \"strength\" $\\bar{r} = |\\mathcal{D}|$, but would be a much more impactful attack.\n\nIn short, I think the authors need to be more clear about how they're evaluating their defense so readers can have a better understanding of their empirical results. It's also possible that I'm missing some details here, but it seems that $\\bar{r}$ is not well defined. I request that the authors:\n\n* Provide a precise definition of how $\\bar{r}$ is measured in practice for both trajectory and transition-level poisoning.\n* Clarify whether they are actually poisoning the datasets or just evaluating the certifications at different theoretical poisoning levels.\n* If actual poisoning is performed, describe the specific attack method used.\n\nI also think the authors should have some discussion on why they do not explicitly consider backdoor attacks. One paper they cite frequently is [2], which is one of the first backdoor poisoning attacks in deep reinforcement learning. The goal of backdoor attacks is to maintain benign episodic return of the agent while inducing adversarial behavior in the agent upon observing a predetermined trigger [3, 4]. I'm not certain, but I believe stronger backdoor attacks like [3] may break your action level robustness without a very large $\\sigma$. Therefore, I think motivating why the authors don't explicitly consider backdoor attacks is important (perhaps leave such analysis to a future work?). In summary, I ask that the authors:\n\n* Explicitly discuss why backdoor attacks are not considered in the current work.\n* Address whether their certification framework could potentially be extended to cover backdoor attacks, or explain why it may not be applicable.\n* Consider adding a discussion on the limitations of their current approach with respect to more sophisticated attacks like backdoors, potentially as part of the future work section.\n\n\n[1] \"Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping\" https://dl.acm.org/doi/10.5555/645528.657613\n\n[2] \"TrojDRL: Trojan Attacks on Deep Reinforcement Learning Agents\" https://arxiv.org/abs/1903.06638\n\n[3] \"SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents\" https://arxiv.org/abs/2405.20539\n\n[4] \"BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasets\" https://arxiv.org/abs/2210.04688", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)