Preserving Expert-Level Privacy in Offline Reinforcement Learning
Authors: Navodita Sharma
Vishnu Vinod
Abhradeep Guha Thakurta
Alekh Agarwal
Borja Balle
Christoph Dann
Aravindan Raghuveer
Venue: Accepted by TMLR ()

Abstract: The offline reinforcement learning (RL) problem aims to learn an optimal policy from historical data collected by one or more behavioural policies (experts) by interacting with an environment. However, the individual experts may be privacy-sensitive in that the learnt policy may retain information about their precise choices. In some domains like personalized retrieval, advertising and healthcare, the expert choices are considered sensitive data.  To provably protect the privacy of such experts, we propose a novel consensus-based expert-level differentially private offline RL training approach compatible with any existing offline RL algorithm. We prove rigorous differential privacy guarantees, while maintaining strong empirical performance. Unlike existing work in differentially private RL, we supplement the theory with proof-of-concept experiments on classic RL environments featuring large continuous state spaces, demonstrating substantial improvements over a natural baseline across multiple tasks.

Published in Transactions on Machine Learning Research (10/2025)
Preserving Expert-Level Privacy in Offline Reinforcement Learning
Navodita Sharma" Google Deep
naroditessarmmd@@ggggg.cccmm
Vishnu Vinod" CeRAI, IIT Madrus
1olkmaninnod2200111mmmmaa
Abhradeep Thakurta Google Mied
athakaurtalkgroole.ccm
Alekh Agarwal Google Research
aickbaggaroul@gygogee..oom
Borja Balle Google DeepMind
bballe@googgle.co
Christoph Dann Google Research
Marislannn@goonle.ccom
Aravindan Raghuveer
araghuweerSggoogle.com
Google DeepMind
Reviewed on OpenReview: https: // openreview. net/ formm? id- 20eeVgCa
Abstract
The offline reinforcement learning (RL) problem aims to learn optimal policy from his- torical data collected by one or more behavioural policies (experts) by interacting with an environment. However, the individual experts may be privacy-scnsitie in that the learnt policy may retain information about their precise choices. In some domains like personalized retrieval, advertising and healthcare, the expert choices are considered sensitive data. To provably protect the privacy of such experts, we propose a novel consensus-basoo expert- level differentially private offline RL training approach compatible with any existing oflline RL algorithm. We prove rigorous differential privacy guarantees, while maintaining strong empirical performance. Unlike existing work in differentially private RL, supplement the theory with proof-of-concept experiments on classic RL environments featuring large continuous state spaces, demonstrating substantial improvements over a natural baseline across multiple tasks.
1 INTRODUCTION
Leveraging existing offline datasets to learn high-quality decision policies via offline Reinforcement Learning (RL) is a critical requirement in many settings ranging from robotics (Fu et al., 2020; Levine et al., 2020) and recommendation systems (Bottou et al., 2013: Ie et al., 2019; Cai et al., 2017) to healthcare applica- tions (Oberst & Sontag, 2019; Tang et al., 2022). Correspondingly; there is now a rich literature on effective techniques (Fujimoto et al., 2019b; Kumar et al., 2019; 2020; Cheng et al., 2022) for learning from such offline datasets, collected using one or more behavioural policies. An often overlooked aspect of this literature is,
These authors contributed equally.

---

Published in Transactions on Machine Learning Research (10/2025)
stable prefix
unstable tail
SGD
DII stable
input trajectory
Offliee RI. Algorithm
Sprivate
Expert Set
ADR
DP-SCD
umel
Figure 1: Training pipeline for Expert-level Differentially Private Offlie RL given expert set II (11, ......, F and input trajectories logged into an offline dataset. Dn. ADR (Algorithm 2) splits input trajectories and adds stable prefixes to Datable discarding the rest to D These are used to train any off thell offiine RL algorithm (see Algorithm 3) and learn an expert-level differentially policy Tprivate-
however, that in privacy sensitive domains such personalized retrieval, advertising, and healthcare, the behavioural policies might reveal private information about the preferences strategies used by the corre- sponding experts (users, advertisers, health care providers etc.) whose decisions underlic the offline data. In such scenarios, we ideally seek to uncover the broadly beneficial strategies employed in making decisions by a large cohort of experts whose demonstrations the offline data is collected with, while not leaking the private information of any given expert. We call this novel setting offline RL with expert-level privacy, and design algorithms with strong privacy guarantees for it.
There has been growing interest in differentially private RL, both in online (Wang & Hegde, 2019; Qiao & Wang, 2023a; Zhou, 2022; Liao et al., 2023) and offline (Qiao & Wang, 2023b) settings. The offline RL literature, which is most relevant to this work, primarily focuses on protecting privacy at the trajectory- level. However, when each expert contributes multiple trajectories to the demonstration dataset, private information can still be leaked under trajectory-level differential privacy. Protecting the privacy of an expert in this scenario requires significantly more careful techniques. Secondly, the prior works mostly focus on tabular or linear settings (Qiao Wang, 2023a;b; Zhou, 2022), where the offline RL problem can be effectively solved using count linear regrossion based techniques, and is hence amenable existing techniques for privatizing counts linear regression models. In contrast, we make no assumptions the size of the state space, the parameterization of the learned policy or the policy underlying expert demonstrations, For more detailed discussion of related work, we refer the reader to Section 2.
Contributions: With this context, our paper makes the following contributions:
We identify and formalize the problem of offline RL with expert-level privacy and motivate with practical examples. We also explain the inadequacy of prior algorithms in this setting.
We provide practical algorithms for offline RL with strong expert-level privacy guarantees. In con- trast with prior approaches which use noisy statisties of the data in offline RL (Qiao & Wang, 2023b) or rely on very strict assumptions like tabular seltings or linear function approximators (Wang & Hegde, 2019), limiting real-world applicability, we remove these constraints, supporting any gradient- basod offline RL. algorithm with general function approximators and continuous state spaces.
DP-SGD (Abadi et al., 2016) is the go-to method today for privately training gradient-based general function approximators. We adapt DP-SGD for expert-level privacy natural bascline in problem setting, Our algorithm identifies subset of the data which can be used in learning without any added noise, and significantly improves upon the DP-SGD baseline since it selectively adds noise to gradients only where it is needed, unlike DP-SGD which uniformly noises all gradient updates. We evaluate our algorithm on standard RL benchmarks, and discuss the results in Section 6.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Our Algorithm: Our algorithm proceeds in two stages; see Figure 1. The first is a data filtering stage that produces a set of trajectory prefixes (denoted D using a variant of the Sparse Vector Technique (SVT) for privacy accounting (Dwork & Roth, 2014), that can be used in training without any further noise addition, in a privacy-proserving manner. The intuition is that members of this set are likely to occur across many experts, and hence using it in training does not violate the privacy of an individual expert. The second stage involves selectively running DP-Stochastic Gradient Descent (DP-SGD)² (Song et al.; 2013; Bassily et al., 2014; Abadi al., 2016), adapted for expert-level privacy on the remainder of the trajectories, D" \ D ettable Our empirical results strongly show that combining both stages of the algorithm described above, performs better than either of them individually. The only assumption we make on the underlying offline RL algorithm is that it is gradient-based, so that we can use DP-SGD to privatize its updates.
We motivate our problem setting using the case of financial portfolio management. Different fund managers employ proprietary and privacy-sensitive investment strategies that guide their trading de- cisions. In such a setting, treating the algorithms used by individual fund managers as expert policies and learning an expert-level private investment strategy (policy) using offiin RL will be beneficial as a starting strategy for new investors and other parties that do not have specialized investment stralegies.
Motivation:
Alternatively, in the case of political campaign optimization in a multi-party democracy, large political parties may use proprictary algorithms to maximize their reach, whereas independent candidates, often lacking the same resources, are placed at a disadvantage. Access to a privately learnt aggregate campaigning strategy will allow such players to compete on a somewhat equal footing, allowing voters greater effective choice.
At a high-level, our setting is similar to the 'warm-start' problem in optimization (Sambharya et al., 2024), translated to offline RL setting where data is privacy-sensitive and cannot be directly trained on.
Note on Experts: Throughout the paper, we use the term "expert" to refer to RL agents, policies, algorithms used in various real-world settings. They do not refer to haman experts like doctors, trading agents etc. We learn an expert-level DP policy, from the offiine trajectories logged by such agents interacting with the environment like behavioural policies in offline RL (Levine et al., 2020; Prudencio et al., 2023).
2 RELATED WORK
There have been recent works exploring the interscction of differential privacy (DP) and reinforcement learning (RL) (Wang & Hegde, 2019; Qiao & Wang, 2023a;b; Zhou, 2022; Liao et al., 2023). Qiao & Wang (2023a;b); Zhou (2022); Wang & Hegde (2019) explore the problem either in the tabular setting or in the linear function approximation setting. In either of the cases there is sufficient statistic (like discrete state-action visit counts or feature covariances to privatize sensitive data) which if privatized, would suffice to privatize the whole algorithm. Specifically, Wang & Hegde (2019) explores the use of differential privacy (DP) in continuous spaces to only protect the value function approximator. Liao et al. (2023) explores RL with linear function approximation under local differential privacy guarantee, a stronger notion of privacy where users contributing data do not trust the central aggregator. Meanwhile, Qiao & Wang (2023a;b); Vietri et al. (2020) work with tabular MDP settings and discrete state-action spaces, often utilizing some form of stato-action visit counts to privatize sensitive data. In our work, we extend existing approaches beyond the state-space and linear function approximalor constraints and propose general algorithm which can be used with continuous state spaces, and with general functional approximation for RL tasks.
Comparison to Prior Work on Private Offline RL:
To the best of our knowledge, expert-level DP has not been dealt with in the context of RL so far. Prior works often limit themselves to exploring trajectory-level privacy, which has finer granularity of privacy protection which makes the setting arguably "easier'. In this work, we adopt an expert-level notion of neighbourhood when a large number of behavioural policies (or experts) are used for data collection. We further note that expert-level privacy differs from existing privacy notions like Joint DP (Chowdhury & Zhou, 2022), or other notions which aim to protect the transitions and rewards of the MDP.
JDD to an SGD based approach where gradients coming from individual experts are clipped, and appropriate Gaussian noise added ensure DP.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Background on DP-SGD and Sparse- (2013); Bassily et al. (2014), and adapted for deep learning by Abadi et al. (2016), proposed gradient noise injection to guarantee differentially private model training. Several variants of DP-SGD have been proposed such as McMahan et al. (2017), which allows user-level DP training in federated learning scenarios and Kairouz et al. (2021) a private variant of the Follow-The- Thee-Regullriizei-Leaeee (DP-FTRL) algorithm that compares favourably with amplified DP-SGD (Abadi et al., 2016; Balle al., 2018).
Sparse-Vector Vector Technique: DP-SGD first introduced in Song et al.
The Sparse-Vector Vector Technique (Roth & Roughgarden, 2010; Hardt & Rothblum, 2010; Dwork & Roth, 2014), developed over a sequence of works, allows one to carefully track the privacy budget across queries, and provide tighter data-dependent privacy guarantees opposed to vanilla composition (Dwork & Roth, 2014). At a high-level, the idea is to identify "stable queries", which have low local sensitivity and do not change their answers when moving to a neighboring dataset. Such queries can be answered without paying any additional privacy cost. In Algorithm 2, we use a modification of this idea to output parts of the expert trajectories that are "stable" without incurring any additional privacy cost (in E).
Relationship with Offline RL Literature: Our approach is agnostic to the underlying offline RL algo- rithm used and can be combined with any off-the-shelf model-based (Kidambi et al., 2020; Yu et al., 2020; 2021) or model-free (Kumar et al., 2019; 2020; Cheng et al., 2022) offiie RL algorithms which use gradient updates to learn an optimal policy. We focus on model-free approaches in our experiments.
3 PRELIMINARIES
3.1
Offline RL
Offline RL (Levine et al., 2020; Prudencio et al., 2023) is a data-driven approach to RL that aims to circumvent the prohibitive cost of interactive data collection in many real-world scenarios. Under this paradigm, a static dataset collected by some (possibly sub-optimal) behaviour policy Ty is used to learn a policy x without any interaction with the environment.
A.r,,), where S is the Let the environment be modeled by a Markov Decision Process (MDP) M= =
state space, A is the action space, r: S'xx A + IR is the reward function, P x A- ASS) is the transition kernel (A(Y) the set of all probability distributions over X), p E A(S) is the initial state distribution and 7 E 1) is the discount. factor.
Definition 3.1 (Offline RL). Given an environment represented by the MDP M= Ar, 7) defined above and a dataset D = {(s;, a4, ri, &D generated with (s,,a;) ~ for some state-action distribution H. ri r(s;; and s P(-|s;, a;), the objective is to find an optimal policy x" : + (AA), defined as
arg max ETT [[[,]]
where R2 -| 7're is the discounted return of trajectory т, generated from क and M as: 81 P. ae ~ m(]ss), Ft r(st, a)) and 8411 ~ P(lst, at) for t 2 1 upto |r| timesteps.
A key concern in offline RL is that the state-action distribution encountered in the offline data to learn a policy क might significantly differ from that encountered upon actually executing π in the MDP. There is a host of existing offline RL methods that address this challenge in different ways (Kumar et al., 2019; Fujimoto et al., 2019b; Kumar et al., 2020; Cheng et al., 2022). We use BCQ (Fujimoto et al., 2019b) and CQL (Kumar et al., 2020) as our offline RL methods due to their strong performance across a varicty of tasks in prior evaluations and popularity in existing offline RL literature..
We consider scenario where the dataset contains demonstrations collected from number of different behavioural policies or experts with varying degrees of optimality. We denote the set of behaviour policies by =I= {T...... }. Imposing the constrain of expert-level privacy can conflict with the rewards attainable by the learned policy. For example, if most experts are similar but a single, high-reward expert diverges from them, an inherent tradcoff ariscs between expert-level privacy and utility. We only consider the privatization of the expert policies (the chosen actions). Thus, information about the environmnent dynamics such as the transition kernel and reward function are not protected under our proposed algorithm.
4

---

Published in Transactions on Machine Learning Research (10/2025)
3.2 Differential Privacy
Differential privacy (DP) (Dwork et al., 2006; Dwork & Roth, 2014) quantifies the loss of privacy associated with data release from any statistical database. Informally, a procedure satisfies DP if the distribution of its outputs on two datasets which differ only in one record is very similar. We use approximate-DP (or (e, 6)-P) throughout our work. This notion is formally defined below.
Definition 3.2 ((E)) For positive real numbers > 0 and 0 S 6 S 1, an algorithm A is (s, 0)-DP iff. for any two neighbouring datasets D, D' E D (D is the space of all data records), and S C (ange(A):
Pr(A(D) E S) S Pr(A(D)) E +)+ +
(1)
where (ange(A) is the set of all possible outputs of A.
Definition 3.2 guarantees that the probability of seeing specific output on any two neighbouring datasets differ at most by factor of ct with relaxation of 8. In most ML applications reasonable privacy guarantees have < 10 and s (Ponomareva et al., 2023), where is the cardinality of the dataset.
One can instantiate Definition 3.2 with different of neighbourhood relations. Definition 3.3 describes our setting of expert-level privacy and the more commonly studied trajectory-level privacy.
Definition 3.3 (Expert- and Trajectory-level privacy). Let D be the domain of all valid trajectories possible in the underlying MDP. Let D E "D" be data set of trajectories generated by set of experts II. D D E 'D' are expert-level neighbours if the set of experts | = 1, where A is the set difference. Alternatively, if D, E D are sets of trajectories and |DD' = 1, then D and D' are trajectory-level neighbours.
We note that the key difference between expert and trajectory level privacy notions, as defined above is that removing an expert π can remove up to D trajectories from the overall dataset, where D" is the set of trajectorics contributed by æ. This is a much larger change than allowed in trajectory-lovel privacy, and naïve treatment would result. in paying an additional cost due to group privacy (Dwork & Roth, 2014), scaling linearly in the number of trajectories each expert contributes. At the same time, as argue below, via a set of settings, expert-level privacy is often a critically required notion of privacy protection.
3.3 Problem Setup
Dataset. Let II denote a set of privacy-sensitiv policies, corresponding to m experts, II = ( Tm D D U... U 1m is the aggregated dataset generated by the interaction of the m experts with the environment. Each D is set of N trajectories logging the interactions of policy
(s1, 02 82, a2,...: SL, aL, 8L+1) of length L obtained by with the MDP .M (S, A,r, Po, 7); where 81 Po, as (((-ss)) and Sh+1 P(-Ish, ox). We denote by [T] the length of the trajectory as the mumber of full state-action pairs it consists of. Further, we use TM- = (8), ap.-.. Sk', Oh', #141), , where h < M2 to denote the sub-trajectory of T starting from the h-th timestep until the h'-th, inchuding the trailing next state $1411-
Note on Trajectory Notation: As in Definition 3.1, trajectories are classically defined as a series of (s,a,r,s') tuples, where the next state s' of one transition is the state of the next transition, Since the rewards associated with transitions not used in the privacy analysis or in the filtering stage of our algorithm, we simplify the classical notation for our analysis and proofs as described above.
Goal. The goal is to learn a policy using offline RL algorithm with expert-level DP guarantees D thus ensuring that the policy learnt in this manner is not much different from a policy learnt using D
We make the following mild assumptions in the design of our solution.
Assumption 3.4. The action space A is discrete.
This is fairly mild assumption satisfied in many offiine RL settings. This arises while identifying stable trajectory prefixes for our algorithm, and we leave an extension to continuous action spaces to future work. Assumption 3.5. We can query the sensitive expert policies, i.e., evaluate T. (a(ss) for a given i, and a.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Similar assumptions reasonable to make in setups involving learning student model from ensemble of privacy-sensitive teacher models, where access to the teacher models are assumed (e.g. Papernot et al. (2017)). The privacy-sensitive experts in our setting are analogous to these privacy-sensitive teacher models. Again, query access only comes up in computing the stability of the trajectory prefix, and can likely be relaxed to using behavior-cloned versions of the experts with a slightly worse utility bound.
Assumption 3.6. Let II, be the class of expert policies considered by our algorithm. Hence, I1 II,. We assume a minimum action probahility (Pmiin) defined as, Pmin I min=en, inf min.e. ([aas)
This value is used in the data filtering stage (first stage) of our algorithm. Note that we can always take Puiin 0. However, our algorithm more meaningful when pmin 0. Examples of common policy classes with Pmin 0 include s-greedy policies and softmax policies with bounded parameters.
Existing work in DP-RL relies on strict assumptions like tabular settings or lincar function approximation, limiting real-world applicability. Our approach removes these constraints and exlends lo any stale-ofthhe-rt gradient- based offline RL algorithm with general function approximators and contimuous state spaces.
4 ALGORITHM
For a given trajectory prefix T we define the total probability of r being generated by all the experts as: (IT) Em [[/=)((())ss))) m<(ag18z) 2, where L., = |r], is the length of trajectory т. Algorithm deems T stable if count, exceeds a carefully chosen threshold, implying that enough experts are likely to generate т. These stable prefixes may be used in training an offline RL algorithm without any noise addition. We need Assumption 3.4 for this step. Given a trajectory, the count function gives the sum of probabilities of each expert traversing it. We use count, (II) to check if enough experts can generate this trajectory.
Algorithm PrefixQuery (ApQ): Tests if count of experts expected to execute a trajectory is large enough
Require: Expert policies 1I, trajectory T = (31, 01, 32, ......), stability threshold 0,
1: Compute expected experts: colnl,(II) + Σren 11 -= ((ylss))
2: if counl,(1) (II) + Lap($) > then return T
3: else return 1
Algorithm/ 2 uses Algorithm as subroutine, traversing up to T trajectories to find stable prefixes. The dataset Ditable returned by Algorithm is composed of these stable trajectory prefixes. The discarded tails of each trajectory (including several full trajectories that were discarded) are collected in another dataset:
D unst = (T441977 if Tisk E DJT Detable for k 2 1 or T, D:
Consider any parameterized gradient-based oflline RL algorithm Ho, where & denotes the parameters. Train- ing this algorithm consists of using (s, a, s') tuples collected in the offiine dataset to update the parameters (by splitting the trajectories into into individual transitions). Having split the dataset into stable and un- stable trajectories, we can now leverage the stability of D Ditable to accumulate the gradient updates of Hs on transitions drawn from this dataset without any noise addition. The full algorithm, detailing the training using a mix of noisy gradient updates for transitions from DIl unst and noiseless gradient updates for transitions from TT etable is given in Algorithm 3.
The sampling probability controls the expected fraction of batches sampled from the unstable dataset Di The DP-SGD updates of the form DP-SGD(ón.. B, 82,N) in Algorithm 3 refer to noisy gradient updates with additive noise scaled as to ensure (Ez, 52) expert-level DP. To do so we add the modifications to the standard DP-SGD algorithm (Abadi et al., 2016):
During gradient updates, we ensure that an expert contributes at most one transition to a batch.
2The full probability of being generated by the given of experts, actually proportional the count function multiplied by the state-transition probabilities (which do not change neighbouring experts).

---

Published in Transactions on Machine Learning Research (10/2025)
Algorithm 2 DataRelease (ApR): Releasing public dataset after privatisation
Require:
Expert dataset D expert policies IT, 81, 61, min action-selection probability Pmin; unstable query cutoff T Σ N. m
- max
|T|
TED
8' ()
/32T log(2/6,))
Cmain (ee'
Cmin Peiin
Dll
stable
0
Reshuffle D
for
I....,T do
T next trajectory from D
7:
A - 0 + a 11g(1 + Lap(2 )
8:
for = 1 to L do
9:
r - (lI Ti. E,
10:
if
and |7| then DII
stable
DJ stable (Tia)
11:
else if =1 then
12:
if i then DII D
stable
stable {T
13:
break
14:
return D stable
To account. for privacy amplification while sampling batches from the dataset, we use the use the fraction experts from which we sample. Hence, the subsampling coefficient q for DP-SGD analysis is p where m is the number of experts.
We detail the modified expert-level DP-SGD, used baseline in Section 6, in Appendix B. See Section 5 for the privacy analysis for the overall pipeline combining Algorithms 2 and 3.
Algorithm Selective DP-SGD for Offinee RL
Require: Stable dataset DII stable Unstable dataset D Sampling probability p, Gradient-basse offline RL algorithm Ho. E,, 82, training steps, batch size
1: Initialize do
for n 1 to N do
6~ Bernoullip))
if b = then @a+1 +- DP-SGD(On, B, N), where sample batch B DII
5: else if b= = 0 then d.+1 +- (GD(d,, B), where we sample batch B ~ Dll panst stable
6: return Hay
Relationship with User-level DP: The relationship between trajectory-level and expert-level DP is analogous to the notions of example-level and user-level privacy in existing DP literature (Tévy et al., 2021; Zhou et al., 2022; Zhao et al., 2024). However, we retain naming conventions from existing RL literature.
We note that the expert-level DP-SGD (discussed above and in Appendix B) update step in Line 6 of Algorithm 3 is compatible with other user-level DP algorithms that can be used for training on the unstable data instead. Most practical approaches for achieving user-level DP based on clipping user-level gradients to bound user contributions (Lévy et al., 2021; Bassily & Sun, 2023; Liu & Asi, 2024; De et al., 2022). These approaches have been empirically tested to be successful on large scale datasets (Charles et al., 2024). Other approaches for achieving user-level DP are mostly theoretical in nature and come with restrictions (such as on the function class) that limit their practical usage. The baseline used for comparison in our paper (described in Appendix B) is a representative this class approaches, and a natural way to tackle our problem statement in such a way that we generalize to any function approximators and continuous, high- dimensional state spaces. We demonstrate that our method outperforms this baseline by identifying subset of the data where noisy training is not required, before running the same baseline the rest of the data.

---

Published in Transactions on Machine Learning Research (10/2025)
Sub- cub-Optimality Guarantees for the Learned Policy: While show in the following section that the Algorithm 3 provides expert-level privacy, we do not provide explicit bounds on the suboptimality of the returned policy. Typical offline RL literature shows that using pessimistic approaches, the learned policy is competitive with any other policy whose state-action distribution is well-covered by the offiine data distribution (Xie et al., 2021; Cheng et al., 2022). However, our training distribution undergoes two changes from the a priori offline data distribution. First is that the stable prefixes might be distributionally rather different from the entire state-action distribution, and potentially emphasize the states and actions near shorter prefixes more likely to be stable. Second, the gradient clipping in
the start of trajectory, DP-SGD adds an additional bias. Understanding the effect of latter with arbitrary function approximation is beyond the scope of this paper. We do, however, study the effects of distributional biases from adding varying amounts of the stable prefixes in our experiments, and find that they are typically very beneficial, particularly in tasks where the optimal behaviour does not change significantly over time.
5 PRIVACY ANALYSIS
We note that all the trajectories may be of various lengths. However, as we shall see in the analysis, the privacy cost in the 5 parameter scales linearly with the length of the trajectory, i.., 81 x L8' (Line 2 in Algorithm 2), where is the per-timestep cost and ò1 is the full cost of the trajectory. By setting L to be the maximum length across all trajectories, we carry out worst-casc analysis of the privacy leakage. For a fixed per-timestep privacy budget, we overestimate the total privacy cost by safely considering the worst-case (maximum) trajectory lengths. In other words, for the benefit of the analysis, we assume that all trajectorics have the same (maximal) length I. = max |r|.
rED
Algorithm 2 can be thought of T iterations of combination of two algorithms Ai and A2:
A samples a random expert क ~ Uni and a trajectory 7 ~ D and converts it into |r| queries, Q = {count,, ()..nt,.oont,, (.)), where T2 = Tis is an i-length prefix of the trajectory т.
county,,
A is invoked on IT and the query set Q", running an iteration of the Sparse- Sparse-Vector Vector Technique (Dwork & Roth, 2014) (SVT). A2 treats 11 as the privacy-sensitive database, Q as the query sequence, and i as the threshold for SVT. A2T,I Q") outputs r [T,1) such that r[i] = T A < k and rlk + 1] = L., where l S the length of the stable prefix. Note that r[i] Arq (II, Tia, e')).
AI, upon receiving r, releases the prefix Tikk- ($1, ($1,a1,... ai.......... Sk. aks 8k+1).
For a fixed stream of queries QT, Algorithm A is DP since it just invokes one round of the Sparse Vector Algorithm (Dwork Roth (2014)). Note that the trajectories are just used for creating the queries for SVT, and the set of expert policies TI is the privacy database. For given trajectory т, A2 identifies the longest prefix which is stable, or in other words, has enough experts which are likely to sample this prefix. Note that, the privacy cost e' for identifying this prefix is independent of the length of the trajectory.
However, Q itself is function of the set of experts II. Despite this, in the following analysis, we show that A = A1 o A2 is also difforentially private (due to the carcfully chosen threshold in Algorithm 2 Line 6). Wo then compose the privacy guarantees over T trajectories, since ADR is simply A applied T times.
We now state two key lemmas for proving that A is differentially private. For given trajectory prefix Tk of length k; let E, denote the event that Ai samples any trajectory r' containing тk as a prefix, i.c., T1s Tk- Lemma 5.1. For trajectory prefix The and neighbouring expert sets II, II', if county, (II) 2 Cmin then,
Pr(E,, < e Pr(E,, III')
(2)
where Cmin and e' are defined in Algorithm 2.
If the probability of seeing a trajectory prefix тı across all experts is large enough, then the probability that Ai samples any trajectory with prefix Tę is stable when one expert changes. See Appendix A for proof.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Lemma 5.2. For any trajectory prefix Th, and expert set II such that count,, (II) < 0 the following holds:
PI( The 0, = T) T)<
(3)
where ,,,00 are defined in Algorilim 2.
Informally, if the total probability of seeing a trajectory prefix тк across all experts is not large enough, then the probability that it is labelled stable by Algorithm is less than 8'. See Appendix A for proof.
Thcorem 5.3. Under Assumption 3.4, A is (2t', & )DD,,, where E', are defined Algorithm
Proof Sketch: The main result used to prove this theorem is the following:
Pr(A(IT) = =) S 2c'
=)) IT)
(4)
which is obtained from Lemma 5.4 that we give below.
Let T denote the set of all possible trajectory prefixes using S. To prove that A is (2E', ( differentially private, we need to prove the following for any pair of neighbouring I1, S Ilmmnn
81 27
Σ mma((Pr(A(1)) = =) - 2² Pr(A(II)) = =),00 - TET
The above can be proved using equation 4, and we defer the full proof to Appendix A. The following lemmn gives the key result used prove Theorem 5.3.
Lemma 5.4. For trajectory prefet Tk and neighboring expert sels II, II, if county, (II) < Cmin then,
Pr(A(II) Tx) Pr(Er, [II ITI) 88.
(5)
If count,, (II) 2 Cmin then,
Pr(A(IT) = TK) < 2d P((A(T))))) + Pr( [T]
[T) 88'
(6)
Proof Sketch: The proof of this lemma follows from Lemmas 5.1 and 5.2. Let r .... TE (T, [T,1]*+1 and y(L) = T4. When & < L we have
Pr(A(II) Tk) Σ =))
aEA
where T a denotes the trajectory prefix Tę followed by action a. And when = L,
Pr(A(II) T (T, = (11) Pr(E,, II)
Consider the case k = L. If countr, (IT) is small enough, then Pr(AA Q = 7(4) is less than 5' (Lemma 5.2) and we get equation 5. On the other hand, if county, (II) is greater than Cmann then Pr(Er, III) is bounded by Lemma 5.1. Using that Ag is always e'DP when the queries are fixed we get equation The case of l < I. less straightforward, but can also be analyzed by considering two cases for each a E A, country-a (II) Z Cmin or count (II) < Cmin- See Appendix for the full proof. O
Theorem 5.5. Under Assumption 3.4, Algoritho 2 is 61)-
Proof. Algorithm executes A with (E1 /8T (2g((22)),, 8, / (2T))-D (see Theorem 5.3), T times. Using advanced composition (Corollary 3.21 in Dwork & Roth (2014)) we get that Algorithm 2 is (#1, d1)-DP. ]
Using all the above, derive the following expert-level privacy guarantee for the overall training pipeline.

---

Published in Transactions on Machine Learning Research (10/2025)
Theorem 5.6. Algorithm 3 is (e, 6)-P), where:
= + 82 and == + 82
Proof. Follows from sequential composition of DP guarantees for the release of stable and DP-SGD training on samples from D
unst-
Note on Novelty: We present a novel, useful and non-trivial exlension of the Sparse Vector Technique (SVT) to preserve expert-level DP in Offline RL. The stream of queries in vanilla SVT, (Algorithm 1, Section 3.6, Dwork Roth (2014)); is not dependent on the private databasc. However, in our setup, the stream of queries is a function of the set of the experts (which the private database to be protected), and directly applying SVT does not ensure privacy. However, we show that despite the query stream being a function of the private database, SVT can work if the threshold (line 6 in Algorithm 2) is chosen in certain way, We use the probability distribution of the query stream derived from the experts' policies to prove this.
6 EXPERIMENTS
Ensuring reasonable privacy requires significant hyperparameter tuning and large number of data points, often impacting performance adversely Intuitively, with the demand for expert-level privacy in our setting, we risk severe degradation of utility since each "record" (cach expert's data) forms a much larger fraction of the overall dataset. We recover most of this performance loss, significantly outperforming DP-SGD.
Dataset Generation: We train 3000 experts each on the Cartpole, Acrobot, LunarLander and HIV Treat- ment environments. Cartpole, Acrobot and LunarLander are from the Gymnasium package (Towers et al., 2023). The HIV Treatment simulator is based on the implementation by Geramifard et al. (2015) of the model described in Ernst et al. (2006). Experts are trained variations of the default environment, created by modifying the values of key parameters (e.g. gravity in LunarLander environment). The trained experts are then used to generate demonstrations on a default setting of the environment, collectively forming the aggregated dataset for oflline RL. This training methodology ensures that the experts show varied (often sub-optimal) behaviours on the default environment. As shown in Figure 2, the experts' return-histograne indicate that the mix of experts we create shows significant diversity in the policics learnt. Further details regarding the environment seltings used to train the expert policies are available in Appendix B.
Ide
300
6.5
203
400
0.02
109
0.232
0.00
150
200
250
0.52
-1.0
-100
-97
190
0.00
190
180
Raturn
203
0.88
250
300
350
400
450
Figure 2: Expert return histograms, with Kernel Density Estimation, for experts trained heterogeneous (left to right) LunarLander, Acrobot, CartPole and HIV Treatment environments. We use experts with wide ranges of performance on the test environment. Return values for HIV treatment normalised by 10
Training Sctup: Our algorithm allows us to use any off-tho-shelf gradient-based offline RL algorithm, and we demonstrate this by experimenting with Conservative Q-Learning (CQL) (Kumar et al., 2020) and Batch Constrained Deep Q-Learning (BCQ) (Fujimoto et al., 2019b). More specifically, we use the DQN (Deep Q-Network) version of CQL, and the discrete-action version of BCQ (Fujimoto et al., 2019a), since we operate under the assumption of discrete action spaces. To represent each of the Q-value functions in both the amgorithms and the generative model BCQ, we use a neural network with 2 hidden layers of 256 units each. During training, we perform a grid search to find the optimal set of eyper-parameters (learning Appendix B.
rate, batch size, sampling probability p. DP-SGD noise) for each environment;
II supervised learning, for example, the accuracy on ImageNet training can go from 75% 47.9% the private case (Kurakin al., 2022), under example-level privacy.
10

---

Published in Transactions on Machine Learning Research (10/2025)
The maximum trajectory length is fixed to 200 for Acrobot and CartPole in the collected demonstrations. While evaluating the final learned policy, the maximum episode length is again set as 200 for Acrobot, we increase this to 1000 for Cartpole for increased difficulty in an otherwise "simple" environment. We allow all DP-SGD training runs to progress until the privacy budget is used up completely.
Baselines: We compare our method against DP-SGD adapted for expert-level privacy. A brief discussion of the changes made are presented in Section 4. A full discussion, along with analysis, is presented in Appendix B. We keep the underlying RL algorithm same for fair comparison. In the absence of prior methods for the expert-level privacy sctting, we limit our empirical studies to this naive bascline. We also train a non-privale policy over the whole dataset using vanilla. CQL/BCQ (ignoring privacy concerns). This serves as an empirical upper-bound on the performance achievable under privacy constraints. In our experiments, we report episodic returns, normalized to lie between (random policy) and (optimal policy) averaged over 10 evaluation runs at the end of training, as a fraction of the return of the non-private baseline.
6.1 Results
Normalized Return
0.8
COL: LunerLander
Normalized Retur
1.00
COL: Acrobot
Normalized Return
1.1
COL: CartPole
Normalized Return
0.06
DOL: HiVTreatment
0.7
0.98
1.0
0.04
9.6
0.96
0.9
0.02
0.5
0.8
0.94
0.4
0.7
0.00
5.0
7.5
10.0
5.0
7.5
10.0
5.0
7.5
10.0
5.0
7.5
10.0
LunarLander
BCQ: Acrobot
CartPole
HiVTreatment
Nermliiedd Return
0.60
0.55
0.04
0.98
Nermllied
0.8
0.45
0.97
0.6
0.02
0.40
0.96
5.0
7.5
10.0
5.0
7.5
10.0
5.0
7.5
10.0
5.0 7.5
10.0
DP-SGD
Ours
Figure 3: Performance of our method and DP-SGD for different values of with 8 1/m, where is the number of experts. We report the episodic return, normalized between (random policy) and 1 (optimal policy) averaged over 10 evaluation runs (with 95% confidence intervals) at the end of training, of the non-private bascline. Our method consistently outperforms DP-SGD, especially in the high regions.
fraction
We observe empirically that our algorithm, which leverages expert consensus to select stable trajectory prefixes prior to oflline RL training, shows superior performance to merely using an adapted expert-level DP-SGD algorithm. Figure 3 compares the performance of our method with DP-SGD across different values of All the results are averaged over 3 runs and 95% confidence intervals are also reported. Our method is able to recover much of the performance loss due DP-SGD, especially in the high e-regions.
We experiment with e < 5 too, but for such low E, Algorithm 2 yields very little stable data, which does not meaningfully influence training. Thus, the model primarily learns from the unstable data. Thus, setting E1 = 0 and E2 = E is optimal as we minimize the noise injected by the DP-SGD training. This is in contrast 3e to the observations for higher & values where we take E1 = 4 82 = å for LunarLander and CartPole. For Acrobot and HVVrreament, we set ε = 6,82 0, that is, we only train on DII Ditable Similarly, we split ō 6, -6 82 A. These results for low are presented in Table 2.
We also note the optimal settings of sampling probabilities in Table 1. Interestingly, for Acrobot (with CQL and BCQ) and HIV Treatment (with CQL) environments, just Ditatte suffices to get the optimal performance in high e-region and we did not need to pay any privacy cost. of DP-SGD during model training. We also report the variation of performance with various mixes the two datasets (by varying p) in Table 2.
We also evaluate the effect of varying the number of experts on the performance of our algorithm relative DP-SGD, in figure 4, on the Acrobot environment. As increases, the likelihood that (IT) exceeds
11

---

Published in Transactions on Machine Learning Research (10/2025)
CQL
BCQ
Environment
E 5
7.5
== 10
6
8 7.5
8 10
LunarLander
1.0
0.8
0.8
0.5
Acrobot
0.0
0.0
0.0
0.0
0.0
0.0
CartPole
1.0
0.9
0.8
1.0
0.5
0,5
HIV Treatment
1.0
0.0
0.0
1.0
0.9
0.9
Table 1: Best choices for probability of sampling from D
unstaable ic, p for Algorithm on test environments
Environment
CQL
BCQ
Environment
3
0
0.. =
0.8
p=
1.0
p= 11.0
0.5
1.0
Acrobot (BCQ)
2.5
0.74
0.95
LunarLander
0.75
0.67
0.37
0.54
Cartpole (BCQ)
2,5
0.28
0.58
CartPole
0.67
1.03
0.92
0.31
0.99
0.67
Cartpole (BCQ)
5.0
0.56
0.60
Table 2:
We perform ablation studies with low, intermediate and high sampling probabilities from the unstable dataset D at E = 10. INF (infeasible) denotes that the stable dataset D stable was emptyy, and training was infcasible. We also vary the allocation of & in the low-privacy regime (e S 5) and report utility.
the noisy threshold 0 also increases, and with it the number and length of trajectory prefixes in the stable dataset D Dtaalee Thus, performance scales with the number of experts. Conversely, for very low values of m, the stable dataset is empty and our algorithm defaults towards the bascline.
Note on Implementation: In the implementa- tion of both our method and DP-SGD, create batches by by shuffling datasets of individual ex- perts and randomly interleaving them such that each batch has at most one data point from each expert. This differs slightly from the poisson sam- pling basod accounting our privacy analysis. This gap between analysis and implementation in our work corresponds the gap between privacy plification by subsampling and shuffling. A promis- ing line of work (Erlingsson et al., 2019; Feldman et al., 2021) aims to derive comparable theoretical amplification guarantoes via shuffling. Empirically, however, the gap utility is expected to be mini- mal (Ponomareva et al., 2023).
DP-SGD
Normalized Return
0.99
Ours
0.98
0.97
0.96
0.95
0.94
1000
2000
3000
No. of experts
Figure 4: Performance of our method and DP-SGD m 1000, 2000 and 3000 experts, on the Acrobot environment with 10.0 and CQL as the underlying offinn RL algorithm. We observe increasing improve- ments of our method over DP-SGD as increases.
We the PLD accountant from the dp-accounting library to account for the pri-
vacy loss in DP-SGD training.
7 CONCLUSION
In this work, we initiate the study of offline RL with expert-level privacy. We describe the novel challenges of the setting, and provide theoretically sound algorithms which are evaluated in proof-of-concept evaluation. In future work, we would like to relax Assumptions 3.4-3.6, by replacing the queries to the experts with behaviour cloned versions and extending our trajectory prefix selection paradigm to continuous action spaces (for e.g using kernel density estimation). Understanding the utility implications of Algorithm 3, despite the influence of our privatization scheme on the training distribution is another important avenue for futurr work. Further, privacy auditing in the reinforcement learning setting is another important open rescarch problem orthogonal our work. Finally, while our empirical evaluation convincingly establishes the promise of this approach, leave a larger evaluation across diverse and larger-scale RT benchmarks for future studies.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Acknowledgements
The authors thank Krishna Pillutla and Balaraman Ravindran for fruitful discussions and helpful comments. We also thank the TMLR reviewers for their helpful leedback. This research was supported part by the Bast-aaccal Fellowship at the Centre for Responsible AI, ITT Madras.
References
Martin Abadi, Audy Chu, lan Goodfellow, Н. Brendan McMahau, 1lya Mironov, Kunal Talwar, and Li Zhang, Deep Learning with Differential Privacy. Tn Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS'16. ACM, 2016.
Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy Amplification by Subsampling: Tight Aualyses via Couplings and Divergences. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Raef Bassily and Ziteng Sun. User-level Private Stochastic Convex Optimization with Optimal Rates. In Intervational Conference on Machine Learning (ICML), 2023.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private Empirical Risk Minimization: Efficient A Algorithms and Tight Error Bounds. In Proceedings of the 55th TEEE Annual Symposium on Foundations of Computer Science (FOCS), pp. 464-44 443,473, 2014. ISBN 9781479965175.
Léon Botton, Jonas Peters, Joaquin Quiñonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Di- pankar Ray, Patrice Simard, and Fd Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14(11), 2013.
Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, and Defeng Guo. Real-time bidding by reinforcement learning in display advertising. In Proceedings of the tenth ACM international conference on web search and data mining, pp. 661-670, 2017.
Zachary Charles, Arun Ganesh, Ryan McKenna, H. Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush. Fine-Tuming Large Language Models with User-Level Differential Privacy. arXiv Preprint, 2024.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline rein- forcement learning. In International Conference on Machine Learning, pp. 3852-38 3878. PMLR, 2022.
Sayak Ray Chowdhury and Xingyu Zhou. Differentially Private Regret Minimization Episodic Markov Decision Processes. In Thitt AAAI Conference on Artificial Intelligence, pp. 6375-6383. AAAI Press, 2022.
Soham De, Leonard Berrada, Jamie Hayes, Samuel L. Smith, and Borja Balle. Unlocking High- High-Accuracy Differentially Private Image Classification through Scale. arXiv Preprint, 2022.
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Foundations and Trends in. Theoretical Computer Science, 14-222111-41 2014. ISSN 1551-305X. doi: 10. 10.1561 /000000000000422
Cyuthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Proc. of Theory of Cryplographyy Conference (TCC), 2006.
Ülfar Erlingsson, Vitaly Feldman, Tlya Mironov, Ananth Raghunathan, Kunal Talwar, and Abhradeep Thakurta. Amplification by Shuffling: From Local to Central Differential Privacy via Anonymity. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 2468 2479. SIAM, 2019.
Damien Ernst, Guy-Bart Stan, Jorge Goncalves, and Louis Wehenkel. Clinical Data Based Optimal STI Strategies for HIV: Reinforcement Learning Approach. In Proceedings of the 45th IEEE Conference on Decision and Control, pp. 667-672. IEEE, 2006.
Vitaly Feldman, Audra McMillan, and Kunal Talwar. Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shuffling. In 62nd IEEE Annual Symposium on Foundations of Computer Science (FOCS), pp. 954-964. TEFE, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4RL: Datasets for Deep Data- BriveDriven Reinforcement Learning. arXiv Preprint, abs/2004..77111 2020.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Scott Fujimoto, Edoardo Conti, Mohammad Chavamzadelı and Joelle Pineau. Benchmarking Batch Deep Reinforce- ment Learning Algorithus. arxie Preprint, 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Exploration. In Kamalika Chandhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 2019b.
Alborz Geramifard, Christoph Dann, Robert H Klein, William Dabney, and Jonathan P How. RLPy: A Value- Function-Baan Based Reinforcement Learming Framework for Education and Research. Journal of Machine Learving Reseurch (JMLR), 16(1):1573-155) 2015.
Moritz Hardt and Guy N Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In 2010 IEEE st annual symposium on foundations of computer science, pp. 61-70. IEEE, 2010.
Eugene le, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng- Tze Cheng, Morgane Lustman, Vince Gatto, Paul Covington, et al. Reinforcement learning for slate-based recommender systems: A tractable decomposition and practical methodology. arXiv preprind, 2019.
Peter Kairouz, Brendan Memahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and Private (Deep) Learning Without. Sampling or Shuffling, Proceedings of the 38th International Conference on Machine Learning, pp- 5213-52222 5225. PMLR, 2021.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-Based Offline Reinforcement Learning. In Advences in Neural Information Processing Systems (NeurIPS), volume 33, pp. 21810- 21823, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Newral Information Processing Systems (NewrIPS), volume 32, 2019.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offiine reinforcement learning. arXiv Preprint, abs/2006.04779, 2020.
Alexey Kurakin, Steve Chien, Shuang Song, Roxana Geambasu, A. Terzis, and Abhradeep Thakurta. Toward Training at ImageNet Scale with Differential Privacy. orXiv Preprint, abs/2201. /2201.12328, 2022.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives Open Problems. arXio Preprint, abs/2005.01643,, 2020.
Daniel Lévy, Ziteng Sun, Kareem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, and Ananda Theertha Suresh. Learning with Lser-Level Privacy. arXiv Preprint, abs/2102. 11845, 2021.
Chonghua Liao, Jiafan He, and Quanquan Gu. Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processes. In Proceedings of The 14th Asian Conference on Machine Learning, volume 189 of Proceedings of Machine Learning Research, pp. 627 642. PMLR, 2023.
Daogao Liu and Hilal Asi. User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Rates. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
H. B. McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2017.
Michael Oberst and David Sontag. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pp. 4881-4899 PMLR, 2019.
Nicolas Papernot, Martin Abadi, Úlfar Erlingsson, Ian J. Goodfellow, and Kunal Talwar. Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data. In 5th International Conference on Learning Representa- tions (ICLR), 2017.
Natalia Ponomareva, Hussein Hazimeh, Alex Kurakin, Zheng Xu, Carson Denison, H. Brendan McMahan, Sergei Vassilvitskii, Steve Chien, and Abhradeep Guha Thakurta. How to DP-fy ML: A Practical Guide Machine Learning with Differential Privacy. Journal of Artificial Intelligence Research, 77:1113 -1201, 2023. ISSN 1076- 9757.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. A Survey Offline Reinforce- ment Learning: Taxonomy, Review, and Open Problens. IEEE Transactions on Newrul Networks and Learning Systems, 2023. ISSN 2162-2388.
Dan Qiao and Yu- Xiang Wang. Near-Optimal Differentially Private Reinforcement Learning. In International Con- ference on Artificial Intelligence and Statistics (A (AISTATS), pp. 9914-9940. PMLR, 2023a.
Dan Qiao and Yu-Xiang Wang. Offline Reinforcement Learning with Differential Privacy. In Advances in Newral Information Processing Systems (NeurIPS), 2023b.
Aaron Roth and Tim Roughgarden. Interactive privacy via the median mechanism. In Proceedings of the Forty-Second
ACM Symposium Theory of Computing (STOC), pp. 765 771, 2010.
Rajiv Sambharya, Georgina Hall, Brandon Amos, and Bartolomeo Stellato. Learning warm-start fixed-point optimization algorithms. J. Mach. Learn. Res., 25:166:1-166:46, 2024.
Shuang Song, Kamalika Chaudluri, and Anand D. Sarwale. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, GlobalSTP 2013 Proceedings, pp. 245-248 2013. ISBN 9781479902484.
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Velez, and Jemna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healtheare. Advances in Nearal Information Processing Systems, 35:34272 34286, 2022.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Cymnasium, 2023.
Giuseppe Vietri, Borja Balle, Akshay Krishnamurthy, and Steven Wu. Private Reinforcement Learning with PAC and Regret Guarantees. In International Conference Machine Learning, pp. 9754-9764. PMLR, 2020.
Baoxiang Wang and Nidhi Hegde. Privacy-Preserving Q-Learning with Functional Noise in Continuous Spaces. In Advances in Neural Information Processing Systems (NewrIPS), volume 32, 2019.
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Cellnan--onnistent pessimism for offline reinforcement learning. Advances neural information processing systems, 34:6683 6694, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-basee offiine policy optimization. Advances in Neural Information Processing Systems, 33: 14129-10 14142, 2020.
Tianhe Yu, Aviral Kumnrr Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conser- valive offline model-based policy optimization. Advances in neural information processing systems, 34:28954 289667, 2021.
Puming Zhao, Lifeng Lai, Li Shen, Qingming Li, Jiafei Wu, and Zhe Liu. A Huber Loss Mininization Approach to Mean Estimation under User-level Differential Privacy. NewrIPS, 2024.
Mingxun Zhou, Tianhao Wang, T.HH. Hubert Chan, Giulia Fanti, and Elaine Shi. Locally Differentially Private Sparse Vector Aggregation. In IEEE Symposium on Security and Privacy, 2022.
Xingyu Zhou. Differentially private reinforcement learning with linear function approximation. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 6(1):1 27, 2022.
15

---

Published in Transactions on Machine Learning Research (10/2025)
A Theorem Proofs
A.1 Proof of Lemma 5.1
Lemma 5.1. For trajectory prefix Th, and neighbouring expert sets II, II' if count,, (II) 2 Cmin then,
Pr(F.,, S e Pr(Fr, ITT)
(2)
where Cmin and e' are as defined in Algorithm 2.
Proof. Recall En, the event a trajectory with prefix ти is generated. Assuming т = $1, 01, $2...., aks 8k+11 the probability that a fixed expert generates such a prefix when interacting with the MDP is po(s1) ((ss)s)))(())))),,,,)) Since the expert is chosen uniformly random, have:
Pr(Er. [II
Σ (0o(s1) (1)=))( 188. a;))] i=1
(8)
po(s1) (T) P(*s+1lss: az)) Σ (mT j=1 ( (ayl8)))
(9)
po(31)
count, (II)
(10)
m
Similarly,
Pr(Er, [I') =
m Po(si) (II) ((5)) count,, (II')
(11)
Where m' = [.] Since IaAI 1,
ccounbry (II countyy (II') < 1 and Im m' =
(12)
This gives us the following,
Pr(En, II) Pr(Er, III)
count Tg (II) x m'
(13)
countr, (II') x
country (11) + 1)
countr. (1)( - 1) m + m 1
S max
(14)
(cound, (IT) 1)m (counl,, (II) 1)m m
count,, (II) - 1) m + 1
max
(15)
(count,
(II) 1)m²
Note that, since m + 1 > count,, Th (II)
m count (II)
(16)
m
count (II) -
count., (TT) 1) m +
count,y (II)
max
(17)
(count,, (II) - 1)m² m
country (II) - 1
This allows us to bound the ratio probabilities Erg for II and II' as
Pr(E,, III) Pr(Er, [II')
count, (II)
countyy (II) - S e
(18)
where the last inequality cound, (TI) 2 Cmin e ce'((ce' 1).
7
16

---

Published in Transactions on Machine Learning Research (10/2025)
A.2 Proof of Lemma 5.2
Lemma 5.2. For any trajectory prefix Th, and expert set II such that count,, (II) < 0 the following holds:
(I. Th, 0, = T)
(3)
where e', e',8',0, 0,0 are defined in Algorillom 2.
Proof.
Pr(ArQ(U,7,045 T) Pr [count, (TT) Lap(4/e') 0 (4/6') + Lap(2/s')]
(19)
< Pr 0 Lap(4/e') > 0 + (4/e') log(1/') + Lap(2/s'))
(20)
= Pr [Lap(4/e)) > (4/E') log(118)) Lap(2/E'))
(21)
Consider u ~ Tap(4/e") and v Lap(2/s'). For a fixed v, we have,
Pr 2 (4/e') log(1/8)) S 8' e (
(22)
2
Using the Laplace tail bound, we account for the randomness of v from equation 21, equation 22 and the probability density function of the Laplace distribution, we have,
Pr Pr[Lap(A/E') > loge111 Lap(2/e')) S (e-1e114)
/ 2
x 8
= {e-bole/2)) de
(23)
4
-00
<
(24)
This gives the statement of the lemma.
A.3 Proof of Lemma 5.4
Lemma 5.4. For trajectory prefiæ Tk and neighboring expert sets 11, 11, if county, (II) < Cmin then,
Pr(A(II) TA) III) III 88.
(5)
If countr, (II) 2 Cmin then,
Pr(A((L) = Th) S 25' Pr(A(II'== = Th) + Pr(E,, 8'
(6)
Proof. We split this proof across 3 lemmas. Lemma A.1 proves this bound for the case county, (II) < Cmin
and & < L. Lemma A.2 handles the case count,, (II) 2 Cmin and L. Finally, Lemma A.3 proves this for k Combining these 3 gives the statement of Lemma 5.4.
Lemma A.1. For any trajectory prefic Tk. lo < L, and neighboring expert sets II, IT' C II pm if countt, (II) < Gwin then,
Pr(A(II) Tk) (I)
where 8' is defined in Algorithm 2.
Proof. Let r T= T.. (T, 1T
Pr(A(II) =T)= Σ (T<) Pr(E II]]
(25)
aEA
where Té denotes the trajectory prefix Tı followed by action
17

---

Published in Transactions on Machine Learning Research (10/2025)
Ag runs the Sparse Vector Algorithm using the the query sequence Q to output sequence of the form T.. Here, each T or 1. is an output of Arq (Algorithm 1). The probability that A2 outputs the sequence r can be upper-bounded by the probability of getting T on the 6th call to ApQ. Formally,
(*** = (x,
(26)
Va, Using count, (II) < Cmin < 0, (since 0 = Cmin Sinin/Pmin/ Lemma 5.2, and equation 26,
TT r)) 8
(27)
Combining equation 25 and equation 27, we get,
Pr(A(II) Th) 8' Σ Pr(Erwall]) 8' Pr(Er.!11)
(28)
Lemma A.2. For any trajectory prefar T (s,, Sk: 0ki sk+1), L, and neighboring expert sets II, C pmon if countr, (II) Cmin then,
Pr(A(IT) = =T)) & 25' Pr(A(TT') = Tx) + Pr IT)J''
where 8' are as defined in Algorithm 2.
Proof.
Let r= TT TL [T,1]AH1.
Pr(A(I) = Σ (II ,tt1 Pr Pr(Er..!11) [I]]
(29)
aEA
Let A' {a a E count (II) Cmin } A A A\A' A'
Va A",
m
countre-a(In (II) = [ 1 (a,ls,)) x Te(a|sx+1) 2 Pmin x
(a = Pminn X count, (II) (30)
Pmin xcount (II) < count (1I) < Cmin
(31)
Cmin = 0
+ count,, (II)
(32)
Pmáin
The output of Ag is the output of a sequence of calls Arq (Algorithm 1). The probability that A outputs the sequence can be upper-bounded by the probability of getting T on the tth call to Apq. Using this fact along with equation 32 and Lemma 5.2, we say,
([(A2(1,QTT>)) =) s (I,. TR, T) <
(33)
18

---

Published in Transactions on Machine Learning Research (10/2025)
Pr(A([I) = Th) = Σ
(II Q*=9)= = =)) Pra [I]]
(34)
aEA
(II, QT()))))))) r) Pr E II]
aEA'
(11, (Tk0) = r) Pr (a [I]]
(35)
Σ
[Pr [II, QTk r) Pr ar IT)
aEA'
+
Pr(En II)
[Using equation 33]
(36)
"EA"
2
Pr(A2 Q***)*=)))))))))))))))) = r) Pr(Fre-all [TT)
"EA'
8' Pr F,, IT]
[ssin:: A2 is (e', 0)-DP and Lemma 5.1]
(37)
< 3 2
Σ (Pr(A2(II, = Pr Br IT') + 8'
(38)
aEA
Pr(A(II) = Th) S 25'
e² Pr(A(II) = TR) + Pr(E,, 11)6
(39)
Lemma A.3. For onl trajectory prefit Tı,, where L is the maximam length of trajectory, and pair of neighboring expert sets II, C 11p Ponnn if countr, (II) < Cmin then,
Pr(A(II) = TL) < Pr(E,, -'.
(40)
If count,, (II) 2 Cuin then,
Pr(A(II) = TL) S 2 Pr(A((I)) TI.)
(41)
Proof. Let 7(6) T4, T4.
Pr(A(II) TT)) Pr(A2(1I,77) = PrE,,
(42)
If country (II) 2 Cmin, then using the fact that As is (e',0)-DP and Lemia 5.1, we can say that,
Pr(A(II) =T.) S 26' Pr(A(Il') =T.)
(43)
Recall that the output of Ag the output of a sequence of calls to ApQ (Algorithm 1). The probability that Ag outputs the sequence r can apper-bounder by the probability of getting on the 4 call to Apq. Formally,
(II -1(4)) Pr(APq TL, 0,=' T)
(44)
If country (II) < Cmins then using count, (II) < Cmin Cminn Conin/Pone Pmin 0, equation 44 and Lemma 5.2, we get,
(r(AA([I,T4)) = =)(1) < Pr(Arq TL, = 8' Pr(A(IT) = =T) & (T), From equation 42)
(45)
(46)
Combining 43 and 46, we get the statement of the lemma.
19

---

Published in Transactions on Machine Learning Research (10/2025)
A.4 Proof of Theorem 5.3
Theorem 5.3. Under Assumplion 3.4, A is (2E', 12 )DD,, where e', di are defoned in Algoritim 2.
Proof. Let T denote the set of all possible trajectory prefixes using S, A. From Lemma 5.4, we have, VT T, and all pairs of neighboring sets of experts 11' _ pmpmn
Pr(A(IT) = 7) S 26' Pr(A(TT) =T) Pr(E,)I))- 8
(47)
where 8' = (2T.T)) To prove that A is (2 difiiiilllffe private, we need to prove the following for any pair of neighbouring II, II' _ II
61
Σ TET
max (Pr(((((( = 7) - e² P((A(TT)=))0)) s 27
(48)
Note that, for any fixed k,
Σ
Pr(E,IlI- =
(49)
where |r] denotes the length of the trajectory prefix T.
Σ (I) = Σ E
PI(E,II) = L
(50)
TET
k=1
[T(T
where L the length of full trajectory. From equation 47, we have,
Σ
max (P((A(t =) e Pr(A(IT)) -),00 PIE L5'
51
(51)
2T
TET
This proves Equ 48, and hence the statement of the theorem.
B EXPERIMENTAL DETAILS
B.1 Dataset Generation
We set Pmnn 0.02 for all the environments.
LunarLander: Among the various variable which govern the dynamics of this environment, we chose to modify gravily (default = 9.8), wind power (default = 0.0) and turbulence (default = 0.0) to train different experts. We modified gravity over 10 equally spaced values from 9.0 to 11.0, wind power over 10 equally spaced values between 0.0 to 5.0, and similarly turbulence over 10 equally spaced values from 0.0 to 0.5, to generate 1000 environment settings. For each of these environment settings, we trained agents using the PPO algorithm for 1e6 updates along with hyperparameter tuning of learning rate and batch size. Fach agent is neural network with hidden-layers of sizes (512, 256, 64). For each environment setting, chose the top 3 agents that performed the best on the corresponding environmenn, to form our 3000 experts.
CartPole: For CartPole, we chose to vary gravity, the magnitude of the force exerted when the carl is pushed and the mass the cart. The default values of gravity, force magnitude and mass of the cart are 9.8, 10.0 and 1.0 respectively. We varied each control over 10 equally spaced values over fixed range. For gravity, we fixed this range as (8.75, 11.0), for force magnitude (9.0, 11.25) and for mass of the cart (0.8, 1.25). Similar to above, for each of these 1000 environment settings, trained agents using the PPO
20

---

Published in Transactions on Machine Learning Research (10/2025)
Train
Env
D
TT1
Train
Env 2
D
Train 43
Env 3
D
Test Environment
Train
Env m
D
II
D
Expert Set
Offline Dataset
Figure 5: Outline of Data Generation scheme; We train experts on multiple environments as described below.
algorithm for 1e6 updates while tuning the learning rate and batch size. Each agent a neural network with hidden-layers of sizes (512, 256, 64). We chose the top 3 agents from each setting to form our 3000 experts.
Acrobot: For this environment, we varied the lengths of the 2 connected links and their masses. The default values of all these parameters is 1.0. We assign the same length to both the links each setting. We vary this length, the mass of link and the mass of link 2 over equally spaced values across fixed ranges for each. We fix these ranges (0.8, 1.2) for the length of the links, (0.9, 1.1) for the mass of link and (0.9, 1.1) for the mass of link 2. The combination of these gives 1000 environment settings. Again, we trained agents using PPO for 1e6 updates over multiple learning rates and batch sizes and picked the lop 3 agents from each setting to form our 3000 experts. Each agent is neural network with hidden-layers of sizes (512, 256, 64).
HIV Treatment: In this domain, the 6-dimensional continuous state represents the concentrations of various cells and viruses. The actions correspond to the dosages of two drugs. The reward is function of the health outcomes of the patient, and it also penalises the side-effects due to the quantity of drugs prescribed. Hence, the default reward function has penalty terms, a penalty of 20000 on the dosage of drug 1, and a penalty of 2000 on the dosage of drug 2 (Ernst. et al., 2006). In the default setting of the environment, we assume that the treatment at each steps is applied for 20 days and that there are 50 time steps in an episode. To train experts, vary the penalty 18750 to 21250, and the penalty variation that we introduce is whether the efficacy of the administered drug is noisy not. So we either add no noise to the drug efficacy, or we add gaussian noise with standard deviation 0.01. All these modifications give us 2000 different environment settings. We trained agents using PPO for 2e6 updates on each of the environment settings and over multiple learning rates and batch sizes. Fach agent represented by a neural network with a single hidden layer of size 512. We then picked the top agents from cach of the 2000 environment settings create experts. We picked 1000 more top-performing agents from the 2nd best agents of all the environment settings to get 3000 experts in total.
drug 1 over 50 equally spaced values from drug 2 over 20 equally spaced values from 1850 to 2150. Another
Once we have chosen 3000 experts for any of the above enviromments, we modily each expert's policy so that for each state, the action with the highest probability gets (1 - (IA) - 1) Pmin) probability (where [A] denotes the total number of actions), whereas all the other actions get Pmon probability of being executed. We then use these modified exports to sample trajectories each from the default environment form our offline dataset.
21

---

Published in Transactions on Machine Learning Research (10/2025)
B.2 Expert-Level DP-SGD
We adapted DP-SGD to the expert-level privacy setting to form the baseline that we compare against. To achieve this, during gradient updates, ensure that expert contributes at most transition to batch. To construct a batch of size b, we randomly sample b experts from the total m experts, and then pick a transition from each selected expert. This is analogous to DP-SGD training on a dataset of size m. For privacy accounting, we use PLD accountant from the tensorflow poroooowpwvoaaaaoee library.
Algorithm 4 Expert-Level DP-SGD (outline)
Require: Model Parameters Φ, Expert Datasets D", Expert Policies 11, Loss function [())
EE=I Learning Rate V,, Batch size b, Noise parameter a, Clipping threshold C, Training steps N
1: Initialize du randomly
for 1 1=....,, N
Sample subset of experts from II with subsampling probability m
For each E B sample transition x (s, a, s') from D to form batch of transitions Btrans 5: For each E Btrans compute gradients g:((t) + V L()),:;)
6: Scale gradients to fit clipping threshold ((+) < (0)t)/maa (1, [1.(
Add gaussian noise to gradients Jz + (Een ā (Φ.) + +NV10, 00((0
8: Update model parameters d++1 t 0e - 17 9t
return ΦT
Algorithm 4 describes a general outline for expert-level DP-SGD which we use as a baseline to compare our method against. The loss function here corresponds to the offline RL algorithm.
In our method, Algorithm is incorporated in the training scheme in Algorithm 3 (lines 5 and 6) in the main paper. At each training step, with probability p, we execute lines 3-8 in Algorithm 4 to train batch of transitions. Otherwise, we sample batch of transitions from D Datable and perform normal SGD update (this has no privacy cost). The privacy analysis for Algorithm 3 proceeds in manner similar to that in standard DP-SGD (Abadi et al., 2016) with the exception of the subsampling probability which must be modified to be q p (in the analysis only) to account for the randomness in picking the unstable dataset for sampling.
B.3 Hyperparameter Tuning
For all the environments, we assume that Punin 0.02. For LunarTander, CartPole and HIVTreatment, for e 5, we set E 0,82 01 0, 8 and resort pure expert-level DP-SGD training since the privacy budget is not enough to get meaningful stable trajectories for training. For Acrobot, we set e 6,82 = 0, 5, 52 O. For all other cases, we set E 3c .2 5
94
8 10
10:
We use Conservative Q-Learning (CQL) (Kumar et al., 2020) and Discrete Mlich-Coosttrinedd Deep Q- Learning (BCQ) (Fujimoto et al., 2019a) for training on the offline data. For Lunar Lander, Acrobot and CartPole, we fix all neural network hidden layer sizes to (256, 256). For HIV Treatment, we use a neural network of one hidden layer of size 512 to represent the Q-function. For DP-SGD updates, we clip all gradient norms 1.0. For Acrobot, for E 10, 7.5, 5, we set T (defined in Algorithm 2) equal to 25, 20, 20 respectively. For LunarLander, for E 10, 7.5, set T' equal to 100 and 62 respectively. For CartPole, for c == 10, 7.5, we set T equal to 25 and 20 respectively. We use T = 50 for HIVTreatment.
We Lune the following hyperparameters for our method: learning rate batch size b, probability of sampling from Di during training (p), and DP-SGD noise multiplier (s) which is the standard deviation of the gaussian noise applicd the gradients divided by the clipping threshold. We perform grid search over all The search spaces for different hyperparameters are as follows: n [0.0001, 0.0005, 0.001, 0.005, 0.01), b [64, 128, 256), (0.9, 0.8, 0.5, 0.0), and [10.0, 20.0, 30.0, 40.0, 50.0, 80.0). For each
22

---

Published in Transactions on Machine Learning Research (10/2025)
configuration of hyperparameters, let the model train for as long possible with the given value of E2. We report the average episodic return obtained over 10 evaluation runs spaced over the last 10000 steps of training. The best hyper-parameter configurations for each setting are given in Table 3.
Table 3: Best hyper-parameter settings across different environments, ofllineRRL algorithms and values.
CQL
BCQ
Environment
?
b
p
p
5
0.001
128
0.0
0.01
256
0.0
Acrobot
7.5
0.001
128
0.0
0.01
256
0.0
10
0.0005
128
0.0
0.01
256
0.0
5
0.0005
256
1.0
80
0.005
64
1.0
10
LunarLander
7.5
0.005
64
0.8
20
0.001
128
0.5
50
10
0.0005
64
0.8
40
0.005
128
0.5
20
5
0.005
256
1.0
20
0.005
128
1.0
10
CartPole
7.5
0.001
128
0.9
50
0.01
128
0.5
20
10
0.0001
128
0.8
80
0.001
128
0.5
40
5.0
0.01
256
1.0
10
0.005
128
1.0
20
HIVTreatment.
7.5
0.0001
64
0.0
0.005
64
0.9
20
10
0.0001
64
0.0
0.005
64
0.9
20
23


Reviews:
[{"review_id": "qDgEuCjD2G", "reviewer": "Reviewer_rtSW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n* As the authors point out, experimental evaluation of DP+RL methods are relatively limited.\n* The modification to the SVT may be of independent interest.\n* The essential structure of the problem is interesting, in that it is not quite group privacy.\n\n**Weaknesses:**\n\n* It is not entirely clear that the experimental settings reflect real scenarios in which privacy is an issue. The example on Page 5 of personalized healthcare is interesting but the closest experiment to this setting is based on synthetic agents, making the privacy question less clear.\n* Some of the mathematical setup is a little unclear, especially since some assumptions (like deterministic awards) are not without loss of generality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates scenarios for reinforcement learning (RL) with differential privacy (DP) where the thing to be kept private are the contributions from different \"expert\" policies which are used in the (offline) training of a new policy.\n\nThe sensitive data used by these offline RL methods take the form of *trajectories*. The contention of the paper is that if a particular trajectory (or prefix thereof) is \"common enough\" among the experts then using a variation of the sparse vector technique is enough to guarantee privacy. However, if the trajectory is less common then additional privacy protections are needed in the form of DP-SGD.\n\nThe main contributions are:\n\n* formulating this kind of problem under privacy constraints\n* providing a new \"sorting\" approach to pull out common trajectories\n* some experimental evidence for the approach\n\nI question whether this approach is truly \"practical\" in the way claimed by the authors, however. It seems quite restricted to schenarios where the experts are simulateable policies and not actual measured data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Motivation/applications:**\n\n* The experiments are done on 3 mechanical/robotic applications and one healthcare application. I am not sure the first three are really reflective of privacy-relevant applications, although perhaps some case could be made. This is OK: we often work with the datasets that we have. For the HIV Treatment application, since it is a simulator, is seems that the approach is not feasible with actual observational data. Perhaps the authors can clarify a bit more how one might work with actual recordings of trajectories.\n\n* I am not sure if the assumption that rewards being deterministic is not without loss of generality. In the healthcare application, the actions and states of patients can be sufficiently different that the rewards are almost surely not deterministic: they would be patient-specific and also location specific. It's not the case that the same experts are treating the same patient in some controlled way. Some discussion on how assumptions do and do not hold in the motivation application(s) would be helpful. \n\n* The privacy analysis assumes that the trajectories are all of the same length. This is actually fine because the trajectories experimentally are all synthetic. The authors should be clearer about whether or not this approach is intended to work with real data trajectories/traces or only with full access to simulators which can generate as many trajectories as we want.\n\n**Mathematical setup:**\n\n* I think formally describing a trajectory would be helpful. It's sort of implicit in Definition 3.1. In particular, it seems that $\\tau \\in \\{ S \\times A \\times R \\times S \\}^{\\ast}$ so that trajectories have infinite horizons (which makes sense for discounted MDP)? But later in Algorithm 1 $\\mathcal{D} = \\{ S \\times A \\}^\\ast$ because the rewards are ignored (which is not really without loss of generality, as noted above. But then on page 7 we get $\\{ S \\times A \\times R \\times S \\}^{\\ast}$ again. So what is it?\n\n* Each expert $\\pi$ induces a distribution on $\\mathcal{D}$, a collection $\\Pi$ of experts a collection of distributions on $\\mathcal{D}$. This makes the trajectory-level neighborhood and expert-level neighborhood definitions have quite different flavors: trajectory-level neighborhood seems quite strong: you would need some sort of \"smoking gun\" trajectory that would only be in one $D$ and not the other $D'$. Is this a practical difference to consider? It seems not from the discussion before 3.3 but it would be good for the authors to clarify.\n\n* In practice, it seems getting the *full set of trajectories* generated by an expert $D^{\\pi_i}$ is unlikely: wouldn't you only get a sample? So is $D^{\\Pi}$ really the data set under consideration?\n\n* Is the $count$ function correctly named? Should this function also include the state transition probabilities and not just the policy actions?\n\n**Algorithms/Experiments:**\n\n* While a lot of work goes into splitting into stable and unstable trajectory sets, it would be nice to see some sort of (experimental) verification that using the unstable set has measurable benefits, since in some cases the training is done only with the stable set.\n\n* How is the clipping level chosen in practice?\n\n* Is the sampling in the experiments done without replacement? There are often gaps in the implementation between privacy amplification by subsampling (in the pseudocode) and actual implementation (shuffle and process in batches).\n\n* (Figure 3) Shouldn't $\\delta << \\frac{1}{m}$ instead of equal to $\\frac{1}{m}$? It was not clear from Appendix B.\n\n* I would have expected a bit more exploration of different $\\epsilon$ levels than just the 3 selected. Is this because training is very expensive? If so, does that limit the practicality of this approach? \n\n* For completeness, some evidence for how things \"break down\" with smaller $\\epsilon$ would be useful to see (even in an appendix). It's not clear if these large $\\epsilon$ values are due to the complexity of the applications or lack of data or some other factors.", "broader_impact_concerns": "From Assumptions 3.5 and 3.6 it seems that the primary motivation is that the experts are themselves machines, making the motivating application in Example 3 incredibly problematic, especially from an ethics perspective: are we going to collect data from sepsis patients based on automated AI experts providing treatments by taking actions and observing trajectories? I don't think this is what the authors mean to suggest but providing this as a motivating application but then allowing unfettered querying of policies are not compatible. Please remove this example or provide a better example that is more compatible with the solution setup.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l82re7YHX7", "reviewer": "Reviewer_CcoS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ The paper proves (ε, δ)-DP under the expert-level neighbor definition, carefully accounting for the challenges of streaming queries dependent on the private dataset. To me, the privacy analysis appears to be rigorous and non-trivial.\n\n+ The proposed framework can be combined with any gradient-based offline RL algorithm, increasing its practical impact and adaptability.\n\n\n**Weaknesses**\n\n+ The paper lacks theoretical bounds on the suboptimality of the learned policy under the DP constraints, which would make the privacy-utility tradeoff more explicit.\n\n+ The current privacy analysis relies on several assumptions, some of which may affect the generalizability of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces and formalizes the setting of expert-level differential privacy in offline reinforcement learning (RL), where the goal is to learn a high-utility policy while protecting the identity and strategies of contributing experts. The authors propose a two-stage algorithm that first filters privacy-safe trajectory prefixes using a novel variant of the Sparse Vector Technique (SVT) and then trains any gradient-based offline RL algorithm via a selectively private SGD, where only the privacy-sensitive portions are noised. Theoretical results establish (ε, δ)-DP guarantees under expert-level neighboring datasets, and empirical evaluations on multiple RL benchmarks demonstrate improved utility compared to baseline DP-SGD methods adapted to expert-level privacy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "One of my major concerns is the motivation of the expert-level DP in offline RL. First, I am not fully convinced by Example on page 5. That is, why we need to protect the privacy of the doctors (if I understand it correctly). Second, it seems to me that the key difference between expert-level and trajectory-level is similar to the difference between user-level and item-level DP. That is, expert-level is similar to user-level in that changing one expert/user, will change multiple data points (trajectories). If this is correct, then my question is -- can we leverage the recent advance in user-level DP optimization (like supervised learning or stochastic optimization) to handle expert-level offline RL?\n\nI am looking forward to seeing the comments from the authors on the above questions. \n\nMinor issue: \"To account for privacy amplification while sampling batches from the dataset, we use the use the\nfraction\" it has two \"use the\"", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FS6CzJi4pR", "reviewer": "Reviewer_f6f8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nNovelty: First to formalize and address expert-level privacy in offline RL, filling a gap in prior work (e.g., trajectory-level DP or tabular/linear settings).\n\nPractical Flexibility: Compatible with any gradient-based offline RL algorithm and general function approximators (e.g., DNNs), enhancing real-world applicability.\n\nTheoretical Soundness: Provides rigorous privacy guarantees via a non-trivial adaptation of SVT, addressing dependencies between queries and the private database (experts).\n\nCompelling Experiments: Extensive evaluation on diverse tasks shows significant gains over DP-SGD (e.g., +20–40% normalized return at \n$\\epsilon=10$). Ablation on expert count (Fig. 4) validates scalability.\n\nClarity: Well-structured, with clear problem setup, algorithm pseudocode, and intuitive explanations.\n\nWeaknesses:\n\nThe assumption may be relaxed. I suggest to explore extensions to continuous actions (e.g., via density estimation) and behavior-cloned experts in future work.\n\nIs the theoretical guarantee optimal? I do not see any further analysis.\n\nAssumption 3.6 is too restrictive. Assuming minimum action probability is not realistic, as in real scenarios, some actions are only taken limited times. However, I think for a theoretical analysis I think that it is OK for the current paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel and timely contribution to privacy-preserving offline reinforcement learning (RL). It addresses the underexplored challenge of expert-level privacy—protecting the sensitive strategies of individual experts contributing to an offline dataset—with a theoretically grounded and empirically validated approach. The work is well-motivated, methodologically sound, and offers practical advancements.\n\nProblem Formulation: Formalizes expert-level differential privacy (DP) for offline RL, where the goal is to protect all trajectories contributed by a single expert, a coarser-grained notion than trajectory-level DP. This is motivated by real-world applications (e.g., healthcare, advertising) where experts (e.g., doctors, advertisers) require privacy.\n\nAlgorithm: Proposes a two-stage consensus-based framework:\n\nStage 1 (Data Filtering): Uses a privatized variant of the Sparse Vector Technique (SVT) to identify \"stable\" trajectory prefixes (common across many experts) that can be used in training without noise.\n\nStage 2 (Selective DP-SGD): Applies expert-level DP-SGD only to \"unstable\" data, reducing overall noise injection.\n\nTheoretical Guarantees: Rigorously proves $(\\epsilon,\\delta)-DP for the entire pipeline, leveraging advanced composition and properties of SVT.\n\nEmpirical Validation: Demonstrates superior performance over an expert-level DP-SGD baseline across four RL benchmarks (CartPole, Acrobot, LunarLander, HIV Treatment), especially for higher $\\epsilon$ (lower privacy). The approach integrates seamlessly with standard offline RL algorithms (CQL, BCQ).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This paper compares the definition of expert level privacy and trajectory level privacy. I think that these concepts are close to \"user level privacy\" and \"item level privacy\". Therefore I suggest to discuss these in related works. \n\n[1] Levy, Daniel, et al. \"Learning with user-level privacy.\" Advances in Neural Information Processing Systems 34 (2021): 12466-12479.\n\n[2] Zhao, Puning, et al. \"A huber loss minimization approach to mean estimation under user-level differential privacy.\" Advances in Neural Information Processing Systems 37 (2024): 130018-130056.\n\n[3] Zhou, Mingxun, et al. \"Locally differentially private sparse vector aggregation.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)