Differentially Private Deep Model-Based Reinforcement Learning

Authors: Alexandre Rio, Merwan Barlier, Igor Colin, Albert Thomas
Primary area: reinforcement learning
Keywords: machine learning, reinforcement learning, privacy, differential privacy, deep learning, model-based, offline

Abstract:
We address private deep offline reinforcement learning (RL), where the goal is to train a policy on standard control tasks that is differentially private (DP) with respect to individual trajectories in the dataset. To achieve this, we introduce PriMORL, a model-based RL algorithm with formal differential privacy guarantees.
PriMORL first learns an ensemble of trajectory-level DP models of the environment from offline data.
It then optimizes a policy on the penalized private model, without any further interaction with the system or access to the dataset. 
In addition to offering strong theoretical guarantees, we empirically demonstrate that PriMORL enables the training of private RL agents on offline continuous control tasks with deep function approximations, whereas current methods are limited to simpler tabular and linear Markov Decision Processes (MDPs). We furthermore outline the trade-offs involved in achieving privacy in this setting.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729892633275

thread_type:
official_review

<thread_object>
creator: reviewer_wk8a

<content>
summary:
In this paper, the authors consider private deep offline reinforcement learning (RL), where the goal is to train a policy on standard control tasks that is differentially private (DP) with respect to individual trajectories in the dataset. To achieve this, they introduce PriMORL, a model-based RL algorithm with formal differential privacy guarantees. PriMORL first learns an ensemble of trajectory-level DP models of the environment from offline data. It then optimizes a policy on the penalized private model, without any further interaction with the system or access to the dataset. In addition to theoretical guarantees, they empirically demonstrate that PriMORL enables the training of private RL agents on offline continuous control tasks with deep function approximations.

strengths:
1. The problem of offline RL with DP is important and well-motivated.
2. This paper proposes a practical solution to the problem.
3. The authors do experiments to support their algorithm.
4. The paper is well-written in general.

weaknesses:
1. The main concern is about technical novelty. The definition of Trajectory-level DP is directly adapted from [1]. The first part directly applies DP-FEDAVG, while the second part is about learning from the private model with pessimism. To the best of my knowledge, [1] is based on the same idea of private model + pessimism. The DP guarantee for the private model is from previous results, and the DP guarantee for learning the policy is from standard post-processing. I do not see any technical challenge in the process. It will be better if the authors could discuss about the challenges.

[1] Dan Qiao and Yu-Xiang Wang. Offline reinforcement learning with differential privacy.

2. Proposition 4.4 only provides an error bound for estimating the value function of $\hat{\pi}$, which is not standard. Is it possible to derive any results about the sub-optimality gap $V^\star-V^{\hat{\pi}}$?

3. In the experiments, the privacy protection is very weak (some $\epsilon$ being close to 100). What will happen for more practical choices of $\epsilon$? E.g. $\epsilon \approx 1$.

questions:
Please refer to the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730171984114

thread_type:
official_review

<thread_object>
creator: reviewer_h3xi

<content>
summary:
This paper introduces PRIMORL, a differentially private (DP) model-based reinforcement learning algorithm for offline settings. PRIMORL trains policies for continuous control tasks while ensuring trajectory-level privacy by learning an ensemble of DP models from offline data. This approach protects against privacy leaks in RL, especially critical in applications where individual trajectories may contain sensitive information. PRIMORL operates in an offline, infinite-horizon setting, leveraging private models to optimize policies without further interaction with the environment. Empirically, PRIMORL demonstrates competitive performance on deep RL tasks, advancing private RL beyond simpler tabular and linear MDPs and addressing practical privacy-performance trade-offs.

strengths:
The paper is clearly organized. The authors first introduce the trajectory-level differential privacy framework within the offline RL setting, then explain the method for training private models using a model ensemble approach, covering both implementation details and theoretical guarantees. Finally, they describe how policy optimization is achieved by incorporating uncertainty techniques, with theoretical support provided as well.

weaknesses:
1.	It is unclear how the private-variant experiments control for \epsilon. In the theoretical guarantees section, \epsilon is presented as a theoretical bound, yet here it seems to be treated as a tunable hyperparameter, with little explanation connecting these two perspectives. 
2.	While the motivation for the work is compelling, the experimental design is relatively simple and basic. I would have liked to see experiments that address the unique challenges of applying DP frameworks within the RL domain, yet this paper lacks a broader experimental analysis to underscore the real-world relevance of introducing DP into RL.

questions:
1.In Theorem 4.2, a general formula for \epsilon^{MA}(z, q, T, \delta) is presented, but neither the main text nor the appendix provides a complete, detailed expression of this formula. Could you include a full derivation of this formula so we can clearly understand how \epsilon^{MA} is calculated based on inputs like z, q, T, and \delta?
2.In Section 4.3.2, the paper discusses handling model uncertainty under a private setting but appears to apply existing uncertainty-handling techniques from non-private settings directly to the private setting. Could you clarify any special considerations or unique aspects of handling uncertainty in the private setting? Specifically, how might model error and uncertainty differ under a private setting, given its unique constraints? We would appreciate any further insights on this point.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645092211

thread_type:
official_review

<thread_object>
creator: reviewer_pahs

<content>
summary:
This paper studied differentially private model-based offline reinforcement learning (RL). The paper proposed a new algorithm that provide differential privacy for training ensemble models. Besides theoretical guarantees, the experiments also demonstrate the effectiveness of the proposed algorithm.

strengths:
1. The proposed method that provides privacy guarantees for training ensemble models is novel.
2. The investigation of the problem is thorough.

weaknesses:
1.  There exist several typos. For example, in line 274, 'it does entirely remove...' I assume it should be 'it does not entirely remove...' because increasing $N$ will degrade the model performance.
2. The major weakness is that there is a lack of a more explicit discussion on each term in the theoretical results, such as $\epsilon_p$ and $\epsilon_p^{DP}$. I would be curious about if these terms depend on privacy parameter or $N$, if so, what should be the approximate dependency.

questions:
It seems that the number of ensembles $N$ plays a vital role in the results. This work has already reduce the dependency to $\sqrt{N}$ in the sensitivity. I am just curious is it possible to further reduce it in training ensemble models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731438818837

thread_type:
official_review

<thread_object>
creator: reviewer_bcjw

<content>
summary:
This work extends model-based offline reinforcement learning, namely MORL, to its differentially private variant, namely PriMORL. The work intends to guarantee trajectory level DP, which means it treats two datasets that differ in at most one (entire) trajectory as neighboring datasets and asks the algorithm to output indistinguishably between them.

The differential privacy is achieved by model privacy, i.e. if the model is learned in a DP way, then by post processing the algorithm is also DP. This though comes with the limitation that the learning algorithm can no longer access the dataset once the model is obtained from the previous phase. To achieve DP model learning, it randomly draws a subset of trajectories and uses this batch of data to estimate a gradient. A clip is then applied to the gradient which bounds the sensitivity of the gradient. The work discusses different clipping techniques which fine tune the clipping threshold. The DP guarantee is then proved by moment accountant.

Once a model is obtained the policy is trained through pessimistic private MDP, which is by the intuition that being aware of the model uncertainty requires pessimisty. This intuition inspires the authors to run soft actor-critic on the  pessimistic variant of MDP, where the reward is reduced by the uncertainty level. This to some extent mitigates the cost of not being able to access the data once the model training is complete.

Many experiments are provided on tasks like pendulum, cartpole, and halfcheetah. The authors didn't seem to compare their algorithms with baseline methods, while there seems to be many. Though, the performance of the proposed algorithm is reasonably good on its own.

strengths:
As a summary, this work investigates a natural setting and models the problem in a reasonable way. The work guarantees DP through model learning and post processing, which is intuitive. The performance of the algorithm seems decent.

weaknesses:
1. Model/Techniques are not exciting.
2. No baseline comparison.
3. Limited testbeds

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1731959064264

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Revision

comment:
We thank the reviewers for their constructive feedback. We have incorporated all the comments in a new version of our paper, which we have just uploaded. We hope that these modifications, along with the discussions, adequately address your concerns and answer your questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732809076104

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New Paper Revision and General Comment (1/2)

comment:
We would like to thank all the reviewers for their feedback and for engaging in valuable discussions during the rebuttal period, providing us with meaningful insights. Efforts made to address the reviewers' concerns enabled us to make impactful improvements to our paper. We have uploaded a revision of our work in which we highlight, with blue text, the significant modifications and additions we made following this rebuttal.

We would like once again to emphasize the relevance, impact, and contributions of our work. In particular, we highlight why the study of private offline RL is well-motivated, re-contextualize our work within the current private RL literature, offer evidence of the novelty and impact of our work, and emphasize the challenges of the setting and our technical contributions.

In this paper, we address offline deep reinforcement learning with differential privacy (DP) guarantees. We use the well-motivated notion of trajectory-level privacy. We propose a model-based approach, PriMORL, and assess it empirically on several continuous control tasks.

**Motivations for the study of private offline RL**
- There are numerous current or potential applications of RL in risk-sensitive scenarios, including recommendation engines, healthcare, personalized finance, and autonomous vehicles. In all these scenarios, RL agents are trained on sensitive, personal user data.
- Privacy threats in RL are well documented, including powerful membership inference attacks. Studies suggest RL is no more immune to privacy leakage than supervised learning or any other field.
- Offline RL is particularly relevant in practical or industrial applications, where online interaction with the environment is often impractical, costly, and/or hazardous.

**Current state of the private RL literature**
- Existing DP approaches in both online and offline RL are predominantly theoretical and have limited practical impact. Specifically, they are limited to episodic tabular and linear MDPs, and the few experiments are restricted to numerical simulations.
- Existing methods cannot intrinsically scale to problems typically encountered in deep RL.
- There is currently no DP RL method for general MDPs in the $\gamma$-discounted MDP setting. More generally, there is currently no DP method for deep RL.

**Why this work is novel and impactful**
- While reinforcement learning encounters the same privacy challenges as other areas of ML, no existing work has proposed a private RL method that matches the versatility, scalability, and empirical effectiveness of \textsc{DP-SGD} for supervised learning. We believe this observation justifies a shift from predominantly theoretical work to practical deep RL approaches with the ability to scale to complex problems.
- We propose the first deep RL approach with formal differential privacy guarantees. It works in general, continuous MDPs in the $\gamma$-discounted infinite-horizon setting.
- For the first time in the DP RL literature, we tackle standard control tasks with deep function approximations. Experiments show that our method can learn deep DP RL agents with limited performance cost compared to the non-private baseline. These results set a new standard for practical DP RL.
- Despite its current limitations, we identify promising directions to scale this approach towards high-dimensional deep RL problems. Overall, this work takes a first step into bridging towards the deployment of private RL agents in practical risk-sensitive applications.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the problem of differentially private learning in the context of offline RL. The primary concern with the paper is on (1) novelty: extending DP to offline RL is straight-forward conceptually, especially using a model-based method. One can simply apply a DP supervised learning method to learn the model and then do planning in the learned model as one would expect. The trajectory-level DP seems to be an non-substantial deviation; (2) Lack of convincing experimental evaluation: While the authors claim that the method in this paper is more scalable than prior work, that is not true. Continuous control task with low-dimensional feature is considered as simple as tabular/linear MDPs. If the authors wish to claim practical relevance for the proposed algorithm, they must demonstrate its performance on benchmarks with visual input.