Local Differential Privacy for Regret Minimization in Reinforcement Learning

√ K/ε regret in any finite-horizon MDP after K episodes , matching the lower bound dependency on the number of episodes K . 1 Introduction . The practical successes of Reinforcement Learning ( RL ) algorithms have led to them becoming ubiquitous in many settings such as digital marketing , healthcare and finance , where it is desirable to provide a personalized service [ e.g. , 1 , 2 ] . However , users are becoming increasingly wary of the amount of personal information that these services require . This is particularly pertinent in many of the aforementioned domains where the data obtained by the RL algorithm are highly sensitive . For example , in healthcare , the state encodes personal information such as gender , age , vital signs , etc . In advertising , it is normal for states to include browser history , geolocalized information , etc . Unfortunately , [ 3 ] has shown that , unless sufficient precautions are taken , the RL agent leaks information about the environment ( i.e. , states containing sensitive information ) . That is to say , observing the policy computed by the RL algorithm is sufficient to infer information about the data ( e.g. , states and rewards ) used to compute the policy ( scenario ¬ ) . This puts users ’ privacy at jeopardy . Users therefore want to keep their sensitive information private , not only to an observer but also to the service provider itself ( i.e. , the RL agent ) . In response , many services are adapting to provide stronger protection of user privacy and personal data , for example by guaranteeing privacy directly on the user side ( scenario ) . This often means that user data ( i.e. , trajectories of states , actions , rewards ) are privatized before being observed by the RL agent . In this paper , we study the effect that this has on the learning problem in RL . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Differential privacy ( DP ) [ 4 ] is a standard mechanism for preserving data privacy , both on the algorithm and the user side . The ( ε , δ ) -DP definition guarantees that it is statistically hard to infer information about the data used to train a model by observing its predictions , thus addressing scenario ¬ . In online learning , ( ε , δ ) -DP has been studied in the multi-armed bandit framework [ e.g. , 5 , 6 ] . However , [ 7 ] showed that DP is incompatible with regret minimization in the contextual bandit problems . This led to considering weaker or different notions of privacy [ e.g. , 7 , 8 ] . Recently , [ 9 ] transferred some of these techniques to RL , presenting the first private algorithm for regret minimization in finite-horizon problems . In [ 9 ] , they considered a relaxed definition of DP called joint differential privacy ( JDP ) and showed that , under JDP constraints , the regret only increases by an additive term which is logarithmic in the number of episodes . Similarly to DP , in the JDP setting the privacy burden lies with the learning algorithm which directly observes user states and trajectories containing sensitive data . In particular , this means that the data itself is not private and could potentially be used –for example by the owner of the application– to train other algorithms with no privacy guarantees . An alternative and stronger definition of privacy is Local Differential Privacy ( LDP ) [ 10 ] . This requires that the user ’ s data is protected at collection time before the learning agent has access to it . This covers scenario and implies that the learner is DP . Intuitively , in RL , LDP ensures that each sample ( states and rewards associated to an user ) is already private when observed by the learning agent , while JDP requires computation on the entire set of samples to be DP . Recently , [ 11 ] showed that , in contrast to DP , LDP is compatible with regret minimization in contextual bandits.1 LDP is thus a stronger definition of privacy , simpler to understand and more user friendly . These characteristics make LDP more suited for real-world applications . However , as we show in this paper , guaranteeing LDP in RL makes the learning problem more challenging . Contributions . In this paper , we study LDP for regret minimization in finite horizon reinforcement learning problems with S states , A actions , and a horizon of H .2 Our contributions are as follows . 1 ) We provide a regret lower bound for ( ε , δ ) -LDP of Ω ( H √ SAK/min { eε − 1 , 1 } ) , showing LDP is inherently harder than JDP , where the lower-bound is only Ω ( H √ SAK + SAH log ( KH ) /ε ) [ 9 ] . 2 ) We propose the first LDP algorithm for regret minimization in RL . We use a general privacypreserving mechanism to perturb information associated to each trajectory and derive LDP-OBI , an optimistic model-based ( ε , δ ) -LDP algorithm with regret guarantees . 3 ) We present multiple privacy-preserving mechanisms that are compatible with LDP-OBI and show that their regret is Õ ( √ K/ε ) up to some mechanism dependent terms depending on S , A , H . 4 ) We perform numerical simulations to evaluate the impact of LDP on the learning process . For comparison , we build a Thompson sampling algorithm [ e.g. , 12 ] for which we provide LDP guarantees but no regret bound . Related Work . The notion of differential privacy was introduced in [ 4 ] and is now a standard in machine learning [ e.g. , 13 , 14 , 15 ] . Several notions of DP have been studied in the literature , including the standard DP and LDP notions . While LDP is a stronger definition of privacy compared to DP , recent works have highlighted that it possible to achieve a trade-off between the two settings in terms of privacy and utility . The shuffling model of privacy [ 16 , 17 , 18 , 19 , 20 ] allows to build ( ε , δ ) -DP algorithm with an additional ( ε′ , δ′ ) -LDP guarantee ( for ε′ ≈ ε+ ln ( n ) , any δ′ > 0 where n is the number of samples ) , hence it is possible to trade-off between DP , LDP , and utility in this setting . However , the scope of this paper is ensuring ( ε , δ ) -LDP guarantees for a fixed ε . In this case , shuffling will not provide an improvement in utility ( error ) ( see Thm 5.2 in Sec . 5.1 of [ 17 ] and App . I ) . The bandit literature has investigated different privacy notions , including DP , JDP and LDP [ 5 , 6 , 21 , 7 , 22 , 23 , 11 , 24 ] . In contextual bandits , [ 7 ] derived an impossibility result for learning under DP by showing a regret lower-bound Ω ( T ) for any ( , δ ) -DP algorithm . Since the contextual bandit problem is a finite-horizon RL problem with horizon H = 1 , this implies that DP is incompatible with regret minimization in RL as well . Regret minimization in RL with privacy guarantees has only been considered in [ 9 ] , where the authors extended the JDP approach from bandit to finite-horizon 1This shows that there are peculiarities in the DP definitions that are unique to sequential decision-making problems such as RL . The discrepancy between DP and LDP in RL is due to the fact that , when guaranteeing DP , actions taken by the learner can not depend on the current state ( this would break the privacy guarantee ) . On the other hand , in the LDP setting , the user executes a policy prescribed by the learner on its end ( i.e. , directly on non-private states ) and send a privatized result ( sequence of states and rewards observed by executing the policy ) to the learner . Hence the user can execute actions based on its current state leading to a sublinear regret . 2We do not explicitly focus on preventing malicious attacks or securing the communication between the RL algorithm and the users . This is outside the scope of the paper . RL problems . They proposed a variation of UBEV [ 25 ] using a randomized response mechanism to guarantee ε-JDP with an additive cost to the regret bound . While local differential privacy [ 10 ] has attracted increasing interest in the bandit literature [ e.g. , 21 , 23 , 11 , 24 ] , it remains unexplored in the RL literature , and we provide the first contribution in that direction . Finally , outside regret minimization , DP has been studied in off-policy evaluation [ 26 ] , in control with DP guarantees on only the reward function [ 27 ] , and in distributional RL [ 28 ] . 2 Preliminaries . We consider a finite-horizon time-homogeneous Markov Decision Process ( MDP ) [ 29 , Chp . 4 ] M = ( S , A , p , r , H ) with state space S , action space A , and horizon H ∈ N+ . Every state-action pair is characterized by a reward distribution with mean r ( s , a ) supported in [ 0 , 1 ] and a transition distribution p ( ·|s , a ) over next state.3 We denote by S = |S| and A = |A| the number of states and actions . A non-stationary Markovian deterministic ( MD ) policy is defined as a collection π = ( π1 , . . . , πH ) of MD policies πh : S → A . For any h ∈ [ H ] : = { 1 , . . . , H } and state s ∈ S , the value functions of a policy π are defined as Qπh ( s , a ) = r ( s , a ) + Eπ [ ∑H i=h+1 r ( si , ai ) ] and V πh ( s ) = Q π ( s , πh ( s ) ) . There exists an optimal Markovian and deterministic policy π ? [ 29 , Sec . 4.4 ] such that V ? h ( s ) = V π ? h ( s ) = maxπ V π h ( s ) . The Bellman equations at stage h ∈ [ H ] are defined as Q ? h ( s , a ) = rh ( s , a ) +maxa′ Es′∼ph ( s , a′ ) [ V ? h+1 ( s ′ ) ] . The value iteration algorithm ( a.k.a . backward induction ) computes Q ? by applying the Bellman equations starting from stage H down to 1 , with V ? H+1 ( s ) = 0 for any s. The optimal policy is simply the greedy policy : π ? h ( s ) = arg maxaQ ? h ( s , a ) . By boundness of the reward , all value functions V πh ( s ) are bounded in [ 0 , H − h+ 1 ] for any h and s. The general interaction protocol . The learning agent ( e.g. , a personalization service ) interacts with an unknown MDP with multiple users in a sequence of episodes k ∈ [ K ] of fixed length H . At each episode k , an user uk arrives and their personal information ( e.g. , location , gender , health status , etc . ) is encoded by the state s1 , k . The learner selects a policy πk that is sent to the user uk for local execution on “ clear ” states . The outcome of the execution , i.e. , a trajectory , Xk = ( skh , akh , rkh , sk , h+1 ) h∈ [ H ] is sent to the learner to update the policy . Note that we have not yet explicitly taken into consideration privacy in here . We evaluate the performance of a learning algorithm A which plays policies π1 , . . . , πK by its cumulative regret after K episodes ∆ ( K ) = K∑ k=1 ( V ? 1 ( s1 , k ) − V πk1 ( s1 , k ) ) . ( 1 )

Summary: This paper studied the regret analysis of reinforcement learning under the tabular setting and the LDP (local differential privacy) constraint. The authors first showed a minimax lower bound that, unlike the additive dependence on $\varepsilon$ under the JDP (joint differential privacy) constraint, the stronger LDP constraint must incur a minimax regret multiplicative in $1/\varepsilon$. Then the authors provided a general paradigm of the upper bound analysis: find private versions of aggregated statistics (rewards, number of visits), and then carry out an optimistic policy (called the LDP-OBI algorithm) based on (UCBs of) these noisy statistics. Depending on the usage of different private mechanisms, a regret multiplicative in $\sqrt{K}/\varepsilon$ could be obtained, but with worse dependence on other parameters $(H,S,A)$. Discussions and numerical experiments on the result are also provided.