Title: Generalized Linear Bandits with Local Differential Privacy 
Authors: Yuxuan Han
Zhipeng Liang
Yang Wang
Jiheng Zhang
Venue: NeurIPS 2021
Keywords: Local Differential Privacy
Contextual Bandits
Online Decision Making

Abstract: Contextual bandit algorithms are useful in personalized online decision-making. However, many applications such as personalized medicine and online advertising require the utilization of individual-specific information for effective learning, while user's data should remain private from the server due to privacy concerns. This motivates the introduction of local differential privacy (LDP), a stringent notion in privacy, to contextual bandits. In this paper, we design LDP algorithms for stochastic generalized linear bandits to achieve the same regret bound as in non-privacy settings. Our main idea is to develop a stochastic gradient-based estimator and update mechanism to ensure LDP. We then exploit the flexibility of stochastic gradient descent (SGD), whose theoretical guarantee for bandit problems is rarely explored, in dealing with generalized linear bandits. We also develop an estimator and update mechanism based on Ordinary Least Square (OLS) for linear bandits. Finally, we conduct experiments with both simulation and real-world datasets to demonstrate the consistently superb performance of our algorithms under LDP constraints with reasonably small parameters $(\varepsilon, \delta)$ to ensure strong privacy protection. 


Review discussion:
First, I must admit I'm not an expert in the area of differential privacy though I've done some work in bandits.
Strengths:
+ The design and analysis of the SGD-based method for ensuring LDP in contextual generalized linear bandits seems to be rather innovative. The SGD-based mechanism is a rather natural algorithm. I am wondering why other natural ideas for linear bandits like LinUCB (or its generalization to generalized linear bandits) fail to ensure LDP. This should probably be discussed.
+ The empirical results on synthetic datasets look very promising.
Weaknesses:
- While the bounds are tight up to the constants, in the absence of the privacy constraint, one can also obtain asymptotically optimal algorithms that match up to the constants (cf. KL-UCB). I am wondering how far the constants in the lower and upper bounds are.
- Theorem 3.3 for the problem-dependent case seems to be lacking a lower bound. Same is true for Thm 4.1. Authors should comment on whether lower bounds can be derived here.
- The writing and math are not particularly careful (and can be very careless) in some places.
For example, it's not clear to me what's h_sub in Assumption 5.
In Assumption 6, "This is the same assumption as 4". What is 4?
- In the last column of Fig 2, it seems like LDP-SGD's regret grows linearly. Is there something problematic here? At the very least, the regret should be concave (log T shape or \sqrt{T} shape).
No
Thanks a lot for recognizing the novelty of our SGD-based estimator in generalized linear bandits and finding our empirical results on synthetic impressive.
We hope the following response can address your remaining concerns.
**Q1: Why other natural ideas for linear bandits like LinUCB (or its generalization to generalized linear bandits) fail to ensure LDP?**
A1: We have mentioned this point in our paper that a variant of LinUCB and GLOC can guarantee LDP [Zheng2020] (line 267-268).
However, the price of providing LDP guarantee in the LinUCB algorithm is to incur an extra $O(T^{\frac{1}{4}})$ factor to the non-private regret (line 42, line 189).
Our paper is partially motivated by this variant of LinUCB and tries to fill in this gap.
The reason why LinUCB can only achieve $O(T^{3/4})$ regret bound is that the privacy noise would enforce linUCB to maintain a larger confidence region for the parameter and thus leads to over-exploration.
In fact, comparing with the $O(\sqrt{d\log(1+t)})$ radius of confidence ellipsoid in [Yadkori11], arguments in [Zheng2020] shows that with LDP noise the radius turns to be $O( \sqrt{d\log(1+t)}+(dt)^{1/4} )$, that's the cause of the extra $T^{1/4}$ factor in their regret bound.
In our paper, we show that the private noise will not influence the error bound of the adaptively updated parameter estimator with respect to $t$ too much so that the regret is of the same order as in non-private case (e.g., in linear bandit, it changes from $O(\frac{\sqrt{d}}{\sqrt{t}})$ in non-private case to $O(\frac{d}{\varepsilon \sqrt{t}}$).
**Q2: How far the upper bounds in our paper and the corresponding lower bounds are? (For points 1 and 2)**
A2: Firstly, we would re-clarify that our worst-case bound in single-parameter case (Theorem 3.1) is optimal in $T$ up to $\log\log T$, which is comparable to the $O(\sqrt{T}\log T)$ bounds in non-private setting with similar assumptions as in ours (e.g. [Sivakumar2020], [Ding2021]).
To the best of our knowledge, no existing literature can close the worst-case upper and lower bounds up to a constant in the setting similar to our single param case.
For regret bounds under general margin conditions in single and multi param cases, few works consider the lower bounds even in non-private cases.
[Hu2019] and [Rigollet2010] consider the lower bound under similar margin conditions for non-parametric bandits.
However, their constructions do not work for parametric classes.
[Goldenshluger2013] considers the lower bound in a similar setting for two-armed bandit as in our multi param case when $\beta$ is specialized to be 1.
Their argument uses the Van-Tree inequality to bound the mean squared estimation error from below for each time $t$.
It is possible to combine the LDP-version Van-Tree inequality in [Barnes2020] with the argument in [Goldenshluger2013] for our problem to derive the $\Omega(\dfrac{\log T}{\varepsilon^2})$ bound.
We shall add this comment to our paper for clarity.
Finally, as mentioned above, to our best knowledge, there exists no literature considering asymptotical optimality in our setting.
Papers considering asymptotical optimality in bandit learning is usually under different assumptions:
One line of these works (e.g., the KL-UCB) considers the asymptotic optimality for non-contextual MAB.
Another line is under the assumption that there are only finite contexts over the whole learning horizon (see, e.g. [Lattimore2017] for fixed action set and [Hao2020] for possible varying action set but with only finite contexts).
In our setting, the context of users can be different at every different time $t$.
So our results are incomparable to these asymptotically optimal results.
Considering the influence of privacy constraints on asymptotical optimality in the settings of [Lattimore2017] and [Hao2020] is indeed an exciting further direction.
**Q3: The writing and math are not careful.**
A3: Thanks for pointing out several typos in our paper. In Assumption 5, "for some h>0" in line 246 should be changed to "for some $h_{sub}>0$".
Also, in Assumption 6, "This is the same assumption as 4" should be changed to "This is the same as Assumption 4". We will fix these two typos and perform a thorough check of the whole paper.
**Q4: Why the cumulative regret line seems growing linearly in the last column of Fig 2?**
A4: Due to the limited size of the real dataset, we can only set the time horizon 1/10 as in the synthetic dataset, and thus the curve in real data seems to grow linearly.
Moreover, as we can see in the figure, the performance of our LDP-SGD is indeed significantly superior to that of LDP-GLOC, and we believe it still maintains a lower growth rate when the time horizon is even longer.
**References:**
[Yadkori11]: Yasin Abbasi-Yadkori et al. "Improved Algorithms for Linear Stochastic Bandits"
[Sivakumar2020]: Vidyashankar Sivakumar et al. "Structured Linear Contextual Bandits: A Sharp and Geometric Smoothed Analysis"
[Hu2019]: Yichun Hu et al. "Smooth Contextual Bandits: Bridging the Parametric and Non-differentiable Regret Regimes"
[Rigollet2010]: Philippe Rigollet et al. "Non-parametric Bandits with Covariates"
[Goldenshluger2013]: Alexander Goldenshluger et al. "A LINEAR RESPONSE BANDIT PROBLEM"
[Barnes2020]: Leighton Pate Barnes et al. "Fisher information under local differential
privacy"
[Lattimore2017]: Tor Lattimore et al. "The End of Optimism? An Asymptotic Analysis of Finite-Armed Linear Bandits"
[Hao2020]: Botao Hao et al. "Adaptive Exploration in Linear Contextual Bandit"
[Ding2021]:  Qin Ding et al. "An Efficient Algorithm For Generalized Linear Bandit: Online Stochastic Gradient Descent and Thompson Sampling"
[Zheng2020]: Kai Zheng et al. "Locally Differentially Private (Contextual) Bandits Learning"
Dear authors,
Thanks for the clarifications on my points. They are mostly clear. I'm happy to up my score to 5.
Reviewer
The paper is an interesting paper taking a pretty standard approach but one that is likely to be highly applicable (user-level privacy). The results are also nice and are likely to add to the ever-growing literature on private bandit problems.
All in all, it is a decent paper and worth presenting at NeurIPS.
No
Thank you so much for your review and supportive comments on the contributions of our paper.
Review remain unaltered.
Originality:
Several novel and surprising contributions.
The paper newly introduces the SGD estimator to bandit algorithms. It first studies the LDP problem in stochastic contextual bandits. By developing different estimators and updating mechanism, the regret bound is proved to be the same, with respect to the order of T, with the non-privacy setting.
Quality:  Technically adequate for its area, solid results.
In synthetic setting of the experiments part, the dimension d=2 is too small for a contextual setting and the number of arms K is also too small which is only set to 3 and 10. I recommend the authors to test on a larger synthetic setting to verify the results.
Also, for the adopted real dataset, the corresponding parameters should be reported.
Clarity:  The paper is mostly clear to read.
Typo:  In line 102, there is an additional ‘and’?
Significance:  The paper has solid contribution to relevant problem. The authors propose LDP contextual bandit frameworks in both single-parameter and multi-parameter settings to deal generalized linear reward structure. And the proposed algorithms are armed with the theoretical guarantee, which achieves the same order of T with the non-privacy setting, and experimental verifications.
No
Thanks for your review and valuable comments. Please find our response to the comments below:
**Q1: The dimension and number of arms are too small, and the corresponding parameters in the real-data experiment should be reported.**
A1: Thanks for your suggestions on our experiment part. We have added more experiments following your advice:
We increase $K$ and $d$ to $20$ in single param experiments and increase $K$ and $d$ to $10$ in multi param experiments. We validate algorithms' performance under $T=10^5$ and $\varepsilon=0.5,1$.
We report the mean and the confidence interval with 0.5 standard deviations of their final cumulative regrets across 10 replications.
In conclusion, our LDP-OLS and LDP-SGD algorithms can still achieve superior performance when K and d increase.
We will add these experiment results in our next version.
|           |       | Single param ($K=20, d=20$) | Multi param ($K=10, d=10$) |
| :-----    | :-----:  | :-----: | :-----: |
| **$\varepsilon=0.5$** | LDP-UCB | 43470.6 (40031.1, 46910.0) | 139727.7 (133931.0, 145524.4)  |
|           | LDP-GLOC | 41000.0 (40973.2, 41026.9) | 148761.8 (147637.3, 149886.3) |
|           | LDP-OLS | 38134.8 (34224.0, 42045.5) |  107514.5 (95606.9, 119422.2) |
|           | LDP-SGD | 26966.9 (24467.7, 29466.2) | 99691.0 (93158.7, 106223.3) |
| **$\varepsilon = 1$**   | LDP-UCB | 41001.0 (40969.4, 41032.6) | 147986.3 (147078.5, 148894.1) |
|           | LDP-GLOC |40712.6 (37455.4, 43969.7)  |  113276.8 (109132.3, 117421.3) |
|           | LDP-OLS | 36196.4 (32044.1, 40348.7) | 91477.0 (86153.1, 96800.9) |
|           | LDP-SGD | 18271.1 (16944.1, 19598.1) | 76127.3 (71008.7, 81245.8) |
As for the report of the corresponding paramters in the real-data experiment, please check our Appendix C.
**Q2: Typo in line 102.**
A2: Thanks for pointing out the typo in line 102!
We will fix it in the future version and perform a thorough check of the whole paper.

Meta-review:
role: PC
data: comment: This paper develops LDP algorithms for contextual bandits. The algorithms are novel, problem is timely, and the results are interesting. The paper should be of interest to both bandit/online learning as well as privacy researchers.  

Decision: Accept (Poster)