Title: Tight Bounds for Machine Unlearning via Differential Privacy
Venue: NeurIPS 2023
Accepted: no
Keywords: machine unlearning

differential privacy

privacy
Abstract: We consider the formulation of "machine unlearning" of Sekhari, Acharya, Kamath, and Suresh (NeurIPS 2021), which formalizes the so-called "right to be forgotten" by requiring that a trained model, upon request, should be able to 'unlearn' a number of points from the training data, as if they had never been included in the first place. Sekhari et al. established some positive and negative results about the number of data points that can be successfully unlearnt by a trained model without impacting the model's accuracy (the "deletion capacity"), showing that machine unlearning could be achieved by using differentially private (DP) algorithms. However, their results left open a gap between upper and lower bounds on the deletion capacity of these algorithms: our work fully closes this gap, obtaining tight bounds on the deletion capacity achievable by DP-based machine unlearning algorithms.

Abstract

We consider the formulation of "machine unlearning" of Sekhari, Acharya, Kamath, and Suresh (NeurIPS 2021), which formalizes the so-called "right to be forgotten" by requiring that a trained model, upon request, should be able to 'unlearn' a number of points from the training data, as if they had never been included in the first place. Sekhari et al. established some positive and negative results about the number of data points that can be successfully unlearnt by a trained model without impacting the model's accuracy (the "deletion capacity"), showing that machine unlearning could be achieved by using differentially private (DP) algorithms. However, their results left open a gap between upper and lower bounds on the deletion capacity of these algorithms: our work fully closes this gap, obtaining tight bounds on the deletion capacity achievable by DP-based machine unlearning algorithms.

1 Introduction

Machine learning models trained on user data are now routinely used virtually everywhere, from recommendation systems to predictive models. In many cases, this user data itself includes some sensitive information (e.g., healthcare or race) or private aspects (customer habits, geographic data), sometimes even protected by law. To address this issue - that the models trained on sensitive datasets must not leak personal or private information - in a principled fashion, one of the leading frameworks is that of differential privacy (DP) [Dwork et al., 2006], which has de facto become the standard for privacy-preserving machine learning over the past decade.

At its core, DP requires that the output of a randomized algorithm M not change drastically if one to modify one of the datapoints: that is, if X,X ′ are two datasets only differing in one user's data, then for all possible outputs S of the algorithm one should have roughly the same probability of observing

S under both inputs:

where

ε >

0

and

δ

∈

(0

,

1]

quantify the privacy guarantee (the smaller values, the better the privacy;

see Section 2 for formal definitions). Intuitively, an algorithm

M

being

(

ε, δ

)

-DP means that its output does not reveal much about any particular user's data, since the output would be nearly identical had

this user's data been completely different.

While the use of differential privacy can mitigate many privacy concerns, it does come with some limitations. The first is the overhead in brings: that is, ensuring differential privacy for a learning

task typically incurs an overhead in the number of data points needed to achieve the same accuracy 31

guarantee. Perhaps more importantly, DP does not solve all possible privacy concerns: even if a ML 32

model is trained on a sensitive dataset in a differentially private way, the dataset may still be subject 33

to some attacks - e.g., if the server where the training data is stored is itself compromised. Somewhat 34

tautologically: DP is not a silver bullet, and only provides meaningful guarantees against the threat 35 models it was meant to address. 36

Another type of concerns focuses on the individual right to maintain control on one's own data : 37 broadly speaking, this is asking that each user can (under some reasonable circumstances) require that 38 their personal data and information be removed from a company's collected data and trained models. 39 This so-called 'right to be forgotten,' which allow people to request that their data be deleted entirely 40 from an ML system, has been passed into legislation or is considered in some form or another by 41 various countries or entities, prominently the European Union's General Data Protection Regulation 42 (GDPR), the California Privacy Rights Act (CCRA), Canada's proposed Consumer Privacy Protection 43 Act (CPPA), and most recently in Australia [Karp, 2023]. 44

45

46

47

48

49

50

51

52

53

54

55

56

However, translating this 'right to be forgotten' into practice comes with a host of challenges, starting with how to formalize it [Cohen et al., 2022] and technically implement it - which recently led to a new area of research in ML and computer science, that of machine unlearning . A naive technical solution would be for a given company to keep the original training set at all times, and, upon a deletion request by a user, remove this user's data from the set before retraining the whole model on the result. This, of course, comes up with two major drawbacks: first, the cost to the company, in terms of time and computational resources, of retraining a large model on a regular basis. Second, the privacy cost , as keeping the training set for an indefinite time in order to be able to handle the deletion requests leaves the door open to potential attacks and data breaches. Fortunately, there have been, over the past few years, a flurry of better (and more involved) approaches to machine unlearning, to handle deletion requests much more efficiently, and requiring to maintain much less of the training set (see, e.g., [Bourtoule et al., 2021, Nguyen et al., 2022], and related work below).

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

The above discussion, still, brings to light an important question: is machine unlearning, paradoxically, at odds with (differential) privacy? What is the connection between the two notions: are they complementary, or is there a trade-off between them?

This is the main question this work sets out to address. Our starting point is the formalization of machine unlearning set forth by Sekhari, Acharya, Kamath, and Suresh [Sekhari et al., 2021], itself reminiscent of the definition of DP (see Definition 2.5 for the formal statement): a pair of algorithms ( A, ¯ A ) is an ( ε, δ ) -unlearning algorithm if (1) A : X ∗ → W is a (randomized) learning algorithm which, given a dataset X ⊆ X ∗ , outputs model parameters A ( X ) ∈ W ; and (2) ¯ A : X ∗ × W × T → W which, on input a set of deletion requests U ⊆ X , previous model parameters w , and some succinct additional 'side information' T ( X ) ∈ T about the original dataset, output updated model parameters w ′ ∈ W from which the data from U has been unlearned, that is, such that and

for every possible set W ⊆ W of model parameters. Loosely speaking, this requires that the outcomes of (a) training a model M via A on the dataset X then unlearning some of the original training data U ⊆ X from M using ¯ A , and (b) training a model M ′ via A directly on the dataset X \ U then unlearning nothing via ¯ A , be nearly indistinguishable.

In their paper, Sekhari et al. [Sekhari et al., 2021] focus on genralization guarantees of unlearning algorithm, i.e., what can be achieved by unlearning algorithms when focusing on population loss, namely, when aiming to minimize

given a prespecified loss function f : W×X → R , where the expectation is over the draw of a new datapoint from the underlying distribution p on the sample space. The quality of a learning algorithm A is then measured by the expected excess risk

where the expectation is taking over the random choice of a dataset X ∼ D n of size n , and the 80 randomness of A itself. The focus of [Sekhari et al., 2021], as is ours, is then to quantify the deletion 81 capacity achievable for ( ε, δ ) -unlearning given a prespecified loss function, that is, the maximum 82

number of data points one can ask to be forgotten (maximum size of the subset U ) before the excess 83 risk increases by more than some threshold (see Definition 2.6). 84

In their paper, [Sekhari et al., 2021] draw a connection between DP learning algorithms and unlearning 85 ones, showing that DP learning algorithms do provide some unlearning guarantees out-of-the-box, 86 and that one can achieve non-trivial unlearning guarantees for convex loss functions by leveraging 87 the literature on differentially private optimization and learning. One of their main results is showing 88 that these DP-based unlearning algorithms, which crucially do not rely on any side information 89 (the additional input T ( X ) ∈ T provided to the unlearning algorithm ¯ A ) can handle strictly fewer 90 deletion requests than general unlearning algorithms which do rely on such side information. 91

92

93

94

95

96

97

98

Their results, however, do not fully characterize the deletion capacity of these 'DP-based' machine unlearning algorithms, leaving a significant gap between their upper and lower bounds. We argue that fully understanding this quantity is crucial, as DP-based unlearning algorithms are exactly those for which there is no conflict between the two notions of DP and unlearning instead, this class of algorithms is the one for which they work hand in hand. This is in contrast to the more general unlearning algorithms relying on maintaining and storing side information about the training set, as this side information can make their deployment susceptible to privacy breaches.

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

1.1 Our contributions

The main contribution of our paper is a tight bound on the 'amount of unlearning' achievable by any machine unlearning algorithm which does not rely on side information. For the sake of exposition, we state in this section informal versions of our results.

Theorem 1.1 (Unlearning For Convex Loss Functions (Informal; see Theorems 3.1 and 3.3)) . Let f : W×X → R be a 1 -Lipschitz convex loss function, where W ⊆ R d is the parameter space. There exists an ( ε, δ ) -machine unlearning algorithm which, trained on a dataset S ⊆ X n , does not store any side information about the training set besides the learned model, and can unlearn up to

datapoints without incurring excess population risk greater than α . Moreover, this is tight: there exists a 1 -Lipschitz linear loss function such that no machine unlearning algorithm can unlearn Ω( nεα √ d log(1 /δ ) ) data points without excess population risk α , unless it stores side information.

Our techniques also allow us to easily establish the analogue for strongly convex optimization:

Theorem 1.2 (Unlearning For Strongly Convex Loss Functions (Informal)) . Let f : W×X → R be a 1 -Lipschitz strongly convex loss function. There exists an ( ε, δ ) -machine unlearning algorithm which, trained on a dataset S ⊆ X n , does not store any side information about the training set besides the learned model, and can unlearn up to

datapoints without incurring excess population risk greater than α . Moreover, this is tight.

We note that, prior to our work, only bounds for the convex loss function case were known, with an upper bound of m = ˜ O ( nεα/ √ d log( e ε /δ )) (loose by polylogarithmic factors for ε = O (1) , as well as an 1 / √ ε factor for ε ≫ 1 ) and a limited lower bound stating that m ≥ 1 is only possible if nε/ √ d = Ω(1) .

Our next contribution, motivated by the similarity of the formalisations of machine unlearning (without side information) and that of differential privacy, is to establish the analogue of key properties of DP for machine unlearning, namely, post-processing and composition of machine unlearning algorithms. To do so, we first identify a natural property of machine unlearning algorithms, which, when satisfied, will allow for the composition properties:

Assumption 1.3 (Unlearning Laziness) . An unlearning algorithm ( ¯ A,A ) is said to be lazy if, when provided with an empty set of deletion requests, the unlearning algorithm ¯ A does not update the model. That is, ¯ A ( ∅ , A ( X ) , T ( X )) = A ( X ) for all datasets X .

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

We again emphasize that this laziness property is not only intuitive, it is also satisfied by several existing unlearning algorithms, and in particular those proposed in Sekhari et al. [2021].

Theorem 1.4 (Post-processing of unlearning) . Let ( ¯ A,A ) be an ( ε, δ ) -unlearning algorithm taking no side information. Let f : W → W be an arbitrary (possibly randomized) function. Then ( f ◦ ¯ A,A ) is also an ( ε, δ ) -unlearning algorithm.

Under our laziness assumption, we also establish the following:

Theorem 1.5 (Chaining of unlearning) . Let ( ¯ A,A ) be a lazy ( ε, δ ) -unlearning algorithm taking no side information, and able to handle up to m deletion requests. Then, the algorithm which uses ( ¯ A,A ) to sequentially unlearn k disjoint deletion requests U 1 , . . . , U k ⊆ X such that | ∪ i U i | ≤ m , outputting

is an ( ε ′ , δ ′ ) -unlearning algorithm, with ε ′ = kε and δ ′ = δ · e kε -1 e ε -1 = O ( kδ ) (for k = O (1 /ε ) ).

and, finally,

Theorem 1.6 (Advanced composition of unlearning) . Let ( ¯ A 1 , A ) , . . . , ( ¯ A k , A ) be lazy ( ε, δ ) -unlearning (with common learning algorithm A ) taking no side information, and define the composition of those unlearning algorithms, ˜ A as

where f : W k → W is any (possibly randomized) function. Then, for every δ ′ > 0 , ( ˜ A,A ) is an ( ε ′ , δ ′ ) -unlearning taking no side information, where ε ′ = k 2 ε 2 + ε √ 2 k ln (1 /δ ′ ) .

1.2 Related work

Albeit recent, the field of machine unlearning has already received considerable attention from the ML community, with an array of studies and papers focusing on practical solutions and their empirical performance. We focus in this section on the works most closely related to ours, mostly theoretical. As discussed earlier, the goal of machine unlearning (Bourtoule et al. [2021]) is to delete what models have learned from data. This problem coincides tangentially with the idea of differential privacy as they both requires to minimize the effect of a (or a group of) sample. The original, stringent definition of unlearning requires ε = 0 (full deletion of the user's data, as if it had never been included in the training set in the first place) in contrast to differential privacy that allows ε > 0 , leaving a possibility for 'memorization.' To relax this definition, Ginart et al. [2019] proposed the probabilistic version of unlearning.

Prior theoretical work of unlearning are mostly disjoint from the differential privacy literature, in spite of a general recognition that the two notions aim to address related issues. Most works on machine unlearning mainly focus on empirical risk minimization of approximate unlearning algorithms (Guo et al. [2020], Izzo et al. [2020]), which seeks to find an approximate minimizer on the remaining dataset after deletion. Closest to our work is the recent paper of Sekhari et al. [2021], which formulated the notion of machine unlearning used in our paper and focused on population loss minimization of approximating unlearning algorithm (i.e., allowing ε > 0 ). Their objectives, however, were somewhat orthogonal to ours, as they focused for a large part on minimizing the space requirements for the side information T ( X ) provided to the unlearning algorithm (while our paper focuses on algorithms which do not rely on any such side information, prone to potential privacy leaks). While their work, to motivate this focus, established partial bounds on the deletion capacity of unlearning algorithm that do not take in extra statistics, these bounds were not tight, and one of our main contributions is closing this gap. Following Sekhari et al. [2021], the notion of online unlearning algorithm - which receive the deletion requests sequentially - was put forward and studied in Suriyakumar and Wilson [2022], again with memory efficiency with respect to the side information in mind; however, their primary focus is on the empirical performance of unlearning algorithm.

Another work closely to ours is the notion of certified data removal proposed by Guo et al. [2020]. 172 The main difference between ( ε, δ ) -certified removal and the definition from Sekhari et al. [2021] is 173 that, in the former, the unlearning mechanism requires access not only to the samples to be deleted 174 (the set U ⊆ X ), but also to the full original training set X : this is exactly the type of constraints our 175 work seeks to avoid, due to the risk of data breach this entails. 176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

1.3 Organization of the paper

We first provide the necessary background and notion on differential privacy, learning, and the formulation of machine unlearning used throughout the paper in Section 2. We then provide a detailed outline of the proof of our main result, Theorem 1.1, in Section 3, before concluding with a discussion of results and future work in Section 4.

Due to space constraints, the details of all other results, as well as omitted proofs, are deferred to the Supplemental.

2 Preliminaries

In this section, we recall some notions and results we will extensively rely on in our proofs and theorems, starting with differential privacy.

2.1 Differential Privacy

Definition 2.1 ((Central) Differential Privacy (DP)) . Fix ε > 0 and δ ∈ [0 , 1] . An algorithm M : X n →Y satisfies ( ε, δ ) -differential privacy (DP) if for every pair of neighboring datasets X,X ′ , and every (measurable) subset S ⊆ Y :

We further say that M satisfies pure differential privacy ( ε -DP) if δ = 0 , otherwise it is approximate differential privacy.

We now recall another notion of differential privacy in terms of Renyi Divergence, from Bun and Steinke [2016].

Definition 2.2 (Zero-Concentrated Differential Privacy (zCDP)) . Arandomized algorithm M : X n → Y satisfies ( ξ, ρ ) -zCDP if for every neighboring datasets (differing on a single entry) X,X ′ ∈ X n , and ∀ α ∈ (1 , ∞ ) :

where D is the α -Renyi divergence between distributions of M ( X ) and M ( X ′ ) . We say that M is ρ -zCDP when ξ = 0 .

We use the following group privacy property of zCDP in the proof later.

Proposition 2.3 ( k -distance group privacy of ρ -zCDP [Bun and Steinke, 2016, Proposition 1.9]) . Let M : X n →Y satisfy ρ -zCDP. Then, M is ( k 2 ρ ) -zCDP for every X,X ′ ∈ X n that differs in at most k entries.

2.2 Learning

We also will require some definitions on learning, specifically with respect to minimizing population loss. Fix any loss function f : W×X , where W is the (model) parameter space and X is the sample space. Then, the generalization loss is defined as

in which the expectation is over the distribution of x (one sample) and w is the learning output. Let F ∗ = min w ∈W F ( w ) be the minimizer of population risk and w ∗ is the corresponding minimizer.

Define learning algorithm A : X n → W that takes in dataset S ∈ X n and returns hypothesis w := A ( S ) ∈ W . The excess risk is given by:

where the expectation is over the randomness of A and S .

Hence, we could define the sample complexity as following ([Sekhari et al., 2021, Definition 1]), which is analogous to deletion capacity, in which will be stated later.

Definition 2.4 (Sample complexity of learning) . The α -sample complexity of a problem is defined as:

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

2.3 Unlearning

As previously discussed, we rely on the definition of unlearning proposed in by Sekhari et al. [2021], and maintain same notation. Note that T ( S ) denotes the data statistics (which could be the entire dataset S or any form of statistic) available to ¯ A .

Definition 2.5 ( ( ε, δ ) -unlearning) . For all S of size n and delete requests U ⊆ S such that | U | ≤ m , and W ⊆ W , a learning algorithm A and an unlearning algorithm ¯ A is ( ε, δ ) -unlearning if:

Our results will be phrased in terms of the deletion capacity, which captures the number of deletion requests an unlearning algorithm can handle before seeing a noticeable drop in its output's accuracy: Definition 2.6 (Deletion Capacity) . Let ε, δ > 0 , S be a dataset of size n drawn i.i.d. from D and let ℓ ( w,z ) be a loss function. For a pair of learning and unlearning algorithm A, ¯ A that are ( ε, δ ) -unlearning, the deletion capacity m A, ¯ A ε,δ is defined as the maximum size of deletions requests set | U | that we can unlearn without doing worse in excess population risk than α :

} where F ∗ := min A ( S ) ∈B F ( ¯ A ( U, A ( S ) , T ( S ))) .

3 Main result

In this section, we provide a detailed outline of our main result on unlearning for convex loss functions, Theorem 1.1, for which we prove the upper and lower bounds separately.

Theorem 3.1 (Deletion capacity from unlearning via DP, Lower Bound) . Suppose W ⊆ R d , and fix any Lipschitz convex loss function. Then there exists a lazy ( ε, δ ) -unlearning algorithm ( ¯ A,A ) , where ¯ A has the form ¯ A ( U, A ( S ) , T ( S )) := A ( S ) (and thus, in particular, takes no side information) with deletion capacity

where the constant in the Ω( · ) only depends on the properties of the loss function.

Proof. Our proof follows the same general outline as that of Sekhari et al. [2021], with a key difference which allows us to derive the tight bound. Namely, we start, similarly, from the observation that any ( ε, δ ) -DP learning algorithm A whose privacy guarantee is with respect to edit distance m between datasets (instead of the usual neighboring relation) readily implies an ( ε, δ ) -machine unlearning algorithm ( ¯ A,A ) , where ¯ A ( w ) = w (i.e., the 'unlearning part' returns its input unchanged).

However, we depart from previous work by how we obtain this ( ε, δ ) -DP algorithm A with respect to edit distance m . The key insight is that instead of starting with any good approximate DP learning algorithm and using the grouposition property of DP to 'upgrade' it to m -edit distance, we instead start with a good zCDP learning algorithm. Indeed, zCDP has much tigher grouposition properties than approximate DP (cf. Proposition 2.3), which in turn leads to better parameters when applying grouposition to achieve DP to groups up to size m : specifically, starting with a ρ 2 / 2 -zCDP standard privacy guarantee (for groups of size 1 ) we would by Proposition 2.3 obtain ( m 2 ρ 2 / 2) -zCDP for neighboring datasets differin in up to m entries. Leveraging then the standard conversion from concentrated to approximate DP [Bun and Steinke, 2016], this implies, for every δ > 0 , an ( ε, δ ) -DP

√

(

= Θ

guarantee for groups of size

m

, where

ε

=

O

(

mρ

log (1

/δ

))

. Thus, choosing

ρ

m

ε

ln (1

/δ

)

would suffice to achieve the desired end privacy guarantee on A (with respect to edit distance up to m ), and thus the ( ε, δ ) -unlearning one for ( ¯ A,A ) .

To do so, however, we crucially need to start with a sufficiently good private learning algorithm A with zCDP guarantees, instead of approximate DP. Fortunately for us, such an algorithm is provided by [Feldman et al., 2020, Theorem 1]:

)

√

Lemma 3.2 (zCDP mini-batch noisy SGD Feldman et al. [2020]) . Fix any L -Lipschitz convex loss 258 function over a convex subset B of R d of diameter D . Then there exists an algorithm A which satisfies 259 ( ρ 2 / 2) -zCDP with excess population loss: 260

where the expectation is taken over the randomness of A .

By the above discussion, using this zCDP-private learning algorithm with ρ = Θ ( ε m √ ln (1 /δ ) ) , we get an excess population loss bounded by

It only remains to show how the claimed deletion capacity bound frollows from this excess population 264 risk guarantee. Construct, as discussed earlier, an unlearning algorithm ¯ A that returns the input 265 without making any changes (and in particular does not require any additional statistics T ( S ) , and 266 satisfies the laziness assumption). Since A is ( ε, δ ) -DP, for any set U ⊆ S, | U | = m , and W ⊆ W , 267

. But since ¯ A ( U, A ( S )) = A ( S ) , this readily yields, letting S ′ := S \ U : 268

which implies that ( A, ¯ A ) is indeed ( ε, δ ) -unlearning for U of size (up to) m .

Recalling the definition of deletion capacity (Definition 2.6), we finally deduce from (2) the deletion capacity with excess population risk less than α :

where the O ( · ) hides constant factors depending only on the loss function (namely, the Lipschitz function L , and the diameter D ).

Theorem 3.3 (Deletion capacity from unlearning via DP, Upper Bound) . There exists a Lipschitz convex loss function (indeed, linear ) for which any ε, δ ) -unlearning algorithm ( ¯ A,A ) which takes no side information must have deletion capacity

Detailed Proof Sketch. We will consider the following linear (and therefore convex and Lipschitz) 277 loss function L ( θ, S ) : 278

for dataset S of n points x 1 , . . . , x n ∈ {-1 √ d , 1 √ d } d . We also define the 1-way marginal query, i.e. 279 average, as: 280

To establish our deletion capacity lower bound with respect to this loss function, we will proceed 281 in three stages: the first, relatively standard, is to relate population loss (what we are interested in) 282 to empirical loss - which allows us to focus on the existence of a 'hard dataset.' The second step 283

261

262

263

269

270

271

272

273

274

275

276

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

is then to establish a sample complexity lower bound on the empirical risk (for this loss function) of any ( ε, δ ) -DP algorithm, via a reduction to differentially private computing of 1-marginals. This step is similar to the one underlying the (weaker) lower bound of Sekhari et al. [2021] (itself relying on an argument of [Bassily et al., 2019]), although a more careful choice of building blocks for the reduction already allows us to obtain an improvement by logarithmic factors.

Third, lift this DP lower bound to a stronger lower bound for DP with respect to edit distance m . This step is quite novel, as it morally corresponds to establishing the converse of the grouposition property of differential privacy (for our specific setting), a converse which does not hold in general. Our argument, relatively simple, will crucially rely on the linearity of our loss function.

We omit the details of the first step (reduction from population to empirical loss) in this detailed outline, as it is quite standard. For the second step, our starting point is the following lower bound of Steinke and Ullman:

Theorem 3.4 (Lower bound for one-way marginals [Steinke and Ullman, 2016, Main Theorem]) . For every ε ∈ (0 , 1) , every function δ = δ ( n ) such that δ ≥ 2 -o ( n ) and δ ≤ 1 /n 1+Ω(1 ) , and for every α ≤ 1 / 10 , if A : {± 1 } n × d → [ ± 1] d is ( ε, δ ) -differentially private and E [ ∥A ( S ) -q ( S ) ∥ 1 ] ≤ αd (i.e., with average-case accuracy α ) for all S ∈ {± 1 } n × d , then we must have

Using this lower bound as a blackbox, we then can adapt the argument of [Bassily et al., 2014, Lemma 5.1, Part 2] to obtain the following stronger result:

Lemma 3.5 (Lower bound for Privately Computing 1-way Marginals) . Let n, d ∈ N , ε > 0 , 2 -on ≤ δ ( n ) ≤ 1 /n 1+Ω(1) . For all α ≤ 1 / 10 , if A is ( ε, δ ) -differentially private and S ⊆ {± 1 √ d } n × d :

where q ( S ) = 1 n ∑ n i =1 x i as before.

Combining the above with the argument strategy of [Bassily et al., 2014, Theorem 5.3] finally yields the main lemma for the second step of our proof:

Lemma 3.6 (Lower bound on empirical loss of ( ε, δ ) -DP algorithms) . Let n, d ∈ N , ε > 0 , and δ = o (1 /n ) . For every ( ε, δ ) -differentially private algorithm with output θ priv , there is a dataset S = { x 1 , . . . , x n } ⊆ {-1 √ d , 1 √ d } d such that

The above lemma establishes a lower bound on the empirical loss of any ( ε, δ ) -differentially private algorithm. To derive from this our claimed lower bound on unlearning algorithms, we need to introduce a dependence on m , the deletion capacity (i.e., number of points to unlearn). This is done in the last (third) step of our argument, via a reduction which establishes a (restricted) converse to the grouposition property of DP.

Recall that an algorithm M : X n → Y satisfies ( ε, δ ) -DP for edit distance m if for every pair of neighboring datasets X,X ′ that differ in up to m entries , and every S ⊆ Y :

Weapply this m -edit distance ( ε, δ ) -DP on Lemma 3.6 by a reduction that shows: for any differentially private algorithm with respect to edit distance at most m must incur an empirical loss given by Lemma 3.6.

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

Lemma 3.7. Suppose there exists an m -edit distance ( ε, δ ) -DP algorithm M that takes in a dataset S of size n to approximate q ( S ) (as defined in (4) ), with empirical loss γ . Then, we can construct a 1 -edit distance (i.e., standard) ( ε, δ ) -DP algorithm M ′ that, on input a dataset S ′ of N = n/m data points, approximates q ( S ′ ) to error γ .

Proof. The reduction is quite simple: given M , construct M ′ as follows for N = n m inputs:

We immediately have that M ′ is ( ε, δ ) -DP for the usual 1 -edit distance between datasets, since M is DP with respect to edit distance m . The sample complexity and error bound then follows direction from n = N × m , where n ≥ N,N ∈ N , m ≥ 1 , and the fact that q ( x 1 , . . . , x N ) = q ( x 1 , . . . , x 1 , x 2 , . . . , x 2 , . . . , x N , . . . , x N ) due to the definition of q .

Combining Lemma 3.7 with Lemma 3.6, we get that any m -edit distance ( ε, δ ) -DP algorithm M approximating q on datasets of size n = N × m must have error γ at least

which, reorganising the terms and recalling the definition of deletion capacity, yields the claimed bound on m A, ¯ A ε,δ .

We note that the proof of Theorem 1.2 follows from a very similar argument; we refer the reader to the Supplemental for details.

4 Discussion and future work

Our work fully characterized deletion capacity of any unlearning algorithm ( ¯ A,A ) minimizing population risk under both convex and strongly convex loss functions, when only given the model parameters (output of the learning algorithm) and the set of deletion requests. This restriction, namely that the unlearning algorithm does not rely on any additional side information, is motivated by the potential privacy risks storing (non-private) side information can entail.

We hope our work will lead to further study of the interplay between differential privacy and machine unlearning, and to additional study of 'DP-like' properties of machine unlearning, such as the postprocessing and composition properties our present work identified. In view of the myriad applications these properties have had in privacy-preserving algorithm design, we believe that their analogue for machine unlearning will prove very useful.

We leave for future work the question of which unlearning guarantees can be obtained from pure differentially private algorithms, and of whether variants of the standard threat model for differential privacy (specifically, pan-privacy, or privacy under continual observation) could have implications for machine unlearning in an online setting where deletion requests come sequentially.

References

Raef Bassily, Adam D. Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 55th IEEE Annual Symposium on Foundations of Computer Science, FOCS , pages 464-473. IEEE Computer Society, 2014. doi: 10.1109/FOCS.2014.56.

Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems , pages 11279-11288, 2019.

Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 42nd IEEE Symposium on Security and Privacy, SP 2021, San Francisco, CA, USA, 24-27 May 2021 , pages 141-159. IEEE, 2021. doi: 10.1109/SP40001.2021.00019. URL https://doi.org/10.1109/ SP40001.2021.00019 .

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Martin Hirt and Adam D. Smith, editors, Theory of Cryptography - 14th International Conference, TCC 2016-B, Beijing, China, October 31 - November 3, 2016, Proceedings, Part I , volume 9985 of Lecture Notes in Computer Science , pages 635-658, 2016. doi: 10.1007/978-3-662-53641-4\_24.

Aloni Cohen, Adam D. Smith, Marika Swanberg, and Prashant Nalini Vasudevan. Control, confidentiality, and the right to be forgotten. CoRR , abs/2210.07876, 2022.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Conference on Theory of Cryptography , TCC '06, pages 265-284, Berlin, Heidelberg, 2006. Springer.

Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , STOC 2020, page 439-449, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369794. doi: 10.1145/3357713.3384335.

Antonio A. Ginart, Melody Y. Guan, Gregory Valiant, and James Y. Zou. Making ai forget you: Data deletion in machine learning. In Neural Information Processing Systems , 2019.

Chuan Guo, Tom Goldstein, Awni Y. Hannun, and Laurens van der Maaten. Certified data removal from machine learning models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 3832-3842. PMLR, 2020. URL http://proceedings.mlr.press/ v119/guo20c.html .

Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Y. Zou. Approximate data deletion from machine learning models: Algorithms and evaluations. CoRR , abs/2002.10077, 2020. URL https://arxiv.org/abs/2002.10077 .

Paul Karp. Australia to consider european-style right to be forgotten privacy laws. The Guardian , 2023. URL https://www.theguardian.com/australia-news/2023/jan/19/ right-to-be-forgotten-australia-europe-gdpr-privacy-laws .

Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. CoRR , abs/2209.02299, 2022. doi: 10.48550/arXiv.2209.02299. URL https://doi.org/10.48550/arXiv.2209.02299 .

Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems , volume 34, pages 18075-18086. Curran Associates, Inc., 2021.

Thomas Steinke and Jonathan R. Ullman. Between pure and approximate differential privacy. J. Priv. Confidentiality , 7(2), 2016. doi: 10.29012/jpc.v7i2.648.

Vinith M. Suriyakumar and Ashia C. Wilson. Algorithms that approximate data removal: New results and limitations. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/ 2022/hash/77c7faab15002432ba1151e8d5cc389a-Abstract-Conference.html .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the machine unlearning problem from the perspective of differential privacy. Specifically, the authors propose to use differentially private models directly so that unlearning update is not necessary (or unlearning is an identity map), and the motivation is to make the unlearning procedure independent of side information (i.e., original training set) to avoid privacy leakage. A tight lower bound on the number of data points that such kind of DP model can be unlearned is shown in the paper, along with some new proving techniques.\n\nSTRENGTHS:\nThis paper uses the idea from Renyi DP and zero-concentrated DP to optimize the DP parameters, and thus refine the lower bound result proposed in previous work by Sekhari et al. An interesting combination of off-the-shelf results from DP literature is used to obtain the improvement of deletion capacity lower bound from $\\widetilde{\\Omega}\\left(\\frac{n \\varepsilon}{\\sqrt{d \\log \\left(e^\\varepsilon/ \\delta\\right)}}\\right)$ to $\\Omega\\left(\\frac{n \\varepsilon}{\\sqrt{d \\log (1 / \\delta)}}\\right)$. Furthermore, a deletion capacity upper bound is studied when the loss function is linear, showing that the proposed lower bound is actually tight.\n\nBesides the improvement in the lower bound, the authors also introduce the post-processing, chain rule, and composition theorem for unlearning analog to classical DP. This could benefit future works to study unlearning via differential privacy.\n\nWEAKNESSES:\nAlthough this paper may be interesting in theory, I do not think it can fit nicely into machine unlearning problems. The main motivation in the paper to use DP directly for unlearning is to avoid the usage of side information, which basically means the original training set. However, for most unlearning scenarios, having the entire training set is not a critical issue, and sometimes even a must when we need to perform model retraining. Consider the case when the size of data points that require deleting is larger than the deletion capacity. We have no choice but to retrain the model from scratch, and we need the remaining training samples to complete the retraining. So I do not think the motivation holds in the first place. Furthermore, a related discussion on the difference between DP and unlearning appears in the previous work by Sekhari et al. (Section 3.2: Strict separation between unlearning and differential privacy), where they show that if one designs the unlearning update carefully, the deletion capacity can be improved to $\\Omega(\\frac{n \\sqrt{\\varepsilon}}{(d \\log (1 / \\delta))^{1 / 4}})$, which enjoys better dependence on the dimension $d$ even compared to the results presented in this paper. As a matter of fact, many unlearning papers have pointed out the fact that using DP directly can lead to large overhead and low utility. Therefore, whether one can use the theoretical results presented in the paper to design better unlearning algorithms is questionable.\n\nQUESTIONS:\nPlease see the weakness section for more details.\n\nLIMITATIONS:\nThe results do not fit nicely with machine unlearning problems, and it would be hard to utilize the results to design unlearning algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies connections between machine unlearning and differential privacy (DP). In machine unlearning, the goal is to remove up to, say, $m$ of the examples from a dataset of size $n$, in such a way that the produced model is close (in some form of statistical or computational distance) to the model that would have been produced if we did not have the m examples to begin with.\n\nA recent paper of SAKS'21 formulates unlearning with the same style that DP is usually defined. This paper follows the same definition.\n\nThe main question studied in this paper is about \"deletion capacity\". Namely, how many examples from the dataset can we delete while we lose the accuracy of the model up to a given parameter $\\alpha$ and keep the machine unlearning \"secure\" with specified parameters $(\\epsilon,\\delta)$ (defined similarly to DP)? More formally, $\\alpha$ is the regret in the agnostic setting, which is the extra risk compared to the best model in the family.\n\nThe main result of the paper is to establish matching upper and lower bounds (up to constant factors) on the deletion capacity for algorithms that basically do not do any deletion and when certain (natural, but still limiting) properties hold on the models and the loss function. Namely, the paper studies how to achieve $(\\epsilon,\\delta)$ privacy when the comparison is made between datasets that have hamming distance $m$, rather than $1$, and while the regret is bounded by $\\alpha$.\n\nAt a technical level, the paper achieves its tight upper and lower bound (on the deletion capacity, within its own defined framework) by actually *not* caring about achieving differential privacy in its standard sense and directly aiming to satisfy the DP Lipschitz property over $m$-close databases out of the box. To achieve tight bounds the paper moves to other notions of DP (based on Reny divergence) first and then comes back to DP after a more effective composition theorem (that exist for such DP style definitions) is applied.\n\nThe paper also studies composition of unlearning, but only the statements are stated in the paper and no discussion is presented.\n\nSTRENGTHS:\nAt a technical level, studying the DP for $m$-close databases (with the motivation of doing nothing for unlearning!) is interesting, and finding tight bounds for such problem is cool. but I would have preferred a more direct depiction of the result up front by saying that what happens here is not really unlearning and is about DP for $m$-close datasets. Then, maybe depicting unlearning as a potential application would be good, since the application to unlearning comes with some limitations that prevent us from using the full capacity of what unlearning allows.\n\nThe proofs also look interesting and as far as I could tell, the difference between this work and previous work is explained well.\n\nWEAKNESSES:\nAs explained above, I think the connection to unlearning is a bit far fetched, as it comes with strong limitations.\n\n(has a related question)\nWhat I understand is that the paper studies tight bounds for settings in which the unlearning is *not even done* and the closeness of the produced models holds due to the DP-like property over $m$-close data sets. I am not sure why this is equivalent to \"not storing anything besides the model itself\". If they are equivalent, this needs a proof.\n\nCitations are not in good shape. Examples:\nThe work of Cohen et al is cited for initiating a formal study of \"the right to be forgotten\", while works like: https://eprint.iacr.org/2020/254 are done earlier.\n\nThe main question of the paper is about privacy vs unlearning, which is also studied in previous (uncited) works like\nhttps://www.usenix.org/conference/usenixsecurity20/presentation/salem\nhttps://arxiv.org/abs/2005.02205\nhttps://arxiv.org/abs/2202.03460:\n\nQUESTIONS:\nSee the question in the section above. In addition:\n\nwhy is theorem 1.6 is called a composition theorem? it seems it would be more natural for a composition theorem to allow up to k batches of deletion in an adaptive way.\n\nline 207.5: why is the distribution p and not D ?\n\nline 229: why is F* defined like this, and not like how it was defined in line 209 ?\n\nthe def of loss in line 277: why is the loss defined like that? The loss should take a model and a full *labeled* example and then output something. Your notion ignores the label (maybe x itself has the label already?) and is defined for a set (it would be a risk, in that case, and it typically takes average).\n\nLIMITATIONS:\nThe paper is clear in that they only study specific \"unearning\" methods that come with limitations. It is mentioned that the limitation is not to store anything other than the model, but my understanding is that their limitation is to not do anything when unlearning. These seem different to me, but hopefully the author(s) will clarify this limitation in rebuttal.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the concept of \"machine unlearning\" within the framework of differential privacy. The authors provide tight bounds on the maximum number of data points that can be successfully unlearned without significantly impacting the model's accuracy. They also establish the analog of key properties of DP for machine unlearning. The paper introduces novel results for convex and strongly convex loss functions, as well as properties of post-processing and composition of unlearning algorithms.\n\nSTRENGTHS:\n1. The paper addresses the important and practical problem of machine unlearning, which enables individuals to request the removal of their data from trained models.\n2. The paper builds upon previous work and provides enhanced theoretical results. It closes the gap between upper and lower bounds on the deletion capacity achievable by differentially private machine unlearning algorithms.\n\nWEAKNESSES:\n1. The paper focuses primarily on theoretical analysis and proofs, but it lacks empirical experiments to validate the proposed machine unlearning algorithms. \n2. The paper considers convex loss and strongly convex loss functions in its theoretical analysis. While these assumptions may hold for some models and applications, they may not be applicable to a wide range of real-world machine learning models, such as deep learning models, which often involve non-convex loss functions. This limitation restricts the generalizability and practical applicability of the proposed algorithms.\n\nQUESTIONS:\n1. Considering that many real-world machine learning models, such as deep learning models, involve non-convex loss functions, how applicable are the proposed machine unlearning algorithms to these models?\n2. How does the deletion capacity impact the overall utility and performance of the machine learning models in practical scenarios?\n\nLIMITATIONS:\n1. Lack of experiments.\n2. Assumptions on loss functions are constraints.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies approximate unlearning with procedures which do not store any side information (and satisfy differential privacy) in convex learning problems and establishes tight upper and lower bounds.\n\nSTRENGTHS:\n1. Machine unlearning has recently gained much interest owing to privacy regulations. The paper studies a certain formulation of approximate unlearning inspired via differential privacy, with the additional restriction of storing no side information. This formulation is particularly appealing since (a). storing side information can be expensive and prone to attacks and (b). it does not require new algorithms for learning and unlearning, but simply (existing) private algorithms. The paper studies the limits of such a formulation for convex learning problems, improving the upper and lower bounds in the previous works. The question is very natural and the authors resolve the loose ends remaining in the prior works.\n\n2. The proofs of improved upper and lower bound are simple yet interesting: the improved upper bound follows due to the use of stronger group-privacy properties of zCDP, as opposed to approximate DP; and the lower bound establishes an interesting \"converse to group-privacy\" due the linear loss function under consideration.\n\nWEAKNESSES:\n1. While the problem is very natural, the final quantitative improvements are rather minor. For constant $\\epsilon$, which is usually the case of DP, the bounds are same. There is also very limited discussion on the quantitative improvements compared to prior work.\n\n2. The scope of the paper seems limited; the paper essentially ties some loose ends present in the prior analysis, which is mostly interesting for theoretical reasons. I don't know if the contributions could have impact on the larger area of machine unlearning.\n\n3. The techniques largely borrow from the differential privacy literature. The lower bound instance and the argument of padding the sample multiple times are present in prior works.\n\nQUESTIONS:\nI would encourage the authors to provide an extended discussion of the quantitative improvements compared to prior work.\n\nLIMITATIONS:\nThe scope of the the work is limited to procedures which store no side information and satisfy approximate unlearning (in the spirit of DP). These restrictions basically leave DP procedures as candidate learning/unlearning algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work derives tight bounds for the deletion capacity of machine unlearning algorithms that are differentially private. These bounds are stated in terms of a deletion capacity formulated as a function of data points that can be removed before the estimation risk (an accuracy measure) becomes too large. The estimation risk (or the excess risk) is assessed over the sampling distribution of i.i.d. data points. The authors derived the results in detail for 1-Lipchitz convex loss functions and presented briefly the parallel result for 1-L strongly convex loss functions.\n\nMy assessment, consisting of strengths, weaknesses, and questions, can be found in the sections below.\n\nSTRENGTHS:\nI find this a well-written paper. For a technical topic that is focused on the proof, the paper nevertheless walks the reader through while offering both clarity and insight. The structure of the paper is also quite reasonable, with the contribution overview very helpful.\n\nWEAKNESSES:\nPlease see my question in the Limitation section. I think the lack of comment for that question in this paper is its biggest weakness.\n\nQUESTIONS:\nMy biggest concern has been raised as a question in the Limitation section, because it is more appropriate under that heading. Below are a few questions on the technical side that could benefit from some clarification.\n\n- Your estimation risk, and the definition of deletion capacity, calls for an expectation over the sampling variability of i.i.d. data points following a distribution D.  I would like to understand better the impact that this assumption has, in particular the independence part, on your results. In particular, how would data dependence change the results? This is an important point of difference between your framework and DP which do not take data variability into consideration (except Pufferfish which you do not use).\n- Related to this, perhaps you could consider including in the sketch proof of Theorem 3.3 (Line 293) some details concerning the reduction from population to empirical losses. I imagine that the issue of data variability comes up here.\n- I am failing to appreciate the significance of requiring the data points to take values in \\pm 1/\\sqrt{d}, where d is the dimension of the parameter space. If we are discussing an asympototic regime (in which the bounds are situated, since they involve both d and n), what does this “shrinking” ball of data mean?\n- A minor point: the word “grouposition” can use some clarification. I think I know what the authors mean but this is not standard vocabulary.\n\nLIMITATIONS:\nLooking at the form of the unlearning algorithm \\bar{A} in Theorem 3.1, am I to conclude that the unlearning algorithm that is to achieve the tight bound as you present in this work is in fact an algorithm that literally *does nothing* to the deletion request, i.e. it outputs A(S) just as before? If my understanding is correct, this point calls for a moral discussion: yes the learning-unlearning pair satisfies Definition 2.5, but with the relaxation of alpha, epsilon, and delta, this technical argument will provide a slippery slope for justified inaction. Maybe your paper is not the first to consider this, but one more paper gets published without paying due attention to these ethical questions, the literature grows a bit more oblivious to common sense.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-01"}