Title: Certified Minimax Unlearning with Generalization Rates and Deletion Capacity
Venue: NeurIPS 2023
Accepted: yes
Keywords: machine unlearning

machin learning privacy

minimax learning

certified removal
Abstract: We study the problem of $(\epsilon,\delta)$-certified machine unlearning for minimax models. Most of the existing works focus on unlearning from standard statistical learning models that have a single variable and their unlearning steps hinge on the direct Hessian-based conventional Newton update. We develop a new $(\epsilon,\delta)$-certified machine unlearning algorithm for minimax models. It proposes a minimax unlearning step consisting of a total Hessian-based complete Newton update and the Gaussian mechanism borrowed from differential privacy. To obtain the unlearning certification, our method injects calibrated Gaussian noises by carefully analyzing the ''sensitivity'' of the minimax unlearning step (i.e., the closeness between the minimax unlearning variables and the retraining-from-scratch variables). We derive the generalization rates in terms of population strong and weak primal-dual risk for three different cases of loss functions, i.e., (strongly-)convex-(strongly-)concave losses. We also provide the deletion capacity to guarantee that a desired population risk can be maintained as long as the number of deleted samples does not exceed the derived amount. With training samples $n$ and model dimension $d$, it yields the order $\mathcal O(n/d^{1/4})$, which shows a strict gap over the baseline method of differentially private minimax learning that has $\mathcal O(n/d^{1/2})$. In addition, our rates of generalization and deletion capacity match the state-of-the-art rates derived previously for standard statistical learning models.

Certified Minimax Unlearning with Generalization Rates and Deletion Capacity

Abstract

We study the problem of ( ϵ, δ ) -certified machine unlearning for minimax models. Most of the existing works focus on unlearning from standard statistical learning models that have a single variable and their unlearning steps hinge on the direct Hessian-based conventional Newton update. We develop a new ( ϵ, δ ) -certified machine unlearning algorithm for minimax models. It proposes a minimax unlearning step consisting of a total Hessian-based complete Newton update and the Gaussian mechanism borrowed from differential privacy. To obtain the unlearning certification, our method injects calibrated Gaussian noises by carefully analyzing the 'sensitivity' of the minimax unlearning step (i.e., the closeness between the minimax unlearning variables and the retraining-from-scratch variables). We derive the generalization rates in terms of population strong and weak primal-dual risk for three different cases of loss functions, i.e., (strongly-)convex-(strongly-)concave losses. We also provide the deletion capacity to guarantee that a desired population risk can be maintained as long as the number of deleted samples does not exceed the derived amount. With training samples n and model dimension d , it yields the order O ( n/d 1 / 4 ) , which shows a strict gap over the baseline method of differentially private minimax learning that has O ( n/d 1 / 2 ) . In addition, our rates of generalization and deletion capacity match the state-of-the-art results derived previously for standard statistical learning models.

1 Introduction

Minimax models have been widely applied in a variety of machine learning applications, including generative adversarial networks [Goodfellow et al., 2014, Arjovsky et al., 2017], adversarially robust learning [Madry et al., 2018, Sinha et al., 2018], and reinforcement learning [Du et al., 2017, Dai et al., 2018]. This is largely credited to the two-variable (i.e., primal and dual variables) model structure of minimax models, which is versatile enough to accommodate such diverse instantiations. As is common in machine learning practice, training a successful minimax model relies crucially on a potentially large corpus of training samples that are contributed by users. This raises privacy concerns for minimax models. Unlike standard statistical learning (STL) models, the privacy studies for minimax models are relatively newer. Most of the existing studies focus on privacy protection during the training phase under the differential privacy (DP) notion [Dwork et al., 2006] and federated minimax learning settings [Sharma et al., 2022]. Recent works in this direction have successfully achieved several optimal generalization performances measured in terms of the population primal-

dual (PD) risk for DP minimax models specifically [Yang et al., 2022, Zhang et al., 2022a, Bassily et al., 2023, Boob and Guzmán, 2023].

Machine unlearning is an emerging privacy-respecting problem concerning already-trained models (i.e., during the post-training phase) [Cao and Yang, 2015, Guo et al., 2020, Sekhari et al., 2021, Graves et al., 2021, Bourtoule et al., 2021, Li et al., 2021, Shibata et al., 2021, Wu et al., 2022, Cheng et al., 2023, Chen et al., 2023, Tarun et al., 2023, Wu et al., 2023, Ghazi et al., 2023, Wang et al., 2023b]. That is, it removes certain training samples from the trained model upon their users' data deletion requests. It is driven by the right to be forgotten, which is mandated by a growing number of user data protection legislations enacted in recent years. Prominent examples include the European Union's General Data Protection Regulation (GDPR) [Mantelero, 2013], the California Consumer Privacy Act (CCPA), and Canada's proposed Consumer Privacy Protection Act (CPPA). Machine unlearning comes with several desiderata. Besides sufficiently removing the influence of the data being deleted, it should be efficient and avoid the prohibitive computational cost of the baseline method to fully retrain the model on the remaining dataset from scratch. To guarantee the sufficiency of data removal, there are exact machine unlearning methods [Cao and Yang, 2015, Ginart et al., 2019, Brophy and Lowd, 2021, Bourtoule et al., 2021, Ullah et al., 2021, Schelter et al., 2021, Chen et al., 2022b,a, Yan et al., 2022, Di et al., 2023, Xia et al., 2023] and approximate machine unlearning methods [Golatkar et al., 2020a, Wu et al., 2020, Golatkar et al., 2020b, Nguyen et al., 2020, Neel et al., 2021, Peste et al., 2021, Golatkar et al., 2021, Warnecke et al., 2023, Izzo et al., 2021, Mahadevan and Mathioudakis, 2021, Mehta et al., 2022, Zhang et al., 2022c, Wang et al., 2023a, Chien et al., 2023a, Lin et al., 2023] (some can offer the rigorous ( ϵ, δ ) -certification [Guo et al., 2020, Sekhari et al., 2021, Suriyakumar and Wilson, 2022, Chien et al., 2023b] inspired by differential privacy). In addition, recent studies also point out the importance of understanding the relationship between the generalization performance and the amount of deleted samples [Sekhari et al., 2021, Suriyakumar and Wilson, 2022]. In particular, they introduce the definition of deletion capacity to formally quantify the number of samples that can be deleted for the after-unlearning model to maintain a designated population risk. However, most existing works so far have focused on machine unlearning for standard statistical learning models with one variable, which leaves it unknown how to design a minimax unlearning method to meet all the desiderata above.

Machine unlearning for minimax models becomes a pressing problem because the trained minimax models also have a heavy reliance on the training data, while the users contributing data are granted the right to be forgotten. In this paper, we study the machine unlearning problem for minimax models under the ( ϵ, δ ) -certified machine unlearning framework. We collect in Table 1 the results in this paper and comparisons with baseline methods that are adapted from previous papers to ( ϵ, δ ) -certified machine unlearning.

Our main contributions can be summarized as follows.

Certified minimax unlearning algorithm: We develop ( ϵ, δ ) -certified minimax unlearning algorithm under the setting of the strongly-convex-strongly-concave loss function. To sufficiently remove the data influence, the algorithm introduces the total Hessian consisting of both direct Hessian and indirect Hessian, where the latter is crucial to account for the inter-dependence between the primal and dual variables in minimax models. It leads to the complete Newton-based minimax

unlearning update. Subsequently, we introduce the Gaussian mechanism from DP to achieve the ( ϵ, δ ) -minimax unlearning certification, which requires careful analysis for the closeness between the complete Newton updated variables and the retraining-from-scratch variables.

Generalization: We provide generalization results for our certified minimax unlearning algorithm in terms of the population weak and strong primal-dual risk, which is a common generalization measure for minimax models.

Deletion capacity: We establish the deletion capacity result, which guarantees that our unlearning algorithm can retain the generalization rates for up to O ( n/d 1 / 4 ) deleted samples. It matches the state-of-the-art result under the STL unlearning setting that can be regarded as a special case of our minimax setting.

Extension to more general losses: We extend the certified minimax unlearning to more general loss functions, including convex-concave, strongly-convex-concave, and convex-strongly-concave losses, and provide the corresponding ( ϵ, δ ) -certification, population primal-dual risk, and deletion capacity results.

Extension with better efficiency: We develop a more computationally efficient extension, which can also support successive and online deletion requests. It saves the re-computation of the total Hessian matrix during the unlearning phase, where the minimax unlearning update can be regarded as a total Hessian-based infinitesimal jackknife. It also comes with slightly smaller population primal-dual risk though the overall rates of the risk and deletion capacity remain the same.

2 Related work

Machine unlearning receives increasing research attention in recent years, mainly due to the growing concerns about the privacy of user data that are utilized for machine learning model training. Since the earliest work by Cao and Yang [2015], a variety of machine unlearning methods have been proposed, which can be roughly divided into two categories: exact unlearning and approximate unlearning.

Exact machine unlearning. Methods for exact machine unlearning aim to produce models that perform identically to the models retrained from scratch. Some exact unlearning methods are designed for specific machine learning models like k-means clustering [Ginart et al., 2019] and random forests [Brophy and Lowd, 2021]. SISA [Bourtoule et al., 2021] proposes a general exact unlearning framework based on sharding and slicing the training data into multiple non-overlapping shards and training independently on each shard. During unlearning, SISA retrains only on the shards containing the data to be removed. GraphEraser [Chen et al., 2022b] and RecEraser [Chen et al., 2022a] further extend SISA to unlearning for graph neural networks and recommendation systems, respectively.

Approximate machine unlearning. Approximate machine unlearning methods propose to make a tradeoff between the exactness in data removal and computational/memory efficiency. Prior works propose diverse ways to update the model parameter and offer different types of unlearning certification. When it comes to the unlearning update, many existing works consider the Newton update-related unlearning step where the Hessian matrix of the loss function plays a key role [Guo et al., 2020, Golatkar et al., 2020a, Peste et al., 2021, Sekhari et al., 2021, Golatkar et al., 2021, Mahadevan and Mathioudakis, 2021, Suriyakumar and Wilson, 2022, Mehta et al., 2022, Chien et al., 2023b]. This unlearning update is motivated by influence functions [Koh and Liang, 2017]. In order to alleviate the computation of the Hessian, Golatkar et al. [2020a] and Peste et al. [2021] utilize Fisher Information Matrix to approximate the Hessian, mitigating its expensive computation and inversion. Mehta et al. [2022] provide a variant of conditional independence coefficient to select sufficient sets for unlearning, avoiding the need to invert the entire Hessian matrix. ML-forgetting [Golatkar et al., 2021] trains a linear weights set on the core dataset which would not change by standard training and a linear weights set on the user dataset containing data to be forgotten. They use an optimization problem to approximate the forgetting Newton update. Suriyakumar and Wilson [2022] leverage the proximal infinitesimal jackknife as the unlearning step in order to be applied to nonsmooth loss functions. In addition, they can achieve better computational efficiency and are capable of dealing with online delete requests. There are also many other designs achieving different degrees of speedup [Wu et al., 2020, Nguyen et al., 2020, Neel et al., 2021, Zhang et al., 2022c].

Apart from the various designs for the unlearning update, there are also different definitions of certified machine unlearning. Early works like Guo et al. [2020] introduce a certified data-removal

mechanism that adds random perturbations to the loss function at training time. Golatkar et al. [2020a] introduce an information-theoretic-based certified unlearning notion and also add random noise to ensure the certification, which is specific to the Fisher Information Matrix and not general enough. More recently, Sekhari et al. [2021] propose the ( ϵ, δ ) -certified machine unlearning definition that does not require introducing additional randomization during training. More essential, Sekhari et al. [2021] points out the importance of providing the generalization performance after machine unlearning. Sekhari et al. [2021], Suriyakumar and Wilson [2022] establish the generalization result in terms of the population risk and derive the deletion capacity guarantee.

However, most of existing works only consider machine unlearning for STL models that minimize a single variable. None of the prior works provide certified machine unlearning pertaining to minimax models, for which the generalization and deletion capacity guarantees are still unknown.

3 Preliminaries and Baseline Solution

3.1 Minimax Learning

The goal of minimax learning is to optimize the population loss F ( ω , ν ) , given by

where f : W × V × Z → R is the loss function, z ∈ Z is a data instance from the distribution D , W ⊆ R d 1 and V ⊆ R d 2 are closed convex domains with regard to primal and dual variables, respectively. Since the data distribution D is unknown in practice, minimax learning turns to optimize the empirical loss F S ( ω , ν ) , given by,

where S = { z 1 , · · · , z n } is the training dataset with z i ∼ D .

We will consider L -Lipschitz, ℓ -smooth and µ ω -strongly-convexµ ν -strongly-concave loss functions, which are described in Assumption 1&2 below and more details can be found in Appendix A.

Assumption 1. For any z ∈ Z , the function f ( ω , ν ; z ) is L -Lipschitz and with ℓ -Lipschitz gradients and ρ -Lipschitz Hessians on the closed convex domain W×V . Moreover, f ( ω , ν ; z ) is convex on W for any ν ∈ V and concave on V for any ω ∈ W .

Assumption 2. For any z ∈ Z , the function f ( ω , ν ; z ) satisfies Assumtion 1 and f ( ω , ν ; z ) is µ ω -strongly convex on W for any ν ∈ V and µ ν -strongly concave on V for any ω ∈ W .

Denote a randomized minimax learning algorithm by A : Z n →W×V and its trained variables by A ( S ) = ( A ω ( S ) , A ν ( S )) ∈ W × V . The generalization performance is a top concern of the trained model variables ( A ω ( S ) , A ν ( S )) [Thekumparampil et al., 2019, Zhang et al., 2020, Lei et al., 2021, Farnia and Ozdaglar, 2021, Zhang et al., 2021, 2022b, Ozdaglar et al., 2022], which can be measured by population weak primal-dual (PD) risk or population strong PD risk, as formalized below.

Definition 1 ( Population Primal-Dual (PD) Risk ) . The population weak PD risk of A ( S ) , △ w ( A ω ( S ) , A ν ( S )) and the population strong PD risk of A ( S ) , △ s ( A ω ( S ) , A ν ( S )) are defined as

Notations. We introduce the following notations that will be used in the sequel. For a twice differentiable function f with the arguments ω ∈ W and ν ∈ V , we use ∇ ω f and ∇ ν f to denote the direct gradient of f w.r.t. ω and ν , respectively and denote its Jacobian matrix as ∇ f = [ ∇ ω f ; ∇ ν f ] . We use ∂ ωω f , ∂ ων f , ∂ νω f , ∂ νν f to denote the second order partial derivatives w.r.t. ω and ν , correspondingly and denote its Hessian matrix as ∇ 2 f = [ ∂ ωω f, ∂ ων f ; ∂ νω f, ∂ νν f ] . We define the total Hessian of the function f w.r.t. ω and ν : D ωω f := ∂ ωω f -∂ ων f · ∂ -1 νν f · ∂ νω f and D νν f := ∂ νν f -∂ νω f · ∂ -1 ωω f · ∂ ων f where ∂ -1 νν f and ∂ -1 ωω f are the shorthand of ( ∂ νν f ( · )) -1 and ( ∂ ωω f ( · )) -1 , respectively, when ∂ νν f and ∂ ωω f are invertible. We also use the shorthand notation ∇ ω f ( ω 1 , ν ; z ) = ∇ ω f ( ω , ν ; z ) | ω = ω 1 .

3.2 ( ϵ, δ ) -Certified Machine Unlearning

An unlearning algorithm ¯ A for minimax models receives the output of a minimax learning algorithm A ( S ) , the set of delete requests U ⊆ S and some additional memory variables T ( S ) ∈ T as input and returns an updated model ( ω u , ν u ) = ( ¯ A ω ( U, A ( S ) , T ( S )) , ¯ A ν ( U, A ( S ) , T ( S ))) ∈ W × V , aiming to remove the influence of U . For the memory variables in T ( S ) , it will not contain the entire training set, but instead its size | T ( S ) | is independent of the training data size n . The mapping of an unlearning algorithm can be formulated as ¯ A : Z m × W × V × T → W × V . We now give the notion of ( ϵ, δ ) -certified unlearning introduced by Sekhari et al. [2021], which is inspired by the definition of differential privacy [Dwork et al., 2006].

Definition 2 ( ( ϵ, δ ) -Certified Unlearning [Sekhari et al., 2021] ) . Let Θ be the domain of W×V . For all S of size n , set of delete requests U ⊆ S such that | U | ≤ m , the pair of learning algorithm A and unlearning algorithm ¯ A is ( ϵ, δ ) -certified unlearning, if ∀ O ⊆ Θ and ϵ, δ > 0 , the following two conditions are satisfied:

where ∅ denotes the empty set and T ( S ) denotes the memory variables available to ¯ A .

The above definition ensures the indistinguishability between the output distribution of (i) the model trained on the set S and then unlearned with delete requests U and (ii) the model trained on the set S \ U and then unlearned with an empty set. Specifically, the unlearning algorithm simply adds perturbations to the output of A ( S \ U ) when the set of delete requests is empty.

Deletion Capacity. Under the definition of certified unlearning, Sekhari et al. [2021] introduce the definition of deletion capacity, which formalizes how many samples can be deleted while still maintaining good guarantees on test loss. Here, we utilize the population primal-dual risk defined in Definition 1 instead of the excess population risk utilized for STL models.

Definition 3 ( Deletion capacity, [Sekhari et al., 2021] ) . Let ϵ, δ, γ > 0 and S be a dataset of size n drawn i.i.d from the data distribution D . Let F ( ω , ν ) be a minimax model and U be the set of deletion requests. For a pair of minimax learning algorithm A and minimax unlearning algorithm ¯ A that satisfies ( ϵ, δ ) -unlearning, the deletion capacity m A, ¯ A ϵ,δ,γ ( d 1 , d 2 , n ) is defined as the maximum number of samples U that can be unlearned while still ensuring the population primal-dual (weak PD or strong PD) risk is at most γ . Let the expectation E [ · ] takes over S ∼ D n and the outputs of the algorithms A and ¯ A . Let d 1 denotes the dimension of domain W and d 2 denotes the dimension of domain V , specifically,

where the ouputs ¯ A ω ( U, A ( S ) , T ( S )) and ¯ A ν ( U, A ( S ) , T ( S )) of the minimax unlearning algorithm ¯ A refer to parameter ω and ν , respectively. △ ( ¯ A ω ( U, A ( S ) , T ( S )) , ¯ A ν ( U, A ( S ) , T ( S )) ) could be the population weak PD risk or population strong PD risk of ¯ A ( U, A ( S ) , T ( S )) .

We set γ = 0 . 01 (or any other small arbitrary constant) throughout the paper.

3.3 Baseline Solution: Certified Minimax Unlearning via Differential Privacy

Since Definition 2 is motivated by differential privacy (DP), it is a natural way to use tools from DP for machine unlearning. For a differentially private learning algorithm A with edit distance m in neighboring datasets, the unlearning algorithm ¯ A simply returns its output A ( S ) without any changes and is independent of the delete requests U as well as the memory variables T ( S ) , i.e., ¯ A ( U, A ( S ) , T ( S )) = A ( S ) .

A number of differentially private minimax learning algorithms can be applied, e.g., Zhang et al. [2022a], Yang et al. [2022], Bassily et al. [2023]. For instance, we can obtain the output A ( S ) = ( A ω ( S ) , A ν ( S )) by calling Algorithm 3 in Zhang et al. [2022a]. Under Assumption 1&2, we then get the population strong PD risk based on [Zhang et al., 2022a, Theorem 4.3] and the group privacy property of DP [Vadhan, 2017, Lemma 7.2.2], as follows,

where we let µ = min { µ ω , µ ν } , κ = ℓ/µ , d = max { d 1 , d 2 } , and m be the edit distance between datasets (i.e., the original dataset and the remaining dataset after removing samples to be forgotten).

The algorithm A satisfies ( ϵ, δ ) -DP for any set U ⊆ S of size m , that is,

Since we have A ( S ) = ¯ A ( U, A ( S ) , T ( S )) and A ( S \ U ) = ¯ A ( ∅ , A ( S \ U ) , T ( S \ U )) , the above privacy guarantee can be converted to the minimax unlearning guarantee in Definition 2, implying that the pair ( A, ¯ A ) is ( ϵ, δ ) -certified minimax unlearning. According to Definition 3, the population strong PD risk in eq.(7) yields the following bound on deletion capacity.

Theorem 1 ( Deletion capacity of unlearning via DP [Sekhari et al., 2021] ) . Denote d = max { d 1 , d 2 } . There exists a polynomial time learning algorithm A and unlearning algorithm ¯ A for minimax problem of the form ¯ A ( U, A ( S ) , T ( S )) = A ( S ) such that the deletion capacity is:

where the constant in ˜ Ω -notation depends on the properties of the loss function f (e.g., strongly convexity and strongly concavity parameters, Lipchitz continuity and smoothness parameters).

However, this DP minimax learning baseline approach provides an inferior deletion capacity. In the following sections, we show that the d 1 / 2 in the denominator of eq.(8) can be further reduced to d 1 / 4 .

4 Certified Minimax Unlearning

In this section, we focus on the setting of the strongly-convex-strong-concave loss function. We first provide the intuition for the design of the minimax unlearning step in Sec.4.1, then provide the formal algorithm in Sec.4.2 and a more efficient extension in Sec.4.3 with analysis of minimax unlearning certification, generalization result, and deletion capacity in Sec.4.4. We will provide extensions to more general loss settings in Sec.5. The proofs for the theorems presented in this and the next sections can be found in Appendix B and C, respectively.

4.1 Intuition for Minimax Unlearning Update

To begin with, we provide an informal derivation for minimax unlearning update to illustrate its design intuition. Given the training set S of size n and the deletion subset U ⊆ S of size m , the aim is to approximate the optimal solution ( ω ∗ S \ , ν ∗ S \ ) of the loss F S \ ( ω , ν ) on the remaining dataset S \ U , given by,

Meanwhile, we have the optimal solution ( ω ∗ S , ν ∗ S ) to the original loss F S ( ω , ν ) after minimax learning. Taking unlearning ω for instance, by using a first-order Taylor expansion for ∇ ω F S \ ( ω ∗ S \ , ν ∗ S \ ) = 0 around ( ω ∗ S , ν ∗ S ) , we have

Since ω ∗ S is a minimizer of F S ( ω , ν ) , from the first-order optimality condition, we can get ∇ ω F S \ ( ω ∗ S , ν ∗ S ) = -1 n -m ∑ z i ∈ U ∇ ω f ( ω ∗ S , ν ∗ S ; z i ) . Now given an auxiliary function V S \ ( ω ) = argmax ν ∈V F S \ ( ω , ν ) (more best response auxiliary functions are introduced in Appendix A, Definition 8), we have ν ∗ S \ = V S \ ( ω ∗ S \ ) . We further get

where the approximate equation ( i ) leaving out the term [ V S \ ( ω ∗ S ) -ν ∗ S ] which is bounded in Appendix A, Lemma 2, and does not affect the overall unlearning guarantee. The approximate equation ( ii ) is the linear approximation step and is the response Jacobian of the auxiliary function V S \ ( ω ) . The approximate equation ( iii ) is due to the implicit function theorem. This gives that

which implies the following approximation of ω ∗ S \ :

The above informal derivation indicates that the minimax unlearning update relies on the total Hessian to sufficiently remove the data influence [Liu et al., 2023, Zhang et al., 2023], rather than the conventional Hessian that appears in standard statistical unlearning [Guo et al., 2020, Sekhari et al., 2021, Suriyakumar and Wilson, 2022, Mehta et al., 2022]. The update in eq.(13) has a close relation to the complete Newton step in the second-order minimax optimization literature [Zhang et al., 2020], which motivates the complete Newton-based minimax unlearning. However, due to the various approximations in the above informal derivation, we cannot have a certified minimax unlearning guarantee. Below, we will formally derive the upper bound for these approximations in the closeness upper bound analysis. Based on the closeness upper bound, we will introduce the Gaussian mechanism to yield distribution indistinguishably result in the sense of ( ϵ, δ ) -certified minimax unlearning.

4.2 Proposed Certified Minimax Unlearning

We first provide algorithms under the setting of the smooth and strongly-convex-strongly-concave (SC-SC) loss function as described in Assumptions 1&2.

Algorithm 1 Minimax Learning Algorithm ( A sc -sc )

Input: Dataset S : { z i } ∼ D , loss function: f ( ω , ν ; z ) .

1: Compute

Output: ( ω ∗ S , ν ∗ S , D ωω F S ( ω ∗ S , ν ∗ S ) , D νν F S ( ω ∗ S , ν ∗ S ))

Algorithm 2 Certified Minimax Unlearning for Strongly-Convex-Strongly-Concave Loss ( ¯ A sc -sc )

Input: Delete requests U : { z j } j m =1 ⊆ S , output of A sc -sc ( S ) : ( ω ∗ S , ν ∗ S ) , memory variables T ( S ) : { D ωω F S ( ω ∗ S , ν ∗ S ) , D νν F S ( ω ∗ S , ν ∗ S ) } , loss function: f ( ω , ν ; z ) , noise parameters: σ 1 , σ 2 .

1: Compute

2: Define

3: ω u = ̂ ω + ξ 1 , where ξ 1 ∼ N (0 , σ 1 I d 1 ) and ν u = ̂ ν + ξ 2 , where ξ 2 ∼ N (0 , σ 2 I d 2 ) . Output: ( ω u , ν u ) .

n i =1 n

Minimax Learning algorithm. We denote our learning algorithm by A sc -sc and the pseudocode is shown in Algorithm 1. Given a dataset S = { z i } n i =1 of size n drawn independently from some distribution D , algorithm A sc -sc computes the optimal solution ( ω ∗ S , ν ∗ S ) to the empirical risk F S ( ω , ν ) . A sc -sc then outputs the point ( ω ∗ S , ν ∗ S ) as well as the additional memory variables T ( S ) := { D ωω F S ( ω ∗ S , ν ∗ S ) , D νν F S ( ω ∗ S , ν ∗ S ) } , which computes and stores the total Hessian of F S ( ω , ν ) at ( ω ∗ S , ν ∗ S ) .

Minimax Unlearning Algorithm We denote the proposed certified minimax unlearning algorithm by ¯ A sc -sc and present its pseudocode in Algorithm 2. Algorithm ¯ A sc -sc takes the following inputs: the set of delete requests U = { z j } j m =1 of size m , the trained minimax model ( ω ∗ S , ν ∗ S ) , and the memory variables T ( S ) . To have the certified minimax unlearning for ω , eq.(15) computes the total Hessian of F S \ ( ω ∗ S , ν ∗ S ) by n n -m D ωω F S ( ω ∗ S , ν ∗ S ) -1 n -m ∑ z i ∈ U D ωω f ( ω ∗ S , ν ∗ S , z i ) , where the former term can be retrieved from the memory set and the latter is computed on the samples to be deleted; eq.(17) computes the intermediate ̂ ω by the complete Newton step based on the total Hessian D ωω F S \ ( ω ∗ S , ν ∗ S ) ; Line 3 injects calibrated Gaussian noise ξ 1 to ensure ( ϵ, δ ) -certified minimax unlearning. The certified minimax unlearning for ν is symmetric. We provide detailed analysis for Algorithm 2 including minimax unlearning certification, generalization results, and deletion capacity in Appendix B.1.

4.3 Certified Minimax Unlearning without Total Hessian Re-computation

We extend Algorithm 2 and propose Algorithm 3 to reduce the computational cost of Algorithm 2. The complete Newton steps in eq.(19) and eq.(20) utilize the total Hessian D ωω F S ( ω ∗ S , ν ∗ S ) and D νν F S ( ω ∗ S , ν ∗ S ) that are directly retrieved from the memory, rather than the updated total Hessian D ωω F S \ ( ω ∗ S , ν ∗ S ) and D νν F S \ ( ω ∗ S , ν ∗ S ) used in Algorithm 2. The form in eq.(19) and eq.(20) can also be regarded as the total Hessian extension of the infinitesimal jackknife. In this way, it gets rid of the computationally demanding part of re-evaluating the total Hessian for samples to be deleted, which significantly reduces the computational cost. It turns out to be the same computational complexity as the state-of-the-art certified unlearning method developed for STL models [Suriyakumar and Wilson, 2022]. Moreover, Algorithm 3 can be more appealing for the successive data deletion setting.

Algorithm 3 Efficient Certified Minimax Unlearning ( ¯ A efficient )

Input: Delete requests U : { z j } j m =1 ⊆ S , output of A sc -sc ( S ) : ( ω ∗ S , ν ∗ S ) , memory variables T ( S ) : { D ωω F S ( ω ∗ S , ν ∗ S ) , D νν F S ( ω ∗ S , ν ∗ S ) } , loss function: f ( ω , ν ; z ) , noise parameters: σ 1 , σ 2 .

1: Compute

2: ˜ ω u = ˜ ω + ξ 1 , where ξ 1 ∼ N (0 , σ 1 I d 1 ) and ˜ ν u = ˜ ν + ξ 2 , where ξ 2 ∼ N (0 , σ 2 I d 2 ) . Output: ( ˜ ω u , ˜ ν u ) .

4.4 Analysis for Algorithm 3

( ϵ, δ ) -Certificated Unlearning Guarantee. The intermediate variables ( ˜ ω , ˜ ν ) are distinguishable in distribution from the retraining-from-scratch variables ( ω ∗ S \ , ν ∗ S \ ) because they are deterministic and the Taylor expansion introduces a certain amount of approximation. The following lemma quantifies the closeness between ( ˜ ω , ˜ ν ) and ( ω ∗ S \ , ν ∗ S \ ) , which can be regarded as the 'sensitivity' when applying the Gaussian mechanism.

Lemma 1 ( Closeness Upper Bound ) . Suppose the loss function f satisfies Assumption 1 and 2, ∥ D ωω F S ( ω ∗ S , ν ∗ S ) ∥ ≥ µ ωω and ∥ D νν F S ( ω ∗ S , ν ∗ S ) ∥ ≥ µ νν . Let µ = min { µ ω , µ ν , µ ωω , µ νν } . Then, we have the closeness bound between ( ˜ ω , ˜ ν ) in Line 1 of Algorithm 3 and ( ω ∗ S \ , ν ∗ S \ ) in eq.(9):

Equipped with Lemma 1, we have the following certified unlearning guarantee by adding Gaussian noise calibrated according to the above closeness result. Due to the minimax structure, our analysis is more involved than the STL case [Sekhari et al., 2021, Suriyakumar and Wilson, 2022].

Theorem 2 ( ( ϵ, δ ) -Minimax Unlearning Certification ) . Under the same settings of Lemma 1, our minimax learning algorithm A sc -sc and unlearning algorithm ¯ A efficient is ( ϵ, δ ) -certified minimax unlearning if we choose

Generalization Guarantee. Theorem 3 below provides the generalization result in terms of the population PD risk for the minimax unlearning algorithm ¯ A efficient .

Theorem 3 ( Population Primal-Dual Risk ) . Under the same settings of Lemma 1 and denote d = max { d 1 , d 2 } , the population weak and strong PD risk for the certified minimax unlearning variables ( ˜ ω u , ˜ ν u ) returned by Algorithm 3 are

Deletion Capacity. The population weak and strong PD risk given in Theorem 3 for the output of unlearning algorithms provides the following bound on deletion capacity.

Theorem 4 ( Deletion Capacity ) . Under the same settings of Lemma 1 and denote d = max { d 1 , d 2 } , the deletion capacity of Algorithm 3 is

where the constant c depends on L, l, ρ, and µ of the loss function f .

5 Certified Minimax Unlearning for Convex-Concave Loss Function

We further extend the certified minimax unlearning for the convex-concave loss function. In addition, Appendix C will provide the extension to convex-strongly-concave and strongly-convex-concave loss functions. Give the convex-concave loss function f ( ω , ν ; z ) , similar to the unlearning for STL models [Sekhari et al., 2021], we define the regularized function as ˜ f ( ω , ν ; z ) = f ( ω , ν ; z ) + λ 2 ∥ ω ∥ 2 -λ 2 ∥ ν ∥ 2 . Suppose the function f satisfies Assumption 1, then the function ˜ f is λ -strongly convex in ω , λ -strongly concave in ν , (2 L + λ ∥ ω ∥ + λ ∥ ν ∥ ) -Lipschitz, √ 2(2 ℓ + λ ) -gradient Lipschitz and ρ -Hessian Lipschitz. It suffices to apply the minimax learning and unlearning algorithms in Sec.4 to the regularized loss function with a properly chosen λ . We denote the learning and unlearning algorithms for convex-concave losses as A c -c and ¯ A c -c . Their implementation details are given in Appendix C. We suppose the SC-SC regularization parameter λ satisfies λ < ℓ . Theorem 5 below summarizes guarantees of ( ϵ, δ ) -certified unlearning and population primal-dual risk (weak and strong) for Algorithm ¯ A c -c .

Theorem 5. Let Assumption 1 hold and d = max { d 1 , d 2 } . Suppose the parameter spaces W and V are bounded so that max ω ∈W ∥ ω ∥ ≤ B ω and max ν ∈V ∥ ν ∥ ≤ B ν . We have,

( a ) ( ϵ, δ ) -Minimax Unlearning Certification: Our minimax learning algorithm A c -c and unlearning algorithm ¯ A c -c is ( ϵ, δ ) -certified minimax unlearning.

( b ) Population Weak PD Risk: The population weak PD risk for ( ω u , ν u ) by algorithm ¯ A c -c is

In particular, by setting λ below

we have the following population weak PD risk,

where c 1 , c 2 , c 3 are constants that depend only on L, l, ρ, B ω and B ν .

( c ) Population Strong PD Risk: The population strong PD risk for ( ω u , ν u ) by algorithm ¯ A c -c is

In particular, by setting λ below

we have the following population strong PD risk,

where c 1 , c 2 , c 3 , c 4 are constants that depend only on L, l, ρ, B ω and B ν .

( d ) Deletion Capacity: The deletion capacity of Algorithm ¯ A c -c is

where the constant c depends on the constants L, l, ρ, B ω and B ν .

6 Conclusion

In this paper, we have studied the certified machine unlearning for minimax models with a focus on the generalization rates and deletion capacity, while existing works in this area largely focus on standard statistical learning models. We have provided a new minimax unlearning algorithm composed of the total Hessian-based complete Newton update and the Gaussian mechanism-based perturbation, which comes with rigorous ( ϵ, δ ) -unlearning certification. We have established generalization results in terms of the population weak and strong primal-dual risk and the correspondingly defined deletion capacity results for the strongly-convex-strongly-concave loss functions, both of which match the state-of-the-art results obtained for standard statistical learning models. We have also provided extensions to other loss types like the convex-concave loss function. In addition, we have provided a more computationally efficient extension by getting rid of the total Hessian re-computation during the minimax unlearning phase, which can be more appealing for the successive data deletion setting. Although our bound for deletion capacity is better than that of DP by an order of d 1 / 4 and matches the state-of-the-art result established for unlearning under the STL setting, it remains unclear whether this bound is tight or not. In future work, we plan to extend to more general settings like the nonconvex-nonconcave loss function setting.

Acknowledgments and Disclosure of Funding

We would like to thank Gautam Kamath for his valuable comments on the presentation of the results in the previous version of the paper. Additionally, we extend our thanks to the reviewers and area chair of NeurIPS 2023 for their constructive comments and feedback. This work was supported in part by the National Natural Science Foundation of China (62072395, 62206207, U20A20178), and the National Key Research and Development Program of China (2020AAA0107705, 2021YFB3100300).

References

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , volume 70, pages 214-223. PMLR, 2017.

Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. In Advances in Neural Information Processing Systems , volume 32, pages 11279-11288, 2019.

Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private algorithms for the stochastic saddle point problem with optimal rates for the strong gap. In Conference on Learning Theory , volume 195, pages 2482-2508. PMLR, 2023.

Digvijay Boob and Cristóbal Guzmán. Optimal algorithms for differentially private stochastic monotone variational inequalities and saddle-point problems. Mathematical Programming , pages 1-43, 2023.

Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy , pages 141-159. IEEE, 2021.

Jonathan Brophy and Daniel Lowd. Machine unlearning for random forests. In International Conference on Machine Learning , volume 139, pages 1092-1104. PMLR, 2021.

Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy , pages 463-480. IEEE, 2015.

Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. Recommendation unlearning. In Proceedings of the ACM Web Conference 2022 , pages 2768-2777. ACM, 2022a.

Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , pages 499-513. ACM, 2022b.

Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7766-7775. IEEE, 2023.

Jiali Cheng, George Dasoulas, Huan He, Chirag Agarwal, and Marinka Zitnik. Gnndelete: A general strategy for unlearning in graph neural networks. In The Eleventh International Conference on Learning Representations . OpenReview.net, 2023.

Eli Chien, Chao Pan, and Olgica Milenkovic. Efficient model updates for approximate unlearning of graph-structured data. In The Eleventh International Conference on Learning Representations . OpenReview.net, 2023a.

Eli Chien, Chao Pan, and Olgica Milenkovic. Efficient model updates for approximate unlearning of graph-structured data. In The Eleventh International Conference on Learning Representations . OpenReview.net, 2023b.

Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear function approximation. In International Conference on Machine Learning , volume 80, pages 1125-1134. PMLR, 2018.

Jimmy Z. Di, Jack Douglas, Jayadev Acharya, Gautam Kamath, and Ayush Sekhari. Hidden poison: Machine unlearning enables camouflaged poisoning attacks. In Advances in Neural Information Processing Systems , volume 37, 2023.

Simon S Du, Jianshu Chen, Lihong Li, Lin Xiao, and Dengyong Zhou. Stochastic variance reduction methods for policy evaluation. In International Conference on Machine Learning , volume 70, pages 1049-1058. PMLR, 2017.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Third Theory of Cryptography Conference , volume 3876, pages 265-284. Springer, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9:211-407, 2014.

Farzan Farnia and Asuman Ozdaglar. Train simultaneously, generalize better: Stability of gradientbased minimax learners. In International Conference on Machine Learning , volume 139, pages 3174-3185. PMLR, 2021.

Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Ayush Sekhari, and Chiyuan Zhang. Ticketed learning-unlearning schemes. In The Thirty Sixth Annual Conference on Learning Theory , pages 5110-5139. PMLR, 2023.

Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. In Advances in neural information processing systems , volume 32, pages 3513-3526, 2019.

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9304-9312, 2020a.

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision-ECCV 2020: 16th European Conference , volume 12374, pages 383-398. Springer, 2020b.

Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 792-801. IEEE, 2021.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems , volume 27, pages 2672-2680, 2014.

Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11516-11524. AAAI Press, 2021.

Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning , volume 119, pages 3832-3842. PMLR, 2020.

Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics , volume 130, pages 2008-2016. PMLR, 2021.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning , volume 70, pages 1885-1894. PMLR, 2017.

Yunwen Lei, Zhenhuan Yang, Tianbao Yang, and Yiming Ying. Stability and generalization of stochastic gradient methods for minimax problems. In International Conference on Machine Learning , volume 139, pages 6175-6186. PMLR, 2021.

Yuantong Li, Chi-Hua Wang, and Guang Cheng. Online forgetting process for linear regression models. In International Conference on Artificial Intelligence and Statistics , pages 217-225. PMLR, 2021.

Shen Lin, Xiaoyu Zhang, Chenyang Chen, Xiaofeng Chen, and Willy Susilo. Erm-ktp: Knowledgelevel machine unlearning via knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20147-20155, 2023.

Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-concave minimax problems. In International Conference on Machine Learning , volume 119, pages 6083-6093. PMLR, 2020.

Junxu Liu, Mingsheng Xue, Jian Lou, Xiaoyu Zhang, Li Xiong, and Zhan Qin. Muter: Machine unlearning on adversarially trained models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4892-4902, 2023.

Luo Luo, Yujun Li, and Cheng Chen. Finding second-order stationary points in nonconvex-stronglyconcave minimax optimization. In Advances in Neural Information Processing Systems , volume 35, pages 36667-36679, 2022.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In The Sixth International Conference on Learning Representations . OpenReview.net, 2018.

Ananth Mahadevan and Michael Mathioudakis. Certifiable machine unlearning for linear models. arXiv preprint arXiv:2106.15093 , 2021.

Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review , 29(3):229-235, 2013.

Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10422-10431. IEEE, 2022.

Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory , volume 132, pages 931-962. PMLR, 2021.

Yurii Nesterov and Boris T Polyak. Cubic regularization of newton method and its global performance. Mathematical Programming , 108(1):177-205, 2006.

Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. In Advances in Neural Information Processing Systems , volume 33, pages 16025-16036, 2020.

Asuman E. Ozdaglar, Sarath Pattathil, Jiawei Zhang, and Kaiqing Zhang. What is a good metric to study generalization of minimax learners? In Advances in Neural Information Processing Systems , volume 35, pages 38190-38203, 2022.

Alexandra Peste, Dan Alistarh, and Christoph H Lampert. SSSE: efficiently erasing samples from trained machine learning models. arXiv preprint arXiv:2107.03860 , 2021.

Sebastian Schelter, Stefan Grafberger, and Ted Dunning. Hedgecut: Maintaining randomised trees for low-latency machine unlearning. In International Conference on Management of Data , pages 1545-1557. ACM, 2021.

Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. In Advances in Neural Information Processing Systems , volume 34, pages 18075-18086, 2021.

Pranay Sharma, Rohan Panda, Gauri Joshi, and Pramod Varshney. Federated minimax optimization: Improved convergence analyses and algorithms. In International Conference on Machine Learning , volume 162, pages 19683-19730. PMLR, 2022.

Takashi Shibata, Go Irie, Daiki Ikami, and Yu Mitsuzumi. Learning with selective forgetting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , pages 989-996. ijcai.org, 2021.

Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In The Sixth International Conference on Learning Representations . OpenReview.net, 2018.

Vinith Suriyakumar and Ashia C Wilson. Algorithms that approximate data removal: New results and limitations. In Advances in Neural Information Processing Systems , volume 35, pages 1889218903, 2022.

Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Kiran K Thekumparampil, Prateek Jain, Praneeth Netrapalli, and Sewoong Oh. Efficient algorithms for smooth minimax optimization. In Advances in Neural Information Processing Systems , volume 32, pages 12659-12670, 2019.

Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory , volume 134, pages 4126-4142. PMLR, 2021.

Salil Vadhan. The complexity of differential privacy. Tutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich , pages 347-450, 2017.

Cheng-Long Wang, Mengdi Huai, and Di Wang. Inductive graph unlearning. In 32nd USENIX Security Symposium , pages 3205-3222. USENIX Association, 2023a.

Weiqi Wang, Zhiyi Tian, Chenhan Zhang, An Liu, and Shui Yu. Bfu: Bayesian federated unlearning with parameter self-sharing. In Proceedings of the 2023 ACM Asia Conference on Computer and Communications Security , pages 567-578. ACM, 2023b.

Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. In 30th Annual Network and Distributed System Security Symposium . The Internet Society, 2023.

Ga Wu, Masoud Hashemi, and Christopher Srinivasa. Puma: Performance unchanged model augmentation for training data removal. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8675-8682, 2022.

Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning , volume 119, pages 10355-10366. PMLR, 2020.

Zhaomin Wu, Junhui Zhu, Qinbin Li, and Bingsheng He. Deltaboost: Gradient boosting decision trees with efficient machine unlearning. Proceedings of the ACM on Management of Data , 1(2): 1-26, 2023.

Haocheng Xia, Jinfei Liu, Jian Lou, Zhan Qin, Kui Ren, Yang Cao, and Li Xiong. Equitable data valuation meets the right to be forgotten in model markets. Proceedings of the VLDB Endowment , 16(11):3349-3362, 2023.

Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient architecture for exact machine unlearning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence , pages 4006-4013. ijcai.org, 2022.

Zhenhuan Yang, Shu Hu, Yunwen Lei, Kush R Vashney, Siwei Lyu, and Yiming Ying. Differentially private sgda for minimax problems. In Uncertainty in Artificial Intelligence , volume 180, pages 2192-2202. PMLR, 2022.

Guojun Zhang, Kaiwen Wu, Pascal Poupart, and Yaoliang Yu. Newton-type methods for minimax optimization. arXiv preprint arXiv:2006.14592 , 2020.

Junyu Zhang, Mingyi Hong, Mengdi Wang, and Shuzhong Zhang. Generalization bounds for stochastic saddle point problems. In International Conference on Artificial Intelligence and Statistics , volume 130, pages 568-576. PMLR, 2021.

Liang Zhang, Kiran K Thekumparampil, Sewoong Oh, and Niao He. Bring your own algorithm for optimal differentially private stochastic minimax optimization. In Advances in Neural Information Processing Systems , volume 35, pages 35174-35187, 2022a.

Shuijing Zhang, Jian Lou, Li Xiong, Xiaoyu Zhang, and Jing Liu. Closed-form machine unlearning for matrix factorization. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages 3278-3287, 2023.

Siqi Zhang, Yifan Hu, Liang Zhang, and Niao He. Uniform convergence and generalization for nonconvex stochastic minimax problems. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) , 2022b.

Zijie Zhang, Yang Zhou, Xin Zhao, Tianshi Che, and Lingjuan Lyu. Prompt certified machine unlearning with randomized gradient smoothing and quantization. In Advances in Neural Information Processing Systems , volume 35, pages 13433-13455, 2022c.

A Additional Definitions and Supporting Lemmas

In this section, we provide additional definitions and supporting lemmas. In the next two sections, Sec.B contains missing proofs in Sec.4 and the online extension to support successive unlearning setting. Sec.C contains missing proofs in Sec.5, as well as detailed algorithm descriptions for the general convex-concave loss function setting.

A.1 Additional Definitions

We first recall the following standard definitions for the loss function f ( ω , ν ; z ) from optimization literature.

Definition 4 ( Function Lipschitz Continuity ) . The function f ( ω , ν ; z ) is L -Lipschitz, i.e., there exists a constant L > 0 such that for all ω , ω ′ ∈ W , ν , ν ′ ∈ V and z ∈ Z , it holds that

Definition 5 ( Gradient Lipschitz Continuity ) . The function f ( ω , ν ; z ) has ℓ -Lipschitz gradients, i.e., there exists a constant ℓ > 0 such that for all ω , ω ′ ∈ W , ν , ν ′ ∈ V and z ∈ Z , it holds that

Definition 6 ( Hessian Lipschitz Continuity ) . The function f ( ω , ν ; z ) has ρ -Lipschitz Hessian, i.e., there exists a constant ρ > 0 such that for all ω , ω ′ ∈ W , ν , ν ′ ∈ V and z ∈ Z , it holds that

Definition 7 ( Strongly-Convex-Strongly-Concave Objective Function ) . The function f ( ω , ν ; z ) is µ ω -strongly convex on W and µ ν -strongly concave on V , i.e., there exist constants µ ω > 0 and µ ν > 0 such that for all ω , ω ′ ∈ W , ν , ν ′ ∈ V and z ∈ Z , it holds that

Definition 8 ( Best Response Auxiliary Functions ) . We introduce auxiliary functions V S ( ω ) and V S \ ( ω ) , given by

and we have ν ∗ S = V S ( ω ∗ S ) and ν ∗ S \ = V S \ ( ω ∗ S \ ) . We can similarly introduce W S ( ν ) and W S \ ( ν ) as

and we have ω ∗ S = W S ( ν ∗ S ) and ω ∗ S \ = W S \ ( ν ∗ S \ ) by this definition.

In addition, we define the primal function P ( ω ) := max ν ∈V F S ( ω , ν ) = F S ( ω , V S ( ω )) , which has gradient ∇ P ( ω ) = ∇ ω F S ( ω , V S ( ω )) and Hessian ∇ 2 ωω P ( ω ) = D ωω F S ( ω , V S ( ω )) (i.e., the total Hessian of F S ). The dual function, its gradient, and Hessian can be similarly defined, e.g., D ( ν ) := min ω ∈W F S ( ω , ν ) = F S ( W S ( ν ) , ν ) .

A.2 Supporting Lemmas

The following lemma provides the distance between V S ( ω ∗ S ) and V S \ ( ω ∗ S ) . Similar result can be derived for the distance between W S ( ν ∗ S ) and W S \ ( ν ∗ S ) .

Lemma 2. Under Assumption 1 and Assumption 2, the variables V S \ ( ω ∗ S ) and ν ∗ S (i.e., V S ( ω ∗ S ) ) defined in Algorithm 1 satisfy the following distance bound

Proof. We observe that

where the inequality ( i ) follows from that ν ∗ S is the maximizer of the function F S ( ω , ν ) , thus F S ( ω ∗ S , V S \ ( ω ∗ S )) -F S ( ω ∗ S , ν ∗ S ) ≤ 0 . The inequality ( ii ) is due to the fact that the function f is L -Lipschitz. Also note that the function F S \ ( ω , ν ) is µ ν -strongly concave, thus we have

Eq.(39) and eq.(40) together give that

which implies that ∥ ν ∗ S -V S \ ( ω ∗ S ) ∥ ≤ 2 Lm µ ν ( n -m ) .

The following lemma provides the distance between ( ω ∗ S \ , ν ∗ S \ ) and ( ω ∗ S , ν ∗ S ) .

Lemma 3. Under Assumption 1 and Assumption 2, the variables ( ω ∗ S \ , ν ∗ S \ ) defined in eq.(9) and ( ω ∗ S , ν ∗ S ) defined in Algorithm 1 satisfy the following guarantees

Proof. We begin with the ω -part,

where the inequality ( i ) holds because ω ∗ S \ is the minimizer of the function F S \ ( ω , ν ) , thus F S \ ( ω ∗ S \ , ν ∗ S \ ) -F S \ ( ω ∗ S , ν ∗ S \ ) ≤ 0 , and the inequality ( ii ) follows from the fact that the function f is L -Lipschitz. Since the function F S ( ω , ν ) is µ ω -strongly convex, we further get

Eq.(43) and eq.(44) together gives that

For the ν -part, we similarly have

where the inequality ( i ) follows from that ν ∗ S \ is the maximizer of the function F S \ ( ω , ν ) , thus F S \ ( ω ∗ S \ , ν ∗ S ) -F S \ ( ω ∗ S \ , ν ∗ S \ ) ≤ 0 . The inequality ( ii ) is due to the fact that the function f is L -Lipschitz. In addition, by the strongly-concave assumption of F S ( ω , ν ) is µ ν , we have

By eq.(46) and eq.(47), we get that

Thus, we have ∥ ν ∗ S \ -ν ∗ S ∥ ≤ 2 Lm µ ν n

In the following, we recall several lemmas (i.e., Lemma 4 to Lemma 8) from existing minimax optimization literature for completeness.

Lemma 4 ([Lin et al., 2020, Lemma 4.3]) . Under Assumption 1 and Assumption 2, for any ω ∈ W , the function V S ( ω ) is ( ℓ/µ ν ) -Lipschitz.

Proof. By the optimality condition of the function V S ( ω ) , we have

Summing the two inequalities above yields

Since the function F S ( ω , ν ) is µ ν -strongly concave in ν , we have

By eq.(49) and eq.(50) with the ℓ -Lipschitz continuity of ∇ F S ( ω , ν ) , we further get

Consequently, we have

Remark 1. The above lemma can be similarly derived for W S to obtain that the best response auxiliary function W S ( ν ) is ( ℓ/µ ω ) -Lipschitz. In the next three lemmas, we focus on the ω -part and omit the ν -part.

Lemma 5 ([Luo et al., 2022, Lemma 3]) . Denote κ ν = ℓ/µ ν . Under Assumption 1 and Assumption 2, for any ω , ω ′ ∈ W , we have

Lemma 6 ([Nesterov and Polyak, 2006, Lemma 1]) . Denote κ ν = ℓ/µ ν . Under Assumption 1 and Assumption 2, for any ω , ω ′ ∈ W , we have

Proof. Recall the definition of the primal function P ( ω ) := max ν ∈V F S ( ω , ν ) and its gradient ∇ P ( ω ) = ∇ ω F S ( ω , V S ( ω )) . Due to the optimality of V S , we have

By taking the total derivative with respect to ω , we get

Taking the total derivative of ω again on ∇ P ( ω ) , we further have

Based on the equality of ∇ 2 P ( ω ) and D ωω F S ( ω , V S ( ω )) above and Lemma 5, we have

Lemma 7. Under Assumption 1 and Assumption 2, for all ω ∈ W and ν ∈ V , we have ∥ D ωω f ( ω , ν ; z ) ∥ ≤ ℓ + ℓ 2 µ ν .

Proof. By the definition of the total Hessian, we have

where the inequality ( i ) uses the triangle inequality and the inequality ( ii ) is due to the function f has ℓ -Lipschitz gradients and f is µ ν -strongly concave in ν , thus we have ∥ ∂ ωω f ( ω , ν ; z ) ∥ ≤ ℓ , ∥ ∂ ων f ( ω , ν ; z ) ∥ ≤ ℓ , ∥ ∂ νω f ( ω , ν ; z ) ∥ ≤ ℓ and ∥ ∂ νν f ( ω , ν ; z ) ∥ ≤ µ -1 ν .

Lemma 8 ([Zhang et al., 2022a, Lemma 4.4]) . Under Assumption 1 and Assumption 2, the population weak PD risk for the minimax learning variables ( ω ∗ S , ν ∗ S ) returned by Algorithm 1 has

Lemma 9 ([Zhang et al., 2021, Theorem 2]) . Under Assumption 1 and Assumption 2, the population strong PD risk for the minimax learning variables ( ω ∗ S , ν ∗ S ) returned by Algorithm 1 has

B Detailed Algorithm Analysis and Missing Proofs in Section 4

B.1 Analysis for Algorithm 2

In the following, we provide the analysis for Algorithm 2 in terms of guarantees of ( ϵ, δ ) -certified unlearning, population primal-dual risk, and deletion capacity and the corresponding proofs.

Lemma 10 ( Closeness Upper Bound ) . Suppose the loss function f satisfies Assumption 1 and 2, ∥ D ωω F S ( ω ∗ S , ν ∗ S ) ∥ ≥ µ ωω and ∥ D νν F S ( ω ∗ S , ν ∗ S ) ∥ ≥ µ νν . Let µ = min { µ ω , µ ν , µ ωω , µ νν } . Then, we have the closeness bound between ( ̂ ω , ̂ ν ) in Line 2 of Algorithm 2 and ( ω ∗ S \ , ν ∗ S \ ) in eq.(9):

Proof. Recall that the empirical loss functions F S \ ( ω , ν ) and F S ( ω , ν ) are

Wefocus on the key term ∇ ω F S \ ( ω ∗ S \ , V S ( ω ∗ S \ )) -∇ ω F S \ ( ω ∗ S , ν ∗ S ) -D ωω F S \ ( ω ∗ S , ν ∗ S )( ω ∗ S \ -ω ∗ S ) , which has the following conversions

We denote κ ν = ℓ/µ ν . For the first term on the right-hand side of the inequality in eq.(64), we have

where the inequality ( i ) is by Lemma 6 and the inequality ( ii ) is by Lemma 3.

For the second term on the right-hand side of the inequality in eq.(64), we have

where the inequality ( i ) follows by the fact that the function ∇ ω f ( · , · ) is ℓ -Lipschitz continuous and ν ∗ S = V S ( ω ∗ S ) . The inequality ( ii ) holds because Lemma 4, and the inequality ( iii ) is by Lemma 3.

For the third term on the right-hand side of the inequality in eq.(64), we have

where the first inequality is by Lemma 7. Plugging eq.(65), eq.(66) and eq.(67) into eq.(64), we further get

The above derivation yields an upper bound result. In the following, we derive a lower bound result. Let x be the vector satisfying the following relation,

Since we have ∇ ω F S \ ( ω ∗ S , ν ∗ S ) = -1 n -m ∑ z i ∈ U ∇ ω f ( ω ∗ S , ν ∗ S ; z i ) and ∇ ω F S \ ( ω ∗ S \ , V S ( ω ∗ S \ )) = 0 due to the optimality of ω ∗ S \ , plugging eq.(69) into eq.(68), we get that

For the left-hand side of eq.(70), with ∥ D ωω F S ( ω ∗ S , ν ∗ S ) ∥ ≥ µ ωω , we have

where the second inequality is by Lemma 7. Combining eq.(70), eq.(68), and the definition of the vector ∥ x ∥ , we get that

Theorem 6 ( ( ϵ, δ ) -Minimax Unlearning Certification ) . Under the same settings of Lemma 10, our minimax learning algorithm A sc -sc and unlearning algorithm ¯ A sc -sc is ( ϵ, δ ) -certified minimax unlearning if we choose

Proof. Our proof for ( ϵ, δ ) -minimax unlearning certification is similar to the one used for the differential privacy guarantee of the Gaussian mechanism [Dwork et al., 2014].

Let ( ω ∗ S , ν ∗ S ) be the output of the learning algorithm A sc -sc trained on dataset S and ( ω u , ν u ) be the output of the unlearning algorithm ¯ A sc -sc running with delete requests U , the learned

model ( ω ∗ S , ν ∗ S ) , and the memory variables T ( S ) . Then we have ( ω ∗ S , ν ∗ S ) = A sc -sc ( S ) and ( ω u , ν u ) = ¯ A sc -sc ( U, A sc -sc ( S ) , T ( S )) . We also denote the intermediate variables before adding noise in algorithm ¯ A sc -sc as ( ̂ ω , ̂ ν ) , and we have ω u = ̂ ω + ξ 1 and ν u = ̂ ν + ξ 2 .

Smilarly, let ( ω ∗ S \ , ν ∗ S \ ) be the output of the learning algorithm A sc -sc trained on dataset S \ U and ( ω u S \ , ν u S \ ) be the output of the unlearning algorithm ¯ A sc -sc running with delete requests ∅ , the learned model ( ω ∗ S \ , ν ∗ S \ ) , and the memory variables T ( S \ U ) . Then we have ( ω ∗ S \ , ν ∗ S \ ) = A sc -sc ( S \ U ) and ( ω u S \ , ν u S \ ) = ¯ A sc -sc ( ∅ , A sc -sc ( S \ U ) , T ( S )) . We also denote the intermediate variables before adding noise in algorithm ¯ A sc -sc as ( ̂ ω S \ , ̂ ν S \ ) , and we have ω u S \ = ̂ ω S \ + ξ 1 and ν u S \ = ̂ ν S \ + ξ 2 . Note that ̂ ω S \ = ω ∗ S \ and ̂ ν S \ = ν ∗ S \ .

We sample the noise ξ 1 ∼ N (0 , σ 1 I d 1 ) and ξ 2 ∼ N (0 , σ 2 I d 2 ) with the scale:

where ∥ ω ∗ S \ -̂ ω ∥ and ∥ ω ∗ S \ -̂ ω ∥ are given in Lemma 10. Then, following the same proof as Dwork et al. [2014, Theorem A.1] together with the composition property of DP [Vadhan, 2017, Lemma 7.2.3], we get that, for any set O ⊆ Θ where Θ := W×V ,

which implies that the algorithm pair A sc -sc and ¯ A sc -sc is ( ϵ, δ ) -certified minimax unlearning.

Theorem 7 ( Population Primal-Dual Risk ) . Under the same settings of Lemma 10 and denote d = max { d 1 , d 2 } , the population weak and strong PD risk for the certified minimax unlearning variables ( ω u , ν u ) returned by Algorithm 2 are

Proof. We begin with the population weak PD risk for the certified minimax unlearning variable ( ω u , ν u ) , which has the following conversions,

(77) where the inequality ( i ) holds because the population loss function F ( ω , ν ) := E [ f ( ω , ν ; z )] is L -Lipschitz continuous. The inequality ( ii ) is by Lemma 8.

By recalling the unlearning update step in Algorithm 2, we have

where the vector ξ 1 ∈ R d 1 is drawn independently from N (0 , σ 2 1 I d 1 ) . From the relation in eq.(78), we further get

where the inequality ( i ) is by the triangle inequality and the inequality ( ii ) follows from the relation in eq.(71), together with the Jensen's inequality to bound E [ ∥ ξ 1 ∥ ] . The equality ( iii ) holds because the vector ξ 1 ∼ N (0 , σ 2 1 I d 1 ) and thus we have E [ ∥ ξ 1 ∥ 2 ] = d 1 σ 2 1 . Furthermore, the inequality ( iv ) is due to the fact that f ( ω , ν ; z ) is L -Lipshitz continuous.

Symmetrically, we have

Plugging eq.(79) and eq.(80) into eq.(77) with d = max { d 1 , d 2 } we get

√

With the noise scale σ 1 and σ 2 being equal to 2(8 2 L 2 ℓ 3 ρ/µ 5 +8 Lℓ 2 /µ 2 ) m 2 n ( µn -( ℓ + ℓ 2 /µ ) m ) ϵ √ 2 log(2 . 5 /δ ) , we can get our generalization guarantee with population weak PD risk:

For the population strong PD risk △ s ( ω u , ν u ) , similarly, we have

where inequality ( i ) is due to the fact that the population loss function F ( ω , ν ) := E [ f ( ω , ν ; z )] is L -Lipschitz continuous. The inequality ( ii ) uses eq.(79), eq.(80) and Lemma 9. With the same noise scale above, we can get the generalization guarantee in terms of strong PD risk below,

Theorem 8 ( Deletion Capacity ) . Under the same settings of Lemma 10 and denote d = max { d 1 , d 2 } , the deletion capacity of Algorithm 2 is √

where the constant c depends on L, l, ρ, and µ of the loss function f .

Proof. By the definition of deletion capacity, in order to ensure the population PD risk derived in Theorem 7 is bounded by γ , it suffices to let:

where the constant c depends on the properties of the loss function f ( ω , ν ; z ) .

B.2 Missing Proofs of Sec.4.4

B.2.1 Proof of Lemma 1 (Closeness Upper Bound)

Proof.

where the inequality ( i ) holds because the triangle inequality and the inequality ( ii ) uses the results in eq.(65) and eq.(66). Now let ˜ x be the vector satisfying the following relation,

Since we have ∇ ω F S \ ( ω ∗ S , ν ∗ S ) = -1 n -m ∑ z i ∈ U ∇ ω f ( ω ∗ S , ν ∗ S ; z i ) and ∇ ω F S \ ( ω ∗ S \ , V S ( ω ∗ S \ )) = 0 due to the optimality of ω ∗ S \ , plugging the above relation into eq.(86), we get

With D ωω F S ( ω ∗ S , ν ∗ S ) ≥ µ ωω , we also have

Combining eq.(89), eq.(88), and the definition of the vector ∥ ˜ x ∥ , we get that

B.2.2 Proof of Theorem 2 ( ( ϵ, δ ) -Minimax Unlearning Certification)

Proof. With the closeness upper bound in Lemma 1 and the given noise scales in eq.(22), the proof is identical to that of Theorem 6.

B.2.3 Proof of Theorem 3 (Population Primal-Dual Risk)

Proof. We start with the population weak PD risk. By eq.(77), we have

By recalling the unlearning step in Algorithm 3, we have

where the vector ξ 1 ∈ R d 1 is drawn independently from N (0 , σ 2 1 I d 1 ) . From the relation in eq.(78), we further get

where the inequality ( i ) uses the triangle inequality and the inequality ( ii ) follows by the relation D ωω F S ( ω ∗ S , ν ∗ S ) ≥ µ ωω ≥ µ , together with the Jensen's inequality to bound E [ ∥ ξ 1 ∥ ] . The equality ( iii ) holds because the vector ξ 1 ∼ N (0 , σ 2 1 I d 1 ) and thus we have E [ ∥ ξ 1 ∥ 2 ] = d 1 σ 2 1 . And the inequality ( iv ) is due to the fact that f ( ω , ν ; z ) is L -Lipshitz continuous. Symmetrically, we have

Plugging eq.(93) and eq.(94) into eq.(91) with d = max { d 1 , d 2 } we get

With the noise scale σ 1 and σ 2 being equal to 2(8 √ 2 L 2 ℓ 3 ρ/µ 6 +2 √ 2 Lℓ 2 /µ 3 ) m 2 n 2 ϵ √ 2 log(2 . 5 /δ ) , we can get our generalization guarantee in terms of population weak PD risk:

For the population strong PD risk, using an application of eq.(83) with Lemma 9, eq.(93), eq.(94) and the noise scales given in Theorem 2, we can get

B.2.4 Proof of Theorem 4 (Deletion Capacity)

Proof. By the definition of deletion capacity, in order to ensure the population weak or strong PD risk derived in Lemma 3 is bounded by γ , it suffices to let m A, ¯ A ϵ,δ,γ ( d 1 , d 2 , n ) ≥ c · n √ ϵ ( d log(1 /δ )) 1 / 4 .

B.3 Efficient Online Unlearning Algorithm

To support successive unlearning requests, similar to the STL case Suriyakumar and Wilson [2022], we further provide an efficient and online minimax unlearning algorithm (denoted by ¯ A online ). The pseudocode of ¯ A online is given in Algorithm 4. Its certified minimax unlearning guarantee, generalization, and deletion capacity can be identically yielded as Algorithm 3, which are omitted here.

Algorithm 4 Efficient Online Certified Minimax Unlearning ( ¯ A online )

Input: Delete request z k ∈ S , early delete requests U : { z j } j m =1 ⊆ S , output of A sc -sc ( S ) : ( ω ∗ S , ν ∗ S ) , memory variables T ( S ) : { D ωω F S ( ω ∗ S , ν ∗ S ) , D νν F S ( ω ∗ S , ν ∗ S ) } , early unlearning variables: ( ˜ ω u U , ˜ ν u U ) , loss function: f ( ω , ν ; z ) , noise parameters: σ 1 , σ 2 .

2: Compute

1: Set ˜ ω ∅ = ω ∗ S and ˜ ν ∅ = ν ∗ S .

.

Output: ( ˜ ω u U ∪{ k } , ˜ ν u U ∪{ k } ) .

C Detailed Algorithm Descriptions and Missing Proofs in Section 5

C.1 Minimax Unlearning Algorithm for Smooth Convex-Concave Loss Function

In this section, we provide minimax learning and minimax unlearning algorithms for smooth convexconcave loss functions based on the counterpart algorithms for the SC-SC setting. Given the convex-concave loss function f ( ω , ν ; z ) , we define the regularized loss function as ˜ f ( ω , ν ; z ) = f ( ω , ν ; z ) + λ 2 ∥ ω ∥ 2 -λ 2 ∥ ν ∥ 2 . Suppose the function f satisfies Assumption 1, then the function ˜ f is λ -strongly convex in ω , λ -strongly concave in ν , (2 L + λ ∥ ω ∥ + λ ∥ ν ∥ ) -Lipschitz, √ 2(2 ℓ + λ ) -gradient Lipschitz and ρ -Hessian Lipschitz. Thus, we can apply the minimax learning in Algorithm 1 and unlearning in Algorithm 2 to the regularized loss function with a properly chosen λ . We denote our learning algorithm by A c -c and unlearning algorithm by ¯ A c -c . The pseudocode is provided in Algorithm 5 and Algorithm 6, respectively. Additionally, we denote the regularized population loss as ˜ F ( ω , ν ) := E z ∼D [ ˜ f ( ω , ν ; z )] and regularized empirical loss as ˜ F S ( ω , ν ) := 1 n ∑ n i =1 ˜ f ( ω , ν ; z i ) .

Algorithm 5 Mimimax Learning Algorithm ( A c -c )

Input: Dataset S : { z i } n i =1 ∼ D n , loss function: f ( ω , ν ; z ) , regularization parameter: λ .

1: Define

2: Run the algorithm A sc -sc on the dataset S with loss function ˜ f . Output: ( ω ∗ S , ν ∗ S , D ωω ˜ F S ( ω ∗ S , ν ∗ S ) , D νν ˜ F S ( ω ∗ S , ν ∗ S )) ← A sc -sc ( S, ˜ f ) .

Algorithm 6 Certified Minimax Unlearning for Convex-Concave Loss ( ¯ A c -c )

Input: Delete requests U : { z j } j m =1 ⊆ S , output of A c -c ( S ) : ( ω ∗ S , ν ∗ S ) , memory variables T ( S ) : { D ωω ˜ F S ( ω ∗ S , ν ∗ S ) , D νν ˜ F S ( ω ∗ S , ν ∗ S ) } , loss function: f ( ω , ν ; z ) , regularization parameter: λ , noise parameters: σ 1 , σ 2 .

1: Define

2: Run the algorithm ¯ A sc -sc with delete requests U , learning variables ( ω ∗ S , ν ∗ S ) , memory variables T ( S ) , loss function ˜ f and noise parameters σ 1 and σ 2 .

Output: ( ω u , ν u ) ← ¯ A sc - sc ( U, ( ω ∗ S , ν ∗ S ) , T ( S ) , ˜ f, σ 1 , σ 2 ) .

C.2 Supporting Lemma

Lemma11. Suppose the function f ( ω , ν ; z ) is L -Lipschitz continuous. Define the function ˜ f ( ω , ν ; z ) as

Given a dataset S = { z i } n i =1 and denote ( ω ∗ S , ν ∗ S ) := arg min ω ∈W max ν ∈V { ˜ F S ( ω , ν ) := 1 n ∑ n i =1 ˜ f ( ω , ν ; z i ) } . Then, the variables ( ω ∗ S , ν ∗ S ) satisfy ∥ ω ∗ S ∥ ≤ L/λ and ∥ ν ∗ S ∥ ≤ L/λ .

Proof. Due to the optimality of ω ∗ S , we have

Plugging in the definition of the function ˜ f in the above, we get that

Then, using the triangle inequality, we have

where the last inequality holds because the function f is L -Lipschitz continuous. Thus we have ∥ ω ∗ S ∥ ≤ L/λ . Similarly, we can get ∥ ν ∗ S ∥ ≤ L/λ .

Lemma 11 implies that the empirical optimizer ( ω ∗ S , ν ∗ S ) returned by Algorithm 6 satisfies ∥ ω ∗ S ∥ ≤ L/λ and ∥ ν ∗ S ∥ ≤ L/λ . Thus our domain of interest are W := { ω |∥ ω ∥ ≤ L/λ } and V := { ν |∥ ν ∥ ≤ L/λ } . Over the set W×V , the function ˜ f ( ω , ν ; z ) is 4 L -Lipschitz continuous. Also, with λ < ℓ , ˜ f ( ω , ν ; z ) has 3 √ 2 ℓ -Lipschitz gradients.

C.3 Proof of Theorem 5 (Certified Minimax Unlearning for Convex-Concave Loss Funcion)

Denote ˜ L = 4 L and ˜ ℓ = 3 √ 2 ℓ , then the function ˜ f is ˜ L -Lipschitz continuous and has ˜ ℓ -Lipschitz gradients. Let ( ω ∗ S \ , ν ∗ S \ ) be the optimal solution of the loss function ˜ F S \ ( ω , ν ) on the remaining dataset, i.e.,

Additionally, we have ∥ D ωω ˜ F S ( ω ∗ S , ν ∗ S ) ∥ ≥ λ and ∥ D νν ˜ F S ( ω ∗ S , ν ∗ S ) ∥ ≥ λ .

Lemma 12. Under the settings of Theorem 5, for any λ > 0 , the population weak and strong PD risk for the minimax learning variables ( ω ∗ S , ν ∗ S ) returned by Algorithm 5 are

Proof. For the function ˜ F ( ω , ν ) , an application of Lemma 8 gives that

By the assumption of bounded parameter spaces W and V so that max ω ∈W ∥ ω ∥ ≤ B ω and max ν ∈V ∥ ν ∥ ≤ B ν , we have the following derivations for the population weak PD risk,

Similarly, an application of Lemma 9 gives that

And we can get the population strong PD risk with the following conversions,

Lemma 13 ( Closeness Upper Bound ) . Under the settings of Theorem 5, we have the closeness bound between the intermediate variables ( ̂ ω , ̂ ν ) in Algorithm 6 and ( ω ∗ S \ , ν ∗ S \ ) in eq.(106):

Proof. Since we now run the algorithms A sc -sc and ¯ A sc -sc with the regularized loss function ˜ f , the proof is identical to that of Lemma 10.

Equipped with the supporting lemmas above, the proof of Theorem 5 can be separated into the proofs of the following three lemmas.

Lemma 14 ( Minimax Unlearning Certification ) . Under the settings of Theorem 5, our minimax learning algorithm A c -c and unlearning algorithm ¯ A c -c is ( ϵ, δ ) -certified minimax unlearning if we choose √

Proof. With the closeness upper bound in Lemma 13 and the given noise scales in eq.(113), the proof is identical to that of Theorem 6.

Lemma 15 ( Population Primal-Dual Risk ) . Under the settings of Theorem 5, the population weak and strong PD risk for ( ω u , ν u ) returned by Algorithm 6 are

Proof. For the population weak PD risk, an application of eq.(77) together with Lemma 12 gives that

According to Algorithm 6, we have the unlearning update step

where ˜ F S \ ( ω ∗ S , ν ∗ S ) := 1 n -m ∑ z i ∈ S \ U ˜ f ( ω , ν ; z i ) . From the relation above, we further get

where the inequality ( i ) uses the triangle inequality and the inequality ( ii ) follows from an application of eq.(71), together with the Jensen's inequality to bound E [ ∥ ξ 1 ∥ ] . The equality ( iii ) holds because the vector ξ 1 ∼ N (0 , σ 2 1 I d 1 ) and thus we have E [ ∥ ξ 1 ∥ 2 ] = d 1 σ 2 1 . The inequality ( iv ) uses the definition of the function ˜ f and the triangle inequality. The inequality ( vi ) is due to the fact that f ( ω , ν ; z ) is L -Lipshitz continuous and Lemma 11. Symmetrically, we have

Plugging eq.(117) and eq.(118) into eq.(115) with noise scales given in Lemma 14, we can get our generalization guarantee in terms of population weak PD risk:

Similarly, using an application of eq.(83) together with Lemma 12, Lemma 14, eq.(117) and eq.(118), we can get the following population strong PD risk:

Lemma 16 ( Deletion Capacity ) . Under the settings of Theorem 5, the deletion capacity of Algorithm 6 is √

where the constant c depends on L, l, ρ, B ω and B ν .

Proof. By the definition of deletion capacity, in order to ensure the population PD risk derived in Lemma 15 is bounded by γ , it suffices to let m A, ¯ A ϵ,δ,γ ( d 1 , d 2 , n ) ≥ c · n √ ϵ ( d log(1 /δ )) 1 / 4 .

C.4 Minimax Unlearning Algorithm for Smooth Convex-Strongly-Concave Loss Function

In this section, we briefly discuss the extension to the smooth C-SC setting. The SC-C setting is symmetric and thus omitted here.

Given the loss function f ( ω , ν ; z ) that satisfies Assumption 1 with µ ν -strong concavity in ν , we define the regularized function as ˜ f ( ω , ν ; z ) = f ( ω , ν ; z ) + λ 2 ∥ ω ∥ 2 . Our minimax learning and minimax unlearning algorithms for C-SC loss function f denoted by A c -sc and ¯ A c -sc are given in Algorithm 7 and Algorithm 8 respectively. Additionally, we denote the regularized population loss by ˜ F ( ω , ν ) := E z ∼D [ ˜ f ( ω , ν ; z )] and regularized empirical loss by ˜ F S ( ω , ν ) := 1 n ∑ n i =1 ˜ f ( ω , ν ; z i ) .

Algorithm 7 Mimimax Learning Algorithm ( A c -sc )

Input: Dataset S : { z i } n i =1 ∼ D n , loss function: f ( ω , ν ; z ) , regularization parameter: λ .

1: Define

2: Run the algorithm A sc -sc on the dataset S with loss function ˜ f . Output: ( ω ∗ S , ν ∗ S , D ωω ˜ F S ( ω ∗ S , ν ∗ S ) , D νν ˜ F S ( ω ∗ S , ν ∗ S )) ← A sc -sc ( S, ˜ f ) .

Algorithm 8 Certified Minimax Unlearning for Convex-Strongly-Concave Loss ( ¯ A c -sc )

Input: Delete requests U : { z j } j m =1 ⊆ S , output of A c -sc ( S ) : ( ω ∗ S , ν ∗ S ) , memory variables T ( S ) : { D ωω ˜ F S ( ω ∗ S , ν ∗ S ) , D νν ˜ F S ( ω ∗ S , ν ∗ S ) } , loss function: f ( ω , ν ; z ) , regularization parameter: λ , noise parameters: σ 1 , σ 2 .

1: Define

2: Run the algorithm ¯ A sc -sc with delete requests U , learning variables ( ω ∗ S , ν ∗ S ) , memory variables T ( S ) , loss function ˜ f and noise parameters σ 1 and σ 2 .

Output: ( ω u , ν u ) ← ¯ A sc - sc ( U, ( ω ∗ S , ν ∗ S ) , T ( S ) , ˜ f, σ 1 , σ 2 ) .

Note that the function ˜ f ( ω , ν ; z ) is λ -strongly convex in ω , µ ν -strongly concave in ν , ( ˜ L := 2 L + λ ∥ ω ∥ ) -Lipschitz, ( ˜ ℓ := √ 2(2 ℓ + λ )) -gradient Lipschitz and ρ -Hessian Lipschitz. We also have ∥ D ωω ˜ F S ( ω ∗ S , ν ∗ S ) ∥ ≥ λ . Let ( ω ∗ S \ , ν ∗ S \ ) be the optimal solution of the loss function ˜ F S \ ( ω , ν ) on the remaining dataset, i.e.,

An application of Lemma 11 implies that the empirical optimizer ( ω ∗ S , ν ∗ S ) returned by Algorithm 8 satisfies ∥ ω ∗ S ∥ ≤ L/λ . Thus our domain of interest are W := { ω |∥ ω ∥ ≤ L/λ } . Over the set W , the function ˜ f ( ω , ν ; z ) is 3 L -Lipschitz continuous. Suppose the strongly-convex regularization parameter λ satisfies λ < ℓ , then ˜ f ( ω , ν ; z ) has 3 √ 2 ℓ -Lipschitz gradients.

The corresponding theoretical results are given below.

Lemma 17 ( Closeness Upper Bound ) . Let Assumption 1 hold. Assume the function f ( ω , ν ; z ) is µ ν -strongly concave in ν and ∥ D νν ˜ F S ( ω ∗ S , ν ∗ S ) ∥ ≥ µ νν . Let µ = min { µ ν , µ νν } . Then, we have the closeness bound between the intermediate variables ( ̂ ω , ̂ ν ) in Algorithm 8 and ( ω ∗ S \ , ν ∗ S \ ) in eq.(124):

Proof. Since we now run the algorithms A sc -sc and ¯ A sc -sc with the regularized loss function ˜ f , the proof is identical to that of Lemma 10.

Lemma 18 ( Minimax Unlearning Certification ) . Under the settings of Lemma 17, our minimax learning algorithm A c -sc and unlearning algorithm ¯ A c -sc is ( ϵ, δ ) -certified minimax unlearning if

we choose

Proof. With the closeness upper bound in Lemma 17 and the given noise scales in eq.(126), the proof is identical to that of Theorem 6.

Lemma 19 ( Population Weak PD Risk ) . Under the same settings of Lemma 17, suppose the parameter space W is bounded so that max ω ∈W ∥ ω ∥ ≤ B ω , the population weak PD risk for the certified minimax unlearning variables ( ω u , ν u ) returned by Algorithm 8 is

where d = max { d 1 , d 2 } . In particular, by setting the regularization parameter λ as:

we have the following population weak PD risk:

where c 1 , c 2 , c 3 , c 4 and c 5 are constants that depend only on L, l, ρ, µ and B ω .

Proof. An application of [Zhang et al., 2021, Theorem 1] gives that

Using the relation above with an application of eq.(77) and eq.(109), we have

By an application of eq.(117), we further get

Plugging eq.(132) and eq.(133) into eq.(131) with noise scales given in Lemma 18, we can get our generalization guarantee:

and

Lemma 20 ( Population Strong PD Risk ) . Under the same settings of Lemma 19, the population strong PD risk for ( ω u , ν u ) returned by Algorithm 8 is

where d = max { d 1 , d 2 } . In particular, by setting the regularization parameter λ as:

we have the following population strong PD risk:

where c 1 , c 2 , c 3 , c 4 , c 5 , c 6 and c 7 are constants that depend only on L, l, ρ, µ and B ω .

Proof. An application of Lemma 9 gives that

Using an application of eq.(83) and eq.(111), together with eq.(138), eq.(132), eq.(133) and the noise scales given in Lemma 18, we have

Lemma 21 ( Deletion Capacity ) . Under the same settings as Lemma 19, the deletion capacity of Algorithm 3 is √

where the constant c depends on L, l, ρ, µ and B ω and d = max { d 1 , d 2 } .

Proof. By the definition of deletion capacity, in order to ensure the population PD risk derived in Lemma 19 or Lemma 20 is bounded by γ , it suffices to let m A, ¯ A ϵ,δ,γ ( d 1 , d 2 , n ) ≥ c · n √ ϵ ( d log(1 /δ )) 1 / 4 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes using hessian-based updates to solve the unlearning problem for strongly-convex-strongly-concave minimax learners. They also provide theoretical analyses of the error of such an unlearning algorithm as well as its sensitivity to deleted data.\n\nSTRENGTHS:\n1. This paper is well-written and structured. The narrative is very easy to follow, and the analysis is strong. \n2. The unlearning update is simple.\n3. While in a restricted setting, the analysis is coherent.\n4. Motivation is obviously, well done, given the connections to privacy. \n\nThis paper operates in a restricted setting with some small issues with the presentation. The issues with presentation are fixable, and I expect the authors will go through and better annotate their notation for the camera-ready version. Moreover, the setting is restricted and not practical for unlearning in modern settings. However, this paper achieves its goal and is a good step in the right direction. I, therefore, advocate for this paper's acceptance.\n\nWEAKNESSES:\n1. There is a notational mistake for Equation (5). You are missing a closing parenthesis.\n2. In equation 6, you should specify that $m$ is the size of $U$.\n3. You miswrote Assumption 2 in Lemma 2.\n4. I feel some work is needed on the motivation of the setting of this paper. I will discuss more of this in limitations. However, this paper seems only to work when the loss is strongly-convex-strongly-concave loss functions (or suffers bad constants for transforming in the c,c case). However, unlearning is meant to be a more practical tool for modern learning, where this condition rarely holds.\n\nQUESTIONS:\n1. What is $\\mathcal{T}$? I understand that it is the set of memory variables, but that should be more formally defined, and I don't find that it is formally defined elsewhere in the paper. \n2. It seems that most of the variables in the statement of Lemma 2 are difficult to understand. It is only after going through the proof that I am able to understand. This should be made more clear. For example, what is $\\rho$ or $l$ here? This suggestion follows for correcting Theorem 5.\n3. I would be interested if the authors of this paper have thought about weakening the strongly-convex-strongly-concave conditions instead assuming gradient-dominance holds (see \"Solving Robust MDPs through No Regret Dynamics\" or \"Global Optimality Guarantees For Policy Gradient Methods\" for examples). \n4. Have this paper's authors considered including some small toy experiments to verify their theoretical results? It would be interesting to see if they can achieve strong unlearning for linear classifiers with regularization under this unlearning framework. \n\nI also believe this paper is missing a relevant citation: \"Accelerated Perceptrons and Beyond\" analyzes the generalization/margin of minimax learners in linear or convex-concave settings. This, however, will not impact my score.\n\nLIMITATIONS:\nYes, they have sufficiently addressed the limitations of their works.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMachine unlearning is a privacy-inspired area to remove certain training samples of users’ data from well-trained model making those samples uninfluential while the unlearning approach does not cause the model to be retrained from scratch (not cause comprehensive computational cost) to achieve the baseline relied on the rest of datasets. The conventional machine unlearning usually applies for standard statistical learning models with one variable, but there is limited work for minimax models. Borrowing from Gaussian mechanism of differential privacy, this work also focuses on epsilon-delta certification of machine unlearning. Moreover, generalization rates and deletion capacity (number of deleted samples) are also very crucial to study in this framework.\n\nSTRENGTHS:\n1. Machine unlearning is a quite new field for privacy of machine learning. This work smoothly introduces the field with related work (a lot of citations) and preliminaries. Also it well states the reason minimax model is important for machine unlearning.\n2. Certified minimax unlearning can be well extended to more general loss functions with reasonable generalization rates and deletion capacity results.\n3. This work supports successive and online deletion requests, which is computationally efficient for unlearning phase with its minimax unlearning update.\n\nWEAKNESSES:\n1. Although this work focuses more on theoretical part of unlearning, it is still better to provide some preliminary results on some datasets. At least in one paragraph of intro or conclusion, authors can discuss scenarios or applications how to make the certified minimax unlearning practical.\n2. This work does not discuss the relationships of certified unlearning between STL model and minimax model theoretically (How are they different).\n\nQUESTIONS:\n1. Compared to other approaches for privacy of machine learning like differential privacy, what is the advantage of machine unlearning for the privacy? Since this work focuses on minimax model, the question could narrow down the specific model.\n2. On Line 98, is it intentionally “extract” or should it be “exact”?\n\nLIMITATIONS:\n1. As weakness 1, authors can consider some implications for the applications by adding a paragraph. Like in the beginning of the introduction, how does this work help GAN, ARL or RL?\n2. As weakness 2, this work emphasizes how important the certified unlearning for minimax model, but this work does not talk too much about the connection to unlearning for STL models.\n3. Instead of putting every proof in the appendix, it is better to add one or two sentences for each theorem or lemma about the clue to prove (what methods will you use? How will you prove?) to convince readers who are interested in the topic but not familiar.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe papers proposes a Newton-based differentially private algorithm for stochastic minimax problems and analyse the generalization rate and deletion capacity for the algorithms.\nThey analyse the generalization bound for weak primal-dual risk in SC-SC, C-SC and SC-C cases. Also the deletion capacity they derive is $O(n/d^{1/4})$, which is better than the baseline result $O(n/d^{1/2})$. The results of generalization bound and deletion capacity match the bound in the pure minimazation case.\n\nSTRENGTHS:\nThe paper is the first one analysing certified machine unlearning for minimax problems and provide certain generalization and deletion capacity guarantee.\n\nWEAKNESSES:\nThere are several weaknesses and issues:\n1. The reviewer is not convinced that weak primal-dual risk is enough to capture the behavior of generalization for minimax problems. In recent literature, people begin to study the strong primal-dual risk for minimax problems even without strong convexity.\nFor example, the authors in ``What is a Good Metric to Study Generalization of Minimax Learners'' derive the result for generalization of strong primal dual  risk in convex-concave setting.\nTherefore, the reviewer thinks the paper can be improved if the authors can extend their result to strong PD risk. \n2. Suggestions for writing: The certified machine unlearning and deletion capacity might not be well-known to general audiums. However, in the abstract and begining of the paper, the authors use too many specific terminology for machine unlearning, deletion, etc. It is not friendly to the audiums. The reviewer suggests the authors can discuss more motivation for this paper and give more explanations for the intuition of these terminologies.\n3. Missing reference: ''Uniform convergence and generalization for nonconvex stochastic minimax problems'',   ''What is a Good Metric to Study Generalization of Minimax Learners''.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of machine learning for the minimax model. By using Newton's step update with the Hessian information of the leftover data and Gaussian Mechanism, the proposed method improves the deletion capacity from $O(n/d^{1/2})$ to $O(n/d^{1/4})$.\n\nSTRENGTHS:\n- Given the rise of GAN and the increasing concern for the privacy of participants in datasets used by GAN/ML models, minimax unlearning is an important problem to study and understand.\n\n- The proposed algorithm shows improvement in both strongly convex-strongly convex and the more general convex-concave settings.\n\n- The improvement of the deletion capacity from $O(n/d^{1/2})$ to $O(n/d^{1/4})$ could be quite significant for practical problems with high-dimensional features.\n\n- The efficient algorithm that dispenses the need for recomputing some of the Hessian information is a nice touch since computing the Hessian matrix can be very computationally expensive.\n\n- The paper is well-written overall.\n\nWEAKNESSES:\n- Some of the technical details are used before being defined properly. For example, in equation (7), $\\kappa$ is defined as $\\kappa = l/\\mu$ even though $l$ and $\\mu$ haven't been defined yet. These quantities are later mentioned in section 4 but it would be better for the readers if those quantities are defined right away.\n\nQUESTIONS:\n- I'm a bit confused about the intuition part where $\\nu^\\star_{S\\}- \\nu^\\star  \\approx -\\partial_{\\nu\\nu}^{-1}F_{S\\}(w_S^\\star,\\nu_S^\\star)\\dot\\partial_{\\nu w}^{-1}F_{S\\}(w_S^\\star,\\nu_S^\\star)(w^\\star_{S\\}- w^\\star)$.  Can the author explain how we can use linear approximation and the implicit function theorem to get this result? Also why do we want to leave out the second term in eq. 60?\n\n- Is there any benefit in doing the update in Algorithm 2 compared to Algorithm 3? As the authors have shown in the proofs of Lemma 1 and Lemma 19, the analyses are basically identical so I'm just curious if there's any good thing about the update of Algorithm 1 and why we want to do it in the first place?\n\n- Seems like there's no improvement in the result in SC-SC case compared to C-C case. Is this the artifact of the algorithm or is this something that usually happens in minimax learning?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies approximate unlearning for minimax problems. They design learning and unlearning procedures and provide bounds on deletion capacity in terms of generalization performance (weak gap). Akin to minimization (statistical learning), the deletion capacity for strongly convex-strongly concave setting, is shown to be, $n/d^{1/4}$, where $n$ is the number of samples and $d$, dimension. The authors also provide extensions to non strongly convex/concave settings and efficient updates.\n\nSTRENGTHS:\nThe problem of machine unlearning has gathered a lot of interest recently, owing to various privacy regulations. Further, the minimax formulation, is widely applicable, especially in robust adversarial learning and reinforcement learning. The paper is the first to study unlearning for minimax settings. Therefore, the topic of the work is natural and timely.\n\nWEAKNESSES:\n1. The paper very closely follows the outline and techniques in prior work of Sekhari et. al. The extensions, to non-strongly convex settings (via regularization) and efficient updates also use techniques directly from prior work. If there are additional challenges due to the minimax (as opposed to min) structure, then I don't think they are communicated well in the write-up. The only relevant section is Section 4.1 \"Intuition for Minimax Unlearning Update\", however, I found it too raw to convey the intuition -- for instance, how does Eqn. 10 follow? In the current state of the write-up, it is difficult to evaluate if there are significant challenges overcome in relation to prior works.\n\n2. Comparison between Section 4.3 and 5.2: It seems to me that both are in the same setting, and achieve the same guarantees, yet Section 5.2 provides a more efficient update. If indeed Section 5.2 is a strict improvement over 4.3, then what is the point of devoting considerable space to the weaker result in Section 4.3.  The authors should re-organize and present the strongest result in the main paper. The space should be used to explain the challenges compared to the minimization setting. If this is not the case, then please explain the differences.\n\n3. Strong gap vs weak gap: The generalization performance considered in the paper is the weak primal-dual gap. In the non-private setting, \"strong\" primal-dual gap (as opposed to weak) is what is typically considered. Further, as explained in Bassily et al. 2023, the strong gap criterion has game-theoretic interpretation and motivation, and moreover, the weak and strong gaps may be arbitrarily apart. Seemingly, the consideration of the weak gap in the paper primarily stems from challenges of studying strong gap under privacy settings, until recently. However, given that the work Bassily et al. 2023 has established optimal rates for strong gap under privacy, can the authors, perhaps borrowing techniques from the aforementioned work, also provide guarantees in terms of the strong gap?\n\nQUESTIONS:\n1. Please answer the question posed in weakness 1.\n2. Please answer the question posed in weakness 2.\n3. Please answer the question posed in weakness 3.\n\nLIMITATIONS:\n1 The work is limited to the approximate unlearning setting, as opposed to exact unlearning. \n2. The authors study generalization performance in terms of weak gap as opposed to strong gap.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}