SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where we let  $\mu = \min \{\mu_{\omega},\mu_{\nu}\}$ ,  $\kappa = \ell /\mu$ ,  $d = \max \{d_1,d_2\}$ , and  $m$  be the edit distance between datasets (i.e., the original dataset and the remaining dataset after removing samples to be forgotten).
The algorithm  $A$  satisfies  $(\epsilon, \delta)$ -DP for any set  $U \subseteq S$  of size  $m$ , that is,
$$
\Pr [ A (S) \in O ] \leq e ^ {\epsilon} \Pr [ A (S \backslash U) \in O ] + \delta \quad \text {a n d} \quad \Pr [ A (S \backslash U) \in O ] \leq e ^ {\epsilon} \Pr [ A (S) \in O ] + \delta .
$$
Since we have  $A(S) = \bar{A}(U, A(S), T(S))$  and  $A(S \setminus U) = \bar{A}(\emptyset, A(S \setminus U), T(S \setminus U))$ , the above privacy guarantee can be converted to the minimax unlearning guarantee in Definition 2, implying that the pair  $(A, \bar{A})$  is  $(\epsilon, \delta)$ -certified minimax unlearning. According to Definition 3, the population strong PD risk in eq.(7) yields the following bound on deletion capacity.
Theorem 1 (Deletion capacity of unlearning via DP [Sekhari et al., 2021]). Denote  $d = \max \{d_1, d_2\}$ . There exists a polynomial time learning algorithm  $A$  and unlearning algorithm  $\bar{A}$  for minimax problem of the form  $\bar{A}(U, A(S), T(S)) = A(S)$  such that the deletion capacity is:
$$
m _ {\epsilon , \delta , \gamma} ^ {A, \bar {A}} \left(d _ {1}, d _ {2}, n\right) \geq \widetilde {\Omega} \left(\frac {n \epsilon}{\sqrt {d \log \left(e ^ {\epsilon} / \delta\right)}}\right), \tag {8}
$$
where the constant in  $\widetilde{\Omega}$ -notation depends on the properties of the loss function  $f$  (e.g., strongly convexity and strongly concavity parameters, Lipschitz continuity and smoothness parameters).
However, this DP minimax learning baseline approach provides an inferior deletion capacity. In the following sections, we show that the  $d^{1/2}$  in the denominator of eq.(8) can be further reduced to  $d^{1/4}$ .
4 Certified Minimax Unlearning
In this section, we focus on the setting of the strongly-convex-strong-concave loss function. We first provide the intuition for the design of the minimax unlearning step in Sec.4.1, then provide the formal algorithm in Sec.4.2 and a more efficient extension in Sec.4.3 with analysis of minimax unlearning certification, generalization result, and deletion capacity in Sec.4.4. We will provide extensions to more general loss settings in Sec.5. The proofs for the theorems presented in this and the next sections can be found in Appendix B and C, respectively.
4.1 Intuition for Minimax Unlearning Update
To begin with, we provide an informal derivation for minimax unlearning update to illustrate its design intuition. Given the training set  $S$  of size  $n$  and the deletion subset  $U \subseteq S$  of size  $m$ , the aim is to approximate the optimal solution  $(\pmb{\omega}_{S^{\backslash}}^{*},\pmb{\nu}_{S^{\backslash}}^{*})$  of the loss  $F_{S^{\backslash}}(\pmb {\omega},\pmb {\nu})$  on the remaining dataset  $S\setminus U$ , given by,
$$
\left(\boldsymbol {\omega} _ {S \backslash} ^ {*}, \boldsymbol {\nu} _ {S \backslash} ^ {*}\right) := \arg \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \left\{F _ {S \backslash} (\boldsymbol {\omega}, \boldsymbol {\nu}) := \frac {1}{n - m} \sum_ {z _ {i} \in S \backslash U} f (\boldsymbol {\omega}, \boldsymbol {\nu}; z _ {i}) \right\}. \tag {9}
$$
Meanwhile, we have the optimal solution  $(\omega_{S}^{*},\nu_{S}^{*})$  to the original loss  $F_{S}(\omega ,\nu)$  after minimax learning. Taking unlearning  $\pmb{\omega}$  for instance, by using a first-order Taylor expansion for  $\nabla_{\omega}F_{S\backslash}(\omega_{S\backslash}^{*},\nu_{S\backslash}^{*}) = 0$  around  $(\omega_{S}^{*},\nu_{S}^{*})$ , we have
$$
\nabla_ {\boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) + \partial_ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \left(\boldsymbol {\omega} _ {S \backslash} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*}\right) + \partial_ {\boldsymbol {\omega} \boldsymbol {\nu}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \left(\boldsymbol {\nu} _ {S \backslash} ^ {*} - \boldsymbol {\nu} _ {S} ^ {*}\right) \approx 0. \tag {10}
$$
Since  $\omega_{S}^{*}$  is a minimizer of  $F_{S}(\omega, \nu)$ , from the first-order optimality condition, we can get  $\nabla_{\omega} F_{S \setminus} (\omega_{S}^{*}, \nu_{S}^{*}) = -\frac{1}{n - m} \sum_{z_i \in U} \nabla_{\omega} f(\omega_{S}^{*}, \nu_{S}^{*}; z_i)$ . Now given an auxiliary function  $\mathsf{V}_{S \setminus} (\omega) = \operatorname{argmax}_{\nu \in V} F_{S \setminus} (\omega, \nu)$  (more best response auxiliary functions are introduced in Appendix A, Definition 8), we have  $\nu_{S \setminus}^{*} = \mathsf{V}_{S \setminus} (\omega_{S \setminus}^{*})$ . We further get
$$
\begin{array}{l} \boldsymbol {\nu} _ {S ^ {\backslash}} ^ {*} - \boldsymbol {\nu} _ {S} ^ {*} = \left[ \mathrm {V} _ {S ^ {\backslash}} \left(\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*}\right) - \mathrm {V} _ {S ^ {\backslash}} \left(\boldsymbol {\omega} _ {S} ^ {*}\right) \right] + \left[ \mathrm {V} _ {S ^ {\backslash}} \left(\boldsymbol {\omega} _ {S} ^ {*}\right) - \boldsymbol {\nu} _ {S} ^ {*} \right] \\ \stackrel {(i)} {\approx} \mathrm {V} _ {S \backslash} \left(\omega_ {S \backslash} ^ {*}\right) - \mathrm {V} _ {S \backslash} \left(\omega_ {S} ^ {*}\right) \stackrel {(i i)} {\approx} \left(\frac {\mathrm {d V} _ {S \backslash} (\omega)}{\mathrm {d} \omega} \Big | _ {\omega = \omega_ {S} ^ {*}}\right) \left(\omega_ {S \backslash} ^ {*} - \omega_ {S} ^ {*}\right) \tag {11} \\ \stackrel {(i i i)} {\approx} - \partial_ {\boldsymbol {\nu} \boldsymbol {\nu}} ^ {- 1} F _ {S \setminus} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}) \cdot \partial_ {\boldsymbol {\nu} \boldsymbol {\omega}} F _ {S \setminus} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}) \cdot (\boldsymbol {\omega} _ {S \setminus} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*}), \\ \end{array}
$$
where the approximate equation (i) leaving out the term  $\left[\mathrm{V}_{S\backslash}\left(\boldsymbol{\omega}_{S}^{*}\right)-\boldsymbol{\nu}_{S}^{*}\right]$  which is bounded in Appendix A, Lemma 2, and does not affect the overall unlearning guarantee. The approximate equation (ii) is the linear approximation step and is the response Jacobian of the auxiliary function  $\mathrm{V}_{S\backslash}(\omega)$ . The approximate equation (iii) is due to the implicit function theorem. This gives that
$$
\partial_ {\omega \omega} F _ {S \backslash} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \left(\omega_ {S \backslash} ^ {*} - \omega_ {S} ^ {*}\right) + \partial_ {\omega \nu} F _ {S \backslash} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \left(\nu_ {S \backslash} ^ {*} - \nu_ {S} ^ {*}\right) = D _ {\omega \omega} F _ {S \backslash} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \left(\omega_ {S \backslash} ^ {*} - \omega_ {S} ^ {*}\right), \tag {12}
$$
which implies the following approximation of  $\omega_{S^{\backslash}}^{*}$ :
$$
\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} \approx \boldsymbol {\omega} _ {S} ^ {*} + \frac {1}{n - m} \left[ \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S ^ {\backslash}} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right). \tag {13}
$$
The above informal derivation indicates that the minimax unlearning update relies on the total Hessian to sufficiently remove the data influence [Liu et al., 2023, Zhang et al., 2023], rather than the conventional Hessian that appears in standard statistical unlearning [Guo et al., 2020, Sekhari et al., 2021, Suriyakumar and Wilson, 2022, Mehta et al., 2022]. The update in eq.(13) has a close relation to the complete Newton step in the second-order minimax optimization literature [Zhang et al., 2020], which motivates the complete Newton-based minimax unlearning. However, due to the various approximations in the above informal derivation, we cannot have a certified minimax unlearning guarantee. Below, we will formally derive the upper bound for these approximations in the closeness upper bound analysis. Based on the closeness upper bound, we will introduce the Gaussian mechanism to yield distribution indistinguishably result in the sense of  $(\epsilon, \delta)$ -certified minimax unlearning.
4.2 Proposed Certified Minimax Unlearning
We first provide algorithms under the setting of the smooth and strongly-convex-strongly-concave (SC-SC) loss function as described in Assumptions 1&2.
Algorithm 1 Minimax Learning Algorithm  $(A_{sc - sc})$
Input: Dataset  $S:\{z_i\}_{i = 1}^n\sim \mathcal{D}^n$  , loss function:  $f(\omega ,\nu ;z)$
1: Compute
$$
\left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \leftarrow \arg \min  _ {\boldsymbol {\omega}} \max  _ {\boldsymbol {\nu}} F _ {S} (\boldsymbol {\omega}, \boldsymbol {\nu}) = \frac {1}{n} \sum_ {i = 1} ^ {n} f (\boldsymbol {\omega}, \boldsymbol {\nu}; z _ {i}). \tag {14}
$$
Output:  $(\omega_{S}^{*},\nu_{S}^{*},\mathrm{D}_{\omega \omega}F_{S}(\omega_{S}^{*},\nu_{S}^{*}),\mathrm{D}_{\nu \nu}F_{S}(\omega_{S}^{*},\nu_{S}^{*}))$
Algorithm 2 Certified Minimax Unlearning for Strongly-Convex-Strongly-Concave Loss  $(\bar{A}_{sc - sc})$
Input: Delete requests  $U: \{z_j\}_{j=1}^m \subseteq S$ , output of  $A_{sc-sc}(S)$ :  $(\omega_S^*, \nu_S^*)$ , memory variables  $T(S)$ :
$\{\mathsf{D}_{\omega \omega}F_S(\pmb {\omega}_S^*,\pmb {\nu}_S^*),\mathsf{D}_{\nu \nu}F_S(\pmb {\omega}_S^*,\pmb {\nu}_S^*)\}$  , loss function:  $f(\pmb {\omega},\pmb {\nu};z)$  , noise parameters:  $\sigma_1,\sigma_2$
$$
\mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) = \frac {1}{n - m} \left(n \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) - \sum_ {z _ {i} \in U} \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right)\right), \tag {15}
$$
$$
\mathbb {D} _ {\boldsymbol {\nu} \boldsymbol {\nu}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) = \frac {1}{n - m} \left(n \mathbb {D} _ {\boldsymbol {\nu} \boldsymbol {\nu}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) - \sum_ {z _ {i} \in U} \mathbb {D} _ {\boldsymbol {\nu} \boldsymbol {\nu}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right)\right). \tag {16}
$$
2: Define
$$
\widehat {\boldsymbol {\omega}} = \boldsymbol {\omega} _ {S} ^ {*} + \frac {1}{n - m} \left[ \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right), \tag {17}
$$
$$
\widehat {\boldsymbol {\nu}} = \boldsymbol {\nu} _ {S} ^ {*} + \frac {1}{n - m} \left[ \mathbb {D} _ {\boldsymbol {\nu} \boldsymbol {\nu}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\nu}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right). \tag {18}
$$
3:  $\pmb{\omega}^{u} = \widehat{\pmb{\omega}} +\pmb{\xi}_{1}$ , where  $\pmb {\xi}_1\sim \mathcal{N}(0,\sigma_1\mathbf{I}_{d_1})$  and  $\pmb {\nu}^u = \widehat{\pmb{\nu}} +\pmb{\xi}_2$ , where  $\pmb {\xi}_2\sim \mathcal{N}(0,\sigma_2\mathbf{I}_{d_2})$ .
Output:  $(\omega^u,\nu^u)$
Minimax Learning algorithm. We denote our learning algorithm by  $A_{sc - sc}$  and the pseudocode is shown in Algorithm 1. Given a dataset  $S = \{z_i\}_{i=1}^n$  of size  $n$  drawn independently from some distribution  $\mathcal{D}$ , algorithm  $A_{sc - sc}$  computes the optimal solution  $(\omega_S^*, \nu_S^*)$  to the empirical risk  $F_S(\omega, \nu)$ .  $A_{sc - sc}$  then outputs the point  $(\omega_S^*, \nu_S^*)$  as well as the additional memory variables  $T(S) := \{\mathrm{D}_{\omega \omega} F_S(\omega_S^*, \nu_S^*), \mathrm{D}_{\nu \nu} F_S(\omega_S^*, \nu_S^*)\}$ , which computes and stores the total Hessian of  $F_S(\omega, \nu)$  at  $(\omega_S^*, \nu_S^*)$ .
Minimax Unlearning Algorithm We denote the proposed certified minimax unlearning algorithm by  $\bar{A}_{sc - sc}$  and present its pseudocode in Algorithm 2. Algorithm  $\bar{A}_{sc - sc}$  takes the following inputs: the set of delete requests  $U = \{z_{j}\}_{j = 1}^{m}$  of size  $m$ , the trained minimax model  $(\omega_{S}^{*},\nu_{S}^{*})$ , and the memory variables  $T(S)$ . To have the certified minimax unlearning for  $\omega$ , eq.(15) computes the total Hessian of  $F_{S\backslash}(\pmb{\omega}_S^*,\pmb{\nu}_S^*)$  by  $\frac{n}{n - m}\mathsf{D}\omega \omega F_S(\pmb{\omega}_S^*,\pmb{\nu}_S^*) - \frac{1}{n - m}\sum_{z_i\in U}\mathsf{D}\omega \omega f(\pmb{\omega}_S^*,\pmb{\nu}_S^*,z_i)$ , where the former term can be retrieved from the memory set and the latter is computed on the samples to be deleted; eq.(17) computes the intermediate  $\widehat{\omega}$  by the complete Newton step based on the total Hessian  $\mathsf{D}\omega \omega F_{S\backslash}(\pmb{\omega}_S^*,\pmb{\nu}_S^*)$ ; Line 3 injects calibrated Gaussian noise  $\xi_{1}$  to ensure  $(\epsilon ,\delta)$ -certified minimax unlearning. The certified minimax unlearning for  $\pmb{\nu}$  is symmetric. We provide detailed analysis for Algorithm 2 including minimax unlearning certification, generalization results, and deletion capacity in Appendix B.1.
4.3 Certified Minimax Unlearning without Total Hessian Re-computation
We extend Algorithm 2 and propose Algorithm 3 to reduce the computational cost of Algorithm 2. The complete Newton steps in eq.(19) and eq.(20) utilize the total Hessian  $\mathsf{D}_{\omega \omega}F_S(\boldsymbol{\omega}_S^*,\boldsymbol{\nu}_S^*)$  and  $\mathsf{D}_{\nu \nu}F_S(\boldsymbol{\omega}_S^*,\boldsymbol{\nu}_S^*)$  that are directly retrieved from the memory, rather than the updated total Hessian  $\mathsf{D}_{\omega \omega}F_{S\setminus}(\boldsymbol{\omega}_S^*,\boldsymbol{\nu}_S^*)$  and  $\mathsf{D}_{\nu \nu}F_{S\setminus}(\boldsymbol{\omega}_S^*,\boldsymbol{\nu}_S^*)$  used in Algorithm 2. The form in eq.(19) and eq.(20) can also be regarded as the total Hessian extension of the infinitesimal jackknife. In this way, it gets rid of the computationally demanding part of re-evaluating the total Hessian for samples to be deleted, which significantly reduces the computational cost. It turns out to be the same computational complexity as the state-of-the-art certified unlearning method developed for STL models [Suriyakumar and Wilson, 2022]. Moreover, Algorithm 3 can be more appealing for the successive data deletion setting.
Algorithm 3 Efficient Certified Minimax Unlearning  $(\bar{A}_{\text{efficient}})$
Input: Delete requests  $U:\{z_j\}_{j = 1}^m\subseteq S$  output of  $A_{sc - sc}(S)$ $(\omega_S^*,\nu_S^*)$  , memory variables  $T(S)$ $\{\mathrm{D}\omega \omega F_S(\pmb {\omega}_S^*,\pmb {\nu}_S^*),\mathrm{D}\nu \nu F_S(\pmb {\omega}_S^*,\pmb {\nu}_S^*)\}$  , loss function:  $f(\boldsymbol {\omega},\boldsymbol {\nu};z)$  , noise parameters:  $\sigma_1,\sigma_2$
$$
\widetilde {\omega} = \omega_ {S} ^ {*} + \frac {1}{n} \left[ \mathbb {D} _ {\omega \omega} F _ {S} \left(\omega_ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\omega} f \left(\omega_ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right), \tag {19}
$$
$$
\tilde {\boldsymbol {\nu}} = \boldsymbol {\nu} _ {S} ^ {*} + \frac {1}{n} \left[ \mathbb {D} _ {\boldsymbol {\nu} \boldsymbol {\nu}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\nu}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right). \tag {20}
$$
2:  $\widetilde{\pmb{\omega}}^u = \widetilde{\pmb{\omega}} +\pmb {\xi}_1$  , where  $\pmb {\xi}_1\sim \mathcal{N}(0,\sigma_1\mathbf{I}_{d_1})$  and  $\widetilde{\pmb{\nu}}^u = \widetilde{\pmb{\nu}} +\pmb {\xi}_2$  , where  $\pmb {\xi}_2\sim \mathcal{N}(0,\sigma_2\mathbf{I}_{d_2})$
Output:  $(\widetilde{\omega}^u,\widetilde{\nu}^u)$
4.4 Analysis for Algorithm 3
$(\epsilon, \delta)$ -Certificated Unlearning Guarantee. The intermediate variables  $(\widetilde{\omega}, \widetilde{\nu})$  are distinguishable in distribution from the retraining-from-scratch variables  $(\boldsymbol{\omega}_{S^{\backslash}}^{*}, \boldsymbol{\nu}_{S^{\backslash}}^{*})$  because they are deterministic and the Taylor expansion introduces a certain amount of approximation. The following lemma quantifies the closeness between  $(\widetilde{\omega}, \widetilde{\nu})$  and  $(\boldsymbol{\omega}_{S^{\backslash}}^{*}, \boldsymbol{\nu}_{S^{\backslash}}^{*})$ , which can be regarded as the "sensitivity" when applying the Gaussian mechanism.
Lemma 1 (Closeness Upper Bound). Suppose the loss function  $f$  satisfies Assumption 1 and 2,  $\| \mathbb{D}_{\omega \omega}F_S(\omega_S^*,\nu_S^*)\| \geq \mu_{\omega \omega}$  and  $\| \mathbb{D}_{\nu \nu}F_S(\omega_S^*,\nu_S^*)\| \geq \mu_{\nu \nu}$ . Let  $\mu = \min \{\mu_{\omega},\mu_{\nu},\mu_{\omega \omega},\mu_{\nu \nu}\}$ . Then, we have the closeness bound between  $(\widetilde{\omega},\widetilde{\nu})$  in Line 1 of Algorithm 3 and  $(\omega_{S^{\backslash}}^*,\nu_{S^{\backslash}}^*)$  in eq.(9):
$$
\left\{\| \omega_ {S ^ {\backslash}} ^ {*} - \widetilde {\omega} \|, \| \nu_ {S ^ {\backslash}} ^ {*} - \widetilde {\nu} \| \right\} \leq \frac {\left(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {6} + 2 \sqrt {2} L \ell^ {2} / \mu^ {3}\right) m ^ {2}}{n ^ {2}}. \tag {21}
$$
Equipped with Lemma 1, we have the following certified unlearning guarantee by adding Gaussian noise calibrated according to the above closeness result. Due to the minimax structure, our analysis is more involved than the STL case [Sekhari et al., 2021, Suriyakumar and Wilson, 2022].
Theorem 2  $((\epsilon, \delta)$ -Minimax Unlearning Certification). Under the same settings of Lemma 1, our minimax learning algorithm  $A_{sc-sc}$  and unlearning algorithm  $\bar{A}_{\text{efficient}}$  is  $(\epsilon, \delta)$ -certified minimax unlearning if we choose
$$
\sigma_ {1} \text {a n d} \sigma_ {2} = \frac {2 \left(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {6} + 2 \sqrt {2} L \ell^ {2} / \mu^ {3}\right) m ^ {2}}{n ^ {2} \epsilon} \sqrt {2 \log (2 . 5 / \delta)}. \tag {22}
$$
Generalization Guarantee. Theorem 3 below provides the generalization result in terms of the population PD risk for the minimax unlearning algorithm  $\bar{A}_{\mathrm{efficient}}$ .
Theorem 3 (Population Primal-Dual Risk). Under the same settings of Lemma 1 and denote  $d = \max \{d_1, d_2\}$ , the population weak and strong PD risk for the certified minimax unlearning variables  $(\widetilde{\omega}^u, \widetilde{\nu}^u)$  returned by Algorithm 3 are
$$
\left\{ \begin{array}{l} \triangle^ {w} \left(\widetilde {\boldsymbol {\omega}} ^ {u}, \widetilde {\boldsymbol {\nu}} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n}\right), \\ \triangle^ {s} \left(\widetilde {\boldsymbol {\omega}} ^ {u}, \widetilde {\boldsymbol {\nu}} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d (\log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n} + \frac {L ^ {2} \ell}{\mu^ {2} n}\right). \end{array} \right. \tag {23}
$$
Deletion Capacity. The population weak and strong PD risk given in Theorem 3 for the output of unlearning algorithms provides the following bound on deletion capacity.
Theorem 4 (Deletion Capacity). Under the same settings of Lemma 1 and denote  $d = \max \{d_1, d_2\}$ , the deletion capacity of Algorithm 3 is
$$
m _ {\epsilon , \delta , \gamma} ^ {A, \bar {A}} \left(d _ {1}, d _ {2}, n\right) \geq c \cdot \frac {n \sqrt {\epsilon}}{\left(d \log (1 / \delta)\right) ^ {1 / 4}}, \tag {24}
$$
where the constant  $c$  depends on  $L, l, \rho$ , and  $\mu$  of the loss function  $f$ .
5 Certified Minimax Unlearning for Convex-Concave Loss Function
We further extend the certified minimax unlearning for the convex-concave loss function. In addition, Appendix C will provide the extension to convex-strongly-concave and strongly-convex-concave loss functions. Give the convex-concave loss function  $f(\omega, \nu; z)$ , similar to the unlearning for STL models [Sekhari et al., 2021], we define the regularized function as  $\widetilde{f}(\omega, \nu; z) = f(\omega, \nu; z) + \frac{\lambda}{2} \| \omega \|^2 - \frac{\lambda}{2} \| \nu \|^2$ . Suppose the function  $f$  satisfies Assumption 1, then the function  $\widetilde{f}$  is  $\lambda$ -strongly convex in  $\omega$ ,  $\lambda$ -strongly concave in  $\nu$ ,  $(2L + \lambda \| \omega \| + \lambda \| \nu \|)$ -Lipschitz,  $\sqrt{2}(2\ell + \lambda)$ -gradient Lipschitz and  $\rho$ -Hessian Lipschitz. It suffices to apply the minimax learning and unlearning algorithms in Sec.4 to the regularized loss function with a properly chosen  $\lambda$ . We denote the learning and unlearning algorithms for convex-concave losses as  $A_{c-c}$  and  $A_{c-c}$ . Their implementation details are given in Appendix C. We suppose the SC-SC regularization parameter  $\lambda$  satisfies  $\lambda < \ell$ . Theorem 5 below summarizes guarantees of  $(\epsilon, \delta)$ -certified unlearning and population primal-dual risk (weak and strong) for Algorithm  $A_{c-c}$ .
Theorem 5. Let Assumption 1 hold and  $d = \max \{d_1, d_2\}$ . Suppose the parameter spaces  $\mathcal{W}$  and  $\mathcal{V}$  are bounded so that  $\max_{\boldsymbol{\omega} \in \mathcal{W}} \| \boldsymbol{\omega} \| \leq B_{\boldsymbol{\omega}}$  and  $\max_{\boldsymbol{\nu} \in \mathcal{V}} \| \boldsymbol{\nu} \| \leq B_{\boldsymbol{\nu}}$ . We have,
(a)  $(\epsilon, \delta)$ -Minimax Unlearning Certification: Our minimax learning algorithm  $A_{c-c}$  and unlearning algorithm  $A_{c-c}$  is  $(\epsilon, \delta)$ -certified minimax unlearning.
(b) Population Weak PD Risk: The population weak PD risk for  $(\omega^u,\nu^u)$  by algorithm  $\bar{A}_{c - c}$  is
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \lambda^ {6} + L ^ {2} \ell^ {2} / \lambda^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \lambda \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right)\right). \tag {25}
$$
In particular, by setting  $\lambda$  below
$$
\lambda = \max  \left\{\frac {L}{\sqrt {B _ {\omega} ^ {2} + B _ {\nu} ^ {2}}} \sqrt {\frac {m}{n}}, \left(\frac {L ^ {2} \ell^ {2} m ^ {2} \sqrt {d \log (1 / \delta)}}{\left(B _ {\omega} ^ {2} + B _ {\nu} ^ {2}\right) n ^ {2} \epsilon}\right) ^ {1 / 4}, \left(\frac {L ^ {3} \ell^ {3} \rho m ^ {2} \sqrt {d \log (1 / \delta)}}{\left(B _ {\omega} ^ {2} + B _ {\nu} ^ {2}\right) n ^ {2} \epsilon}\right) ^ {1 / 7} \right\}, \tag {26}
$$
we have the following population weak PD risk,
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(c _ {1} \sqrt {\frac {m}{n}} + c _ {2} \left(\frac {d \log (1 / \delta)}{\epsilon^ {2}}\right) ^ {1 / 8} \sqrt {\frac {m}{n}} + c _ {3} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 7} \left(\frac {m}{n}\right) ^ {2 / 7}\right), \tag {27}
$$
where  $c_{1}, c_{2}, c_{3}$  are constants that depend only on  $L, l, \rho, B_{\omega}$  and  $B_{\nu}$ .