SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \nabla_ {\omega} F _ {S} \left(\omega , \mathrm {V} _ {S} (\omega)\right) - \nabla_ {\omega} F _ {S} \left(\omega^ {\prime}, \mathrm {V} _ {S} \left(\omega^ {\prime}\right)\right) - \mathrm {D} _ {\omega \omega} F _ {S} \left(\omega^ {\prime}\right) \left(\omega - \omega^ {\prime}\right) \right\| \leq \frac {M}{2} \| \omega - \omega^ {\prime} \| ^ {2}, \tag {54}
$$
where  $M = 4\sqrt{2}\kappa_{\pmb{\nu}}^{3}\rho$
Proof. Recall the definition of the primal function  $P(\omega) \coloneqq \max_{\nu \in \mathcal{V}} F_S(\omega, \nu)$  and its gradient  $\nabla P(\omega) = \nabla_\omega F_S(\omega, \mathrm{V}_S(\omega))$ . Due to the optimality of  $\mathrm{V}_S$ , we have
$$
\nabla_ {\nu} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) = 0. \tag {55}
$$
By taking the total derivative with respect to  $\omega$ , we get
$$
\partial_ {\nu \omega} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) + \partial_ {\nu \nu} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) \nabla \mathrm {V} _ {S} (\omega) = 0. \tag {56}
$$
Taking the total derivative of  $\omega$  again on  $\nabla P(\omega)$ , we further have
$$
\begin{array}{l} \nabla^ {2} P (\omega) = \partial_ {\omega \omega} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) + \partial_ {\omega \nu} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) \nabla \mathrm {V} _ {S} (\omega) \\ = \partial_ {\omega \omega} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) - \partial_ {\omega \nu} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) \partial_ {\nu \nu} ^ {- 1} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) \partial_ {\nu \omega} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) \tag {57} \\ = \mathrm {D} _ {\omega \omega} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)). \\ \end{array}
$$
Based on the equality of  $\nabla^2 P(\omega)$  and  $\mathrm{D}_{\omega \omega}F_S(\omega ,\mathrm{V}_S(\omega))$  above and Lemma 5, we have
$$
\begin{array}{l} \left\| \nabla_ {\omega} F _ {S} (\omega , \mathrm {V} _ {S} (\omega)) - \nabla_ {\omega} F _ {S} \left(\omega^ {\prime}, \mathrm {V} _ {S} \left(\omega^ {\prime}\right)\right) - \mathrm {D} _ {\omega \omega} F _ {S} \left(\omega^ {\prime}\right) \left(\omega - \omega^ {\prime}\right) \right\| \\ = \left\| \nabla P (\omega) - \nabla P \left(\omega^ {\prime}\right) - \nabla^ {2} P \left(\omega^ {\prime}\right) \left(\omega - \omega^ {\prime}\right) \right\| \tag {58} \\ \leq \frac {M}{2} \| \boldsymbol {\omega} - \boldsymbol {\omega} ^ {\prime} \| ^ {2}. \\ \end{array}
$$
Lemma 7. Under Assumption 1 and Assumption 2, for all  $\omega \in \mathcal{W}$  and  $\pmb{\nu} \in \mathcal{V}$ , we have  $\| \mathrm{D}_{\omega \omega} f(\omega, \pmb{\nu}; z) \| \leq \ell + \frac{\ell^2}{\mu_{\nu}}$ .
Proof. By the definition of the total Hessian, we have
$$
\begin{array}{l} \| \mathbb {D} _ {\omega \omega} f (\omega , \nu ; z) \| = \| \partial_ {\omega \omega} f (\omega , \nu ; z) - \partial_ {\omega \nu} f (\omega , \nu ; z) \partial_ {\nu \nu} ^ {- 1} f (\omega , \nu ; z) \partial_ {\nu \omega} f (\omega , \nu ; z) \| \\ \stackrel {(i)} {\leq} \left\| \partial_ {\omega \omega} f (\omega , \nu ; z) \right\| + \left\| \partial_ {\omega \nu} f (\omega , \nu ; z) \partial_ {\nu \nu} ^ {- 1} f (\omega , \nu ; z) \partial_ {\nu \omega} f (\omega , \nu ; z) \right\| \tag {59} \\ \stackrel {(i i)} {\leq} \ell + \ell \cdot \mu_ {\boldsymbol {\nu}} ^ {- 1} \cdot \ell = \ell + \frac {\ell^ {2}}{\mu_ {\boldsymbol {\nu}}}, \\ \end{array}
$$
where the inequality  $(i)$  uses the triangle inequality and the inequality  $(ii)$  is due to the function  $f$  has  $\ell$ -Lipschitz gradients and  $f$  is  $\mu_{\nu}$ -strongly concave in  $\pmb{\nu}$ , thus we have  $\| \partial_{\omega \omega} f(\omega, \nu; z) \| \leq \ell$ ,  $\| \partial_{\omega \nu} f(\omega, \nu; z) \| \leq \ell$ ,  $\| \partial_{\nu \omega} f(\omega, \nu; z) \| \leq \ell$  and  $\| \partial_{\nu \nu} f(\omega, \nu; z) \| \leq \mu_{\nu}^{-1}$ .
Lemma 8 ([Zhang et al., 2022a, Lemma 4.4]). Under Assumption 1 and Assumption 2, the population weak PD risk for the minimax learning variables  $(\omega_S^*, \nu_S^*)$  returned by Algorithm 1 has
$$
\triangle^ {w} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \leq \frac {2 \sqrt {2} L ^ {2}}{\mu n}. \tag {60}
$$
Lemma 9 ([Zhang et al., 2021, Theorem 2]). Under Assumption 1 and Assumption 2, the population strong PD risk for the minimax learning variables  $(\omega_{S}^{*},\nu_{S}^{*})$  returned by Algorithm 1 has
$$
\triangle^ {s} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \leq \frac {2 \sqrt {2} L ^ {2}}{n} \cdot \sqrt {\frac {\ell^ {2}}{\mu_ {\boldsymbol {\omega}} \mu_ {\boldsymbol {\nu}}} + 1} \cdot \left(\frac {1}{\mu_ {\boldsymbol {\omega}}} + \frac {1}{\mu_ {\boldsymbol {\nu}}}\right) \leq \frac {8 L ^ {2} \ell}{\mu^ {2} n}. \tag {61}
$$
B Detailed Algorithm Analysis and Missing Proofs in Section 4
B.1 Analysis for Algorithm 2
In the following, we provide the analysis for Algorithm 2 in terms of guarantees of  $(\epsilon, \delta)$ -certified unlearning, population primal-dual risk, and deletion capacity and the corresponding proofs.
Lemma 10 (Closeness Upper Bound). Suppose the loss function  $f$  satisfies Assumption 1 and 2,  $\| \mathbb{D}_{\omega \omega}F_S(\omega_S^*,\nu_S^*)\| \geq \mu_{\omega \omega}$  and  $\| \mathbb{D}_{\nu \nu}F_S(\omega_S^*,\nu_S^*)\| \geq \mu_{\nu \nu}$ . Let  $\mu = \min \{\mu_{\omega},\mu_{\nu},\mu_{\omega \omega},\mu_{\nu \nu}\}$ . Then, we have the closeness bound between  $(\widehat{\omega},\widehat{\nu})$  in Line 2 of Algorithm 2 and  $(\omega_{S^{\backslash}}^{*},\nu_{S^{\backslash}}^{*})$  in eq.(9):
$$
\left\{\| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\omega}} \|, \| \boldsymbol {\nu} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\nu}} \| \right\} \leq \frac {(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {5} + 8 L \ell^ {2} / \mu^ {2}) m ^ {2}}{n (\mu n - (\ell + \ell^ {2} / \mu) m)}. \tag {62}
$$
Proof. Recall that the empirical loss functions  $F_{S^{\setminus}}(\omega, \nu)$  and  $F_{S}(\omega, \nu)$  are
$$
F _ {S \backslash} (\boldsymbol {\omega}, \boldsymbol {\nu}) := \frac {1}{n - m} \sum_ {z _ {i} \in S \backslash U} f (\boldsymbol {\omega}, \boldsymbol {\nu}; z _ {i}), \quad \text {a n d} \quad F _ {S} (\boldsymbol {\omega}, \boldsymbol {\nu}) := \frac {1}{n} \sum_ {z _ {i} \in S} f (\boldsymbol {\omega}, \boldsymbol {\nu}; z _ {i}). \tag {63}
$$
We focus on the key term  $\nabla_{\omega}F_{S^{\backslash}}(\omega_{S^{\backslash}}^{*},\nu_{S}(\omega_{S^{\backslash}}^{*})) - \nabla_{\omega}F_{S^{\backslash}}(\omega_{S}^{*},\nu_{S}^{*}) - \mathsf{D}_{\omega \omega}F_{S^{\backslash}}(\omega_{S}^{*},\nu_{S}^{*})(\omega_{S^{\backslash}}^{*} - \omega_{S}^{*})$  which has the following conversions
$$
\begin{array}{l} \left\| \nabla_ {\omega} F _ {S \backslash} \left(\omega_ {S ^ {\backslash}} ^ {*}, \mathrm {V} _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}\right)\right) - \nabla_ {\omega} F _ {S \backslash} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) - \mathrm {D} _ {\omega \omega} F _ {S \backslash} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \left(\omega_ {S ^ {\backslash}} ^ {*} - \omega_ {S} ^ {*}\right) \right\| \\ = \| \frac {n}{n - m} [ \nabla_ {\omega} F _ {S} (\omega_ {S ^ {\backslash}} ^ {*}, \mathsf {V} _ {S} (\omega_ {S ^ {\backslash}} ^ {*})) - \nabla_ {\omega} F _ {S} (\omega_ {S} ^ {*}, \nu_ {S} ^ {*}) - \mathsf {D} _ {\omega \omega} F _ {S} (\omega_ {S} ^ {*}, \nu_ {S} ^ {*}) (\omega_ {S ^ {\backslash}} ^ {*} - \omega_ {S} ^ {*}) ] \\ - \frac {1}{n - m} \sum_ {z _ {i} \in U} \left[ \nabla_ {\omega} f \left(\omega_ {S ^ {\backslash}} ^ {*}, \mathrm {V} _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}\right); z _ {i}\right) - \nabla_ {\omega} f \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}; z _ {i}\right) \right] \\ + \frac {1}{n - m} \sum_ {z _ {i} \in U} \mathrm {D} _ {\omega \omega} f \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}; z _ {i}\right) \left(\omega_ {S \backslash} ^ {*} - \omega_ {S} ^ {*}\right) \| \tag {64} \\ \leq \frac {n}{n - m} \| \nabla_ {\boldsymbol {\omega}} F _ {S} (\boldsymbol {\omega} _ {S ^ {\setminus}} ^ {*}, \mathsf {V} _ {S} (\boldsymbol {\omega} _ {S ^ {\setminus}} ^ {*})) - \nabla_ {\boldsymbol {\omega}} F _ {S} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}) - \mathsf {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}) (\boldsymbol {\omega} _ {S ^ {\setminus}} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*}) \| \\ + \frac {1}{n - m} \sum_ {z _ {i} \in U} \| \nabla_ {\boldsymbol {\omega}} f (\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*}, \mathsf {V} _ {S} (\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*}); z _ {i}) - \nabla_ {\boldsymbol {\omega}} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) \| \\ + \frac {1}{n - m} \| \sum_ {z _ {i} \in U} \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) (\boldsymbol {\omega} _ {S \backslash} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*}) \|. \\ \end{array}
$$
We denote  $\kappa_{\nu} = \ell /\mu_{\nu}$ . For the first term on the right-hand side of the inequality in eq.(64), we have
$$
\begin{array}{l} \frac {n}{n - m} \| \nabla_ {\boldsymbol {\omega}} F _ {S} (\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*}, \mathsf {V} _ {S} (\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*})) - \nabla_ {\boldsymbol {\omega}} F _ {S} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}) - \mathsf {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}) (\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*}) \| \\ \stackrel {(i)} {\leq} \frac {n}{n - m} \cdot 2 \sqrt {2} \kappa_ {\nu} ^ {3} \rho \| \boldsymbol {\omega} _ {S \backslash} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*} \| ^ {2} \stackrel {(i i)} {\leq} \frac {8 \sqrt {2} \kappa_ {\nu} ^ {3} \rho L ^ {2} m ^ {2}}{\mu_ {\omega} ^ {2} n (n - m)} \leq \frac {8 \sqrt {2} \rho L ^ {2} \ell^ {3} m ^ {2}}{\mu^ {5} n (n - m)}, \tag {65} \\ \end{array}
$$
where the inequality  $(i)$  is by Lemma 6 and the inequality  $(ii)$  is by Lemma 3.
For the second term on the right-hand side of the inequality in eq.(64), we have
$$
\begin{array}{l} \frac {1}{n - m} \sum_ {z _ {i} \in U} \| \nabla_ {\omega} f (\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*}, \mathbf {V} _ {S} (\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*}); z _ {i}) - \nabla_ {\omega} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) \| \\ \stackrel {(i)} {\leq} \frac {1}{n - m} \cdot m \ell \sqrt {\| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*} \| ^ {2} + \| \mathrm {V} _ {S} \left(\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*}\right) - \mathrm {V} _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}\right) \| ^ {2}} \tag {66} \\ \stackrel {(i i)} {\leq} \frac {1}{n - m} \cdot m \ell \sqrt {\| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*} \| ^ {2} + \kappa_ {\nu} ^ {2} \| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \boldsymbol {\omega} _ {S} ^ {*} \| ^ {2}} \\ \stackrel {(i i i)} {\leq} \frac {2 L l m ^ {2} \sqrt {1 + \kappa_ {\nu} ^ {2}}}{\mu_ {\omega} n (n - m)} \leq \frac {2 \sqrt {2} L l \kappa_ {\nu} m ^ {2}}{\mu n (n - m)} \leq \frac {2 \sqrt {2} L l ^ {2} m ^ {2}}{\mu^ {2} n (n - m)}, \\ \end{array}
$$
where the inequality  $(i)$  follows by the fact that the function  $\nabla_{\omega}f(\cdot ,\cdot)$  is  $\ell$  -Lipschitz continuous and  $\pmb{\nu}_{S}^{*} = \mathrm{V}_{S}(\pmb{\omega}_{S}^{*})$  . The inequality (ii) holds because Lemma 4, and the inequality (iii) is by Lemma 3.
For the third term on the right-hand side of the inequality in eq.(64), we have
$$
\begin{array}{l} \frac {1}{n - m} \| \sum_ {z _ {i} \in U} \mathsf {D} _ {\pmb {\omega} \pmb {\omega}} f (\pmb {\omega} _ {S} ^ {*}, \pmb {\nu} _ {S} ^ {*}; z _ {i}) (\pmb {\omega} _ {S \setminus} ^ {*} - \pmb {\omega} _ {S} ^ {*}) \| \\ \leq \left(\ell + \frac {\ell^ {2}}{\mu_ {\nu}}\right) \cdot \frac {2 L m ^ {2}}{\mu_ {\omega} n (n - m)} \leq \frac {4 L \ell^ {2} m ^ {2}}{\mu^ {2} n (n - m)}, \tag {57} \\ \end{array}
$$
where the first inequality is by Lemma 7. Plugging eq.(65), eq.(66) and eq.(67) into eq.(64), we further get
$$
\begin{array}{l} \left\| \nabla_ {\omega} F _ {S \backslash} \left(\omega_ {S \backslash} ^ {*}, \mathrm {V} _ {S} \left(\omega_ {S \backslash} ^ {*}\right)\right) - \nabla_ {\omega} F _ {S \backslash} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) - \mathrm {D} _ {\omega \omega} F _ {S \backslash} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \left(\omega_ {S \backslash} ^ {*} - \omega_ {S} ^ {*}\right) \right\| \\ \leq \left(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {5} + 8 L \ell^ {2} / \mu^ {2}\right) \frac {m ^ {2}}{n (n - m)}. \tag {68} \\ \end{array}
$$
The above derivation yields an upper bound result. In the following, we derive a lower bound result. Let  $\mathbf{x}$  be the vector satisfying the following relation,
$$
\omega_ {S \backslash} ^ {*} = \omega_ {S} ^ {*} + \frac {1}{n - m} \left[ \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \mathbf {x}. \tag {69}
$$
Since we have  $\nabla_{\omega}F_{S\setminus}(\pmb{\omega}_{S}^{*},\pmb{\nu}_{S}^{*}) = -\frac{1}{n - m}\sum_{z_{i}\in U}\nabla_{\omega}f(\pmb{\omega}_{S}^{*},\pmb{\nu}_{S}^{*};z_{i})$  and  $\nabla_{\omega}F_{S\setminus}(\pmb{\omega}_{S\setminus}^{*},\mathsf{V}_{S}(\pmb{\omega}_{S\setminus}^{*})) = 0$  due to the optimality of  $\pmb{\omega}_{S\setminus}^{*}$ , plugging eq.(69) into eq.(68), we get that
$$
\left\| \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \cdot \mathbf {x} \right\| \leq \left(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {5} + 8 L \ell^ {2} / \mu^ {2}\right) \frac {m ^ {2}}{n (n - m)}. \tag {70}
$$
For the left-hand side of eq.(70), with  $\| \mathbb{D}_{\omega \omega}F_S(\pmb{\omega}_S^*,\pmb{\nu}_S^*)\| \geq \mu_{\omega \omega}$ , we have
$$
\begin{array}{l} \| \mathrm {D} _ {\omega \omega} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \cdot \mathbf {x} \| = \frac {1}{n - m} \| \left[ \sum_ {z _ {i} \in S \backslash U} \mathrm {D} _ {\omega \omega} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \right] \cdot \mathbf {x} \| \\ = \frac {1}{n - m} | | [ \sum_ {z _ {i} \in S} \mathsf {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) - \sum_ {z _ {i} \in U} \mathsf {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) ] \cdot \mathbf {x} | | \\ \geq \frac {1}{n - m} \left(\| n \mathrm {D} _ {\omega \omega} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \| - \| \sum_ {z _ {i} \in U} \mathrm {D} _ {\omega \omega} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \|\right) \cdot \| \mathbf {x} \| \\ \geq \frac {\left(\mu_ {\omega \omega} n - \left(\ell + \ell^ {2} / \mu_ {\nu}\right) m\right)}{n - m} \| \mathbf {x} \| \geq \frac {\left(\mu n - \left(\ell + \ell^ {2} / \mu\right) m\right)}{n - m} \| \mathbf {x} \|, \tag {71} \\ \end{array}
$$
where the second inequality is by Lemma 7. Combining eq.(70), eq.(68), and the definition of the vector  $\| \mathbf{x}\|$ , we get that
$$
\left\| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\omega}} \right\| = \| \mathbf {x} \| \leq \frac {\left(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {5} + 8 L \ell^ {2} / \mu^ {2}\right) m ^ {2}}{n (\mu n - (\ell + \ell^ {2} / \mu) m)}. \tag {72}
$$
Symmetrically, we can get that  $\| \pmb{\nu}_{S^{\backslash}}^{*} - \widehat{\pmb{\nu}}\| \leq \frac{(8\sqrt{2}L^{2}\ell^{3}\rho / \mu^{5} + 8L\ell^{2} / \mu^{2})m^{2}}{n(\mu n - (\ell + \ell^{2} / \mu)m)}.$
Theorem 6  $((\epsilon, \delta)$ -Minimax Unlearning Certification). Under the same settings of Lemma 10, our minimax learning algorithm  $A_{sc - sc}$  and unlearning algorithm  $\bar{A}_{sc - sc}$  is  $(\epsilon, \delta)$ -certified minimax unlearning if we choose
$$
\sigma_ {1} a n d \sigma_ {2} = \frac {2 \left(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {5} + 8 L \ell^ {2} / \mu^ {2}\right) m ^ {2}}{n (\mu n - (\ell + \ell^ {2} / \mu) m) \epsilon} \sqrt {2 \log (2 . 5 / \delta)}. \tag {73}
$$
Proof. Our proof for  $(\epsilon, \delta)$ -minimax unlearning certification is similar to the one used for the differential privacy guarantee of the Gaussian mechanism [Dwork et al., 2014].
Let  $(\omega_{S}^{*},\nu_{S}^{*})$  be the output of the learning algorithm  $A_{sc - sc}$  trained on dataset  $S$  and  $(\omega^u,\nu^u)$  be the output of the unlearning algorithm  $\bar{A}_{sc - sc}$  running with delete requests  $U$ , the learned
model  $(\omega_{S}^{*},\nu_{S}^{*})$ , and the memory variables  $T(S)$ . Then we have  $(\omega_{S}^{*},\nu_{S}^{*}) = A_{sc - sc}(S)$  and  $(\omega^{u},\nu^{u}) = \bar{A}_{sc - sc}(U,A_{sc - sc}(S),T(S))$ . We also denote the intermediate variables before adding noise in algorithm  $\bar{A}_{sc - sc}$  as  $(\widehat{\omega},\widehat{\nu})$ , and we have  $\omega^u = \widehat{\omega} +\xi_1$  and  $\pmb {\nu}^u = \widehat{\pmb{\nu}} +\pmb {\xi}_2$
Similarly, let  $(\omega_{S^{\backslash}}^{*},\nu_{S^{\backslash}}^{*})$  be the output of the learning algorithm  $A_{sc - sc}$  trained on dataset  $S\setminus U$  and  $(\omega_{S^{\backslash}}^{u},\nu_{S^{\backslash}}^{u})$  be the output of the unlearning algorithm  $\bar{A}_{sc - sc}$  running with delete requests  $\emptyset$ , the learned model  $(\omega_{S^{\backslash}}^{*},\nu_{S^{\backslash}}^{*})$ , and the memory variables  $T(S\setminus U)$ . Then we have  $(\omega_{S^{\backslash}}^{*},\nu_{S^{\backslash}}^{*}) = A_{sc - sc}(S\setminus U)$  and  $(\omega_{S^{\backslash}}^{u},\nu_{S^{\backslash}}^{u}) = \bar{A}_{sc - sc}(\emptyset,A_{sc - sc}(S\setminus U),T(S))$ . We also denote the intermediate variables before adding noise in algorithm  $\bar{A}_{sc - sc}$  as  $(\widehat{\omega}_{S^{\backslash}},\widehat{\nu}_{S^{\backslash}})$ , and we have  $\omega_{S^{\backslash}}^{u} = \widehat{\omega}_{S^{\backslash}} + \pmb{\xi}_{1}$  and  $\pmb{\nu}_{S^{\backslash}}^{u} = \widehat{\pmb{\nu}}_{S^{\backslash}} + \pmb{\xi}_{2}$ . Note that  $\widehat{\omega}_{S^{\backslash}} = \omega_{S^{\backslash}}^{*}$  and  $\widehat{\nu}_{S^{\backslash}} = \pmb{\nu}_{S^{\backslash}}^{*}$ .
We sample the noise  $\pmb{\xi}_1\sim \mathcal{N}(0,\sigma_1\mathbf{I}_{d_1})$  and  $\pmb {\xi}_2\sim \mathcal{N}(0,\sigma_2\mathbf{I}_{d_2})$  with the scale:
$$
\left\{ \begin{array}{l} \sigma_ {1} = \| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\omega}} \| \cdot \frac {\sqrt {2 \log (2 . 5 / \delta)}}{\epsilon / 2} = \| \widehat {\boldsymbol {\omega}} _ {S ^ {\backslash}} - \widehat {\boldsymbol {\omega}} \| \cdot \frac {\sqrt {2 \log (2 . 5 / \delta)}}{\epsilon / 2}, \\ \sigma_ {2} = \| \boldsymbol {\nu} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\nu}} \| \cdot \frac {\sqrt {2 \log (2 . 5 / \delta)}}{\epsilon / 2} = \| \widehat {\boldsymbol {\nu}} _ {S ^ {\backslash}} - \widehat {\boldsymbol {\nu}} \| \cdot \frac {\sqrt {2 \log (2 . 5 / \delta)}}{\epsilon / 2}, \end{array} \right. \tag {74}
$$
where  $\| \pmb{\omega}_{S\backslash}^{*} - \widehat{\pmb{\omega}}\|$  and  $\| \pmb{\omega}_{S\backslash}^{*} - \widehat{\pmb{\omega}}\|$  are given in Lemma 10. Then, following the same proof as Dwork et al. [2014, Theorem A.1] together with the composition property of DP [Vadhan, 2017, Lemma 7.2.3], we get that, for any set  $O\subseteq \Theta$  where  $\Theta \coloneqq \mathcal{W}\times \mathcal{V}$ ,
$$
\Pr \left[ \left(\widehat {\boldsymbol {\omega}}, \widehat {\boldsymbol {\nu}}\right) \in O \right] \leq e ^ {\epsilon} \Pr \left[ \left(\widehat {\boldsymbol {\omega}} _ {S \backslash}, \widehat {\boldsymbol {\nu}} _ {S \backslash}\right) \in O \right] + \delta , \text {a n d} \Pr \left[ \left(\widehat {\boldsymbol {\omega}} _ {S \backslash}, \widehat {\boldsymbol {\nu}} _ {S \backslash}\right) \in O \right] \leq e ^ {\epsilon} \Pr \left[ \left(\widehat {\boldsymbol {\omega}}, \widehat {\boldsymbol {\nu}}\right) \in O \right] + \delta , \tag {75}
$$
which implies that the algorithm pair  $A_{sc - sc}$  and  $\bar{A}_{sc - sc}$  is  $(\epsilon, \delta)$ -certified minimax unlearning.
Theorem 7 (Population Primal-Dual Risk). Under the same settings of Lemma 10 and denote  $d = \max \{d_1, d_2\}$ , the population weak and strong PD risk for the certified minimax unlearning variables  $(\omega^u, \nu^u)$  returned by Algorithm 2 are
$$
\left\{ \begin{array}{l} \triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n}\right), \\ \triangle^ {s} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{\eta^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n} + \frac {L ^ {2} \ell}{\mu^ {2} n}\right). \end{array} \right. \tag {76}
$$
Proof. We begin with the population weak PD risk for the certified minimax unlearning variable  $(\omega^u, \nu^u)$ , which has the following conversions,
$$
\begin{array}{l} \triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \\ = \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}\right) \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ F \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) \right] \\ = \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}\right) - F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) + F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ F \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) - F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) + F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ \leq \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}\right) - F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) \right] + \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) \right] \\ - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ F \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) - F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ = \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}\right) - F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) \right] + \max  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ (- F) \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) - (- F) \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ + \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ \end{array}
$$