SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\stackrel {(i)} {\leq} \mathbb {E} \left[ L \| \boldsymbol {\omega} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \right] + \mathbb {E} \left[ L \| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \| \right] + \triangle^ {w} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right)
$$
$$
\stackrel {(i i)} {\leq} \mathbb {E} \left[ L \| \boldsymbol {\omega} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \right] + \mathbb {E} \left[ L \| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \| \right] + \frac {2 \sqrt {2} L ^ {2}}{\mu n}, \tag {77}
$$
where the inequality  $(i)$  holds because the population loss function  $F(\omega, \nu) \coloneqq \mathbb{E}[f(\omega, \nu; z)]$  is  $L$ -Lipschitz continuous. The inequality  $(ii)$  is by Lemma 8.
By recalling the unlearning update step in Algorithm 2, we have
$$
\boldsymbol {\omega} ^ {u} = \boldsymbol {\omega} _ {S} ^ {*} + \frac {1}{n - m} \left[ \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \boldsymbol {\xi} _ {1}, \tag {78}
$$
where the vector  $\pmb{\xi}_1\in \mathbb{R}^{d_1}$  is drawn independently from  $\mathcal{N}(0,\sigma_1^2\mathbf{I}_{d_1})$ . From the relation in eq.(78), we further get
$$
\begin{array}{l} \mathbb {E} \left[ \| \boldsymbol {\omega} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \right] = \mathbb {E} \left[ \left\| \frac {1}{n - m} \left[ \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \cdot \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \boldsymbol {\xi} _ {1} \right\| \right] \\ \stackrel {(i)} {\leq} \frac {1}{n - m} \mathbb {E} \left[ \left\| \left[ \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S ^ {\backslash}} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \cdot \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \right\| \right] + \mathbb {E} [ \| \boldsymbol {\xi} _ {1} \| ] \\ \stackrel {(i i)} {\leq} \frac {1}{n - m} \cdot \frac {n - m}{(\mu n - \ell (1 + \ell / \mu) m)} \mathbb {E} \left[ \left\| \sum_ {z _ {i} \in U} \nabla_ {\omega} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \right\| \right] + \sqrt {\mathbb {E} [ \| \boldsymbol {\xi} _ {1} \| ^ {2} ]} \\ \stackrel {(i i i)} {=} \frac {1}{(\mu n - \ell (1 + \ell / \mu) m)} \mathbb {E} \left[ \left\| \sum_ {z _ {i} \in U} \nabla_ {\omega} f \left(\omega_ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \right\| \right] + \sqrt {d _ {1}} \sigma_ {1} \\ \stackrel {(i v)} {\leq} \frac {m L}{(\mu n - \ell (1 + \ell / \mu) m)} + \sqrt {d _ {1}} \sigma_ {1}, \tag {79} \\ \end{array}
$$
where the inequality  $(i)$  is by the triangle inequality and the inequality  $(ii)$  follows from the relation in eq.(71), together with the Jensen's inequality to bound  $\mathbb{E}[\|\pmb{\xi}_1\|]$ . The equality  $(iii)$  holds because the vector  $\pmb{\xi}_1 \sim \mathcal{N}(0, \sigma_1^2\mathbf{I}_{d_1})$  and thus we have  $\mathbb{E}[\|\pmb{\xi}_1\|^2] = d_1\sigma_1^2$ . Furthermore, the inequality  $(iv)$  is due to the fact that  $f(\omega, \nu; z)$  is  $L$ -Lipschitz continuous.
Symmetrically, we have
$$
\mathbb {E} \left[ \| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \| \right] \leq \frac {m L}{(\mu n - \ell (1 + \ell / \mu) m)} + \sqrt {d _ {2}} \sigma_ {2}. \tag {80}
$$
Plugging eq.(79) and eq.(80) into eq.(77) with  $d = \max \{d_1,d_2\}$  we get
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \frac {2 m L ^ {2}}{(\mu n - \ell (1 + \ell / \mu) m)} + \sqrt {d} \left(\sigma_ {1} + \sigma_ {2}\right) L + \frac {2 \sqrt {2} L ^ {2}}{\mu n}. \tag {81}
$$
With the noise scale  $\sigma_{1}$  and  $\sigma_{2}$  being equal to  $\frac{2(8\sqrt{2}L^2\ell^3\rho/\mu^5+8L\ell^2/\mu^2)m^2}{n(\mu n-(\ell+\ell^2/\mu)m)\epsilon}\sqrt{2\log(2.5/\delta)}$ , we can get our generalization guarantee with population weak PD risk:
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n}\right). \tag {82}
$$
For the population strong PD risk  $\triangle^s (\omega^u,\nu^u)$ , similarly, we have
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} F \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} F \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) \right] \\ = \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \left(F \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}\right) - F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) + F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right)\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \left(F \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) - F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) + F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right)\right) \right] \\ = \mathbb {E} [ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} (F (\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}) - F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}) + F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu})) ] - \mathbb {E} [ \min  _ {\boldsymbol {\omega} \in \mathcal {W}} (F (\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}) - F (\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}) + F (\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}))) ] \\ \leq \mathbb {E} [ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} (F (\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}) - F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu})) + \max  _ {\boldsymbol {\nu} \in \mathcal {V}} F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}) ] \\ - \mathbb {E} \left[ \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \left(F \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) - F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right)\right) + \min  _ {\boldsymbol {\omega} \in \mathcal {W}} F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ = \mathbb {E} [ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} (F (\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}) - F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu})) ] + \mathbb {E} [ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}) ] \\ - \mathbb {E} \left[ \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \left(F \left(\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}\right) - F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right)\right) \right] - \mathbb {E} \left[ \min  _ {\boldsymbol {\omega} \in \mathcal {W}} F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ = \mathbb {E} [ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} (F (\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu}) - F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu})) ] + \mathbb {E} [ \max  _ {\boldsymbol {\omega} \in \mathcal {W}} ((- F) (\boldsymbol {\omega}, \boldsymbol {\nu} ^ {u}) - (- F) (\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}))) ] \\ + \mathbb {E} [ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} F (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} F (\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}) ] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} \mathbb {E} \left[ L \| \boldsymbol {\omega} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \right] + \mathbb {E} \left[ L \| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \| \right] + \triangle^ {s} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \\ \stackrel {(i i)} {\leq} \frac {2 m L ^ {2}}{(\mu n - \ell (1 + \ell / \mu) m)} + \sqrt {d} \left(\sigma_ {1} + \sigma_ {2}\right) L + \frac {8 L ^ {2} \ell}{\mu^ {2} n}, \tag {83} \\ \end{array}
$$
where inequality  $(i)$  is due to the fact that the population loss function  $F(\omega, \nu) \coloneqq \mathbb{E}[f(\omega, \nu; z)]$  is  $L$ -Lipschitz continuous. The inequality  $(ii)$  uses eq.(79), eq.(80) and Lemma 9. With the same noise scale above, we can get the generalization guarantee in terms of strong PD risk below,
$$
\triangle^ {s} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n} + \frac {L ^ {2} \ell}{\mu^ {2} n}\right). \tag {84}
$$
Theorem 8 (Deletion Capacity). Under the same settings of Lemma 10 and denote  $d = \max \{d_1, d_2\}$ , the deletion capacity of Algorithm 2 is
$$
m _ {\epsilon , \delta , \gamma} ^ {A, \bar {A}} \left(d _ {1}, d _ {2}, n\right) \geq c \cdot \frac {n \sqrt {\epsilon}}{\left(d \log (1 / \delta)\right) ^ {1 / 4}}, \tag {85}
$$
Proof. By the definition of deletion capacity, in order to ensure the population PD risk derived in Theorem 7 is bounded by  $\gamma$ , it suffices to let:
$$
m _ {\epsilon , \delta , \gamma} ^ {A, \bar {A}} (d _ {1}, d _ {2}, n) \geq c \cdot \frac {n \sqrt {\epsilon}}{(d \log (1 / \delta)) ^ {1 / 4}},
$$
where the constant  $c$  depends on the properties of the loss function  $f(\omega, \nu; z)$ .
B.2 Missing Proofs of Sec.4.4
B.2.1 Proof of Lemma 1 (Closeness Upper Bound)
Proof. By the definition of the functions  $F_{S}(\omega, \nu)$  and  $F_{S^{\backslash}}(\omega, \nu)$ , we have
$$
\begin{array}{l} \| \nabla_ {\omega} F _ {S ^ {\backslash}} \left(\omega_ {S ^ {\backslash}} ^ {*}, \mathrm {V} _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}\right)\right) - \nabla_ {\omega} F _ {S ^ {\backslash}} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) - \frac {n}{n - m} \mathrm {D} _ {\omega \omega} F _ {S} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \left(\omega_ {S ^ {\backslash}} ^ {*} - \omega_ {S} ^ {*}\right)) \| \\ = \| \frac {n}{n - m} \left[ \nabla_ {\omega} F _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}, \mathbb {V} _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}\right)\right) - \nabla_ {\omega} F _ {S} \left(\omega_ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) - \mathbb {D} _ {\omega \omega} F _ {S} \left(\omega_ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \left(\omega_ {S ^ {\backslash}} ^ {*} - \omega_ {S} ^ {*}\right) \right] \\ - \frac {1}{n - m} \sum_ {z _ {i} \in U} \left[ \nabla_ {\omega} f \left(\omega_ {S ^ {\backslash}} ^ {*}, \mathrm {V} _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}\right); z _ {i}\right) - \nabla_ {\omega} f \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}; z _ {i}\right) \right] \| \\ \stackrel {(i)} {\leq} \frac {n}{n - m} \| \nabla_ {\omega} F _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}, \mathrm {V} _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}\right)\right) - \nabla_ {\omega} F _ {S} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) - \mathrm {D} _ {\omega \omega} F _ {S} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \left(\omega_ {S ^ {\backslash}} ^ {*} - \omega_ {S} ^ {*}\right) \| \tag {86} \\ + \frac {1}{n - m} \sum_ {z _ {i} \in U} \| \nabla_ {\omega} f \left(\omega_ {S ^ {\backslash}} ^ {*}, \mathrm {V} _ {S} \left(\omega_ {S ^ {\backslash}} ^ {*}\right); z _ {i}\right) - \nabla_ {\omega} f \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}; z _ {i}\right) \| \\ \stackrel {(i i)} {\leq} \frac {8 \sqrt {2} \rho L ^ {2} \ell^ {3} m ^ {2}}{\mu^ {5} n (n - m)} + \frac {2 \sqrt {2} L l ^ {2} m ^ {2}}{\mu^ {2} n (n - m)}, \\ \end{array}
$$
where the inequality  $(i)$  holds because the triangle inequality and the inequality  $(ii)$  uses the results in eq.(65) and eq.(66). Now let  $\widetilde{\mathbf{x}}$  be the vector satisfying the following relation,
$$
\boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} = \boldsymbol {\omega} _ {S} ^ {*} + \frac {1}{n} \left[ \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \widetilde {\mathbf {x}}. \tag {87}
$$
Since we have  $\nabla_{\omega}F_{S\setminus}(\pmb{\omega}_{S}^{*},\pmb{\nu}_{S}^{*}) = -\frac{1}{n - m}\sum_{z_{i}\in U}\nabla_{\omega}f(\pmb{\omega}_{S}^{*},\pmb{\nu}_{S}^{*};z_{i})$  and  $\nabla_{\omega}F_{S\setminus}(\pmb{\omega}_{S\setminus}^{*},\mathsf{V}_{S}(\pmb{\omega}_{S\setminus}^{*})) = 0$  due to the optimality of  $\pmb{\omega}_{S\setminus}^{*}$ , plugging the above relation into eq.(86), we get
$$
\left\| \frac {n}{n - m} \mathsf {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \widetilde {\mathbf {x}} \right\| \leq \left(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {5} + 2 \sqrt {2} L \ell^ {2} / \mu^ {2}\right) \frac {m ^ {2}}{n (n - m)}. \tag {88}
$$
With  $\mathrm{D}_{\omega \omega}F_{S}(\omega_{S}^{*},\nu_{S}^{*})\geq \mu_{\omega \omega}$  , we also have
$$
\| \frac {n}{n - m} \mathrm {D} _ {\omega \omega} F _ {S} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}\right) \widetilde {\mathbf {x}} \| \geq \frac {\mu_ {\omega \omega} n}{n - m} \| \widetilde {\mathbf {x}} \| \geq \frac {\mu n}{n - m} \| \widetilde {\mathbf {x}} \|. \tag {89}
$$
Combining eq.(89), eq.(88), and the definition of the vector  $\| \widetilde{\mathbf{x}}\|$ , we get that
$$
\left\| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \widetilde {\boldsymbol {\omega}} \right\| = \| \widetilde {\mathbf {x}} \| \leq \frac {(8 \sqrt {2} L ^ {2} \ell^ {3} \rho / \mu^ {6} + 2 \sqrt {2} L \ell^ {2} / \mu^ {3}) m ^ {2}}{n ^ {2}}. \tag {90}
$$
Symmetrically, we can get  $\| \pmb{\nu}_{S^{\backslash}}^{*} - \widetilde{\pmb{\nu}}\| \leq \frac{(8\sqrt{2}L^{2}\ell^{3}\rho / \mu^{6} + 2\sqrt{2}L\ell^{2} / \mu^{3})m^{2}}{n^{2}}$
B.2.2 Proof of Theorem 2  $(\epsilon, \delta)$ -Minimax Unlearning Certification
Proof. With the closeness upper bound in Lemma 1 and the given noise scales in eq.(22), the proof is identical to that of Theorem 6.  $\square$
B.2.3 Proof of Theorem 3 (Population Primal-Dual Risk)
Proof. We start with the population weak PD risk. By eq.(77), we have
$$
\triangle^ {w} \left(\widetilde {\boldsymbol {\omega}} ^ {u}, \widetilde {\boldsymbol {\nu}} ^ {u}\right) \leq \mathbb {E} \left[ L \| \widetilde {\boldsymbol {\omega}} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \right] + \mathbb {E} \left[ L \| \widetilde {\boldsymbol {\nu}} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \| \right] + \frac {2 \sqrt {2} L ^ {2}}{\mu n}. \tag {91}
$$
By recalling the unlearning step in Algorithm 3, we have
$$
\widetilde {\boldsymbol {\omega}} ^ {u} = \boldsymbol {\omega} _ {S} ^ {*} + \frac {1}{n} \left[ \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \boldsymbol {\xi} _ {1}, \tag {92}
$$
$$
\begin{array}{l} \mathbb {E} \big [ \| \widetilde {\boldsymbol {\omega}} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \big ] = \mathbb {E} \left[ \left\| \frac {1}{n} \left[ \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \cdot \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \boldsymbol {\xi} _ {1} \right\| \right] \\ \leq \frac {i}{n} \mathbb {E} \left[ \left\| \left[ \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \cdot \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \right\| \right] + \mathbb {E} [ \| \boldsymbol {\xi} _ {1} \| ] \tag {93} \\ \stackrel {(i i)} {\leq} \frac {1}{n} \cdot \mu^ {- 1} \mathbb {E} \left[ \left\| \sum_ {z _ {i} \in U} \nabla_ {\omega} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) \right\| \right] + \sqrt {\mathbb {E} [ \| \boldsymbol {\xi} _ {1} \| ^ {2} ]} \\ \stackrel {(i i i)} {=} \frac {1}{\mu n} \mathbb {E} \left[ \left\| \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) \right\| \right] + \sqrt {d _ {1}} \sigma_ {1} \stackrel {(i v)} {\leq} \frac {m L}{\mu n} + \sqrt {d _ {1}} \sigma_ {1}, \\ \end{array}
$$
where the inequality  $(i)$  uses the triangle inequality and the inequality  $(ii)$  follows by the relation  $\mathsf{D}_{\omega \omega}F_S(\pmb{\omega}_S^*,\pmb{\nu}_S^*)\geq \mu_{\omega \omega}\geq \mu$ , together with the Jensen's inequality to bound  $\mathbb{E}[\|\pmb{\xi}_1\|]$ . The equality  $(iii)$  holds because the vector  $\pmb{\xi}_1\sim \mathcal{N}(0,\sigma_1^2\mathbf{I}_{d_1})$  and thus we have  $\mathbb{E}[\|\pmb{\xi}_1\|^2] = d_1\sigma_1^2$ . And the inequality  $(iv)$  is due to the fact that  $f(\pmb {\omega},\pmb {\nu};z)$  is  $L$ -Lipshtz continuous. Symmetrically, we have
$$
\mathbb {E} \left[ \left\| \tilde {\boldsymbol {\nu}} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \right\| \right] \leq \frac {m L}{\mu n} + \sqrt {d _ {2}} \sigma_ {2}. \tag {94}
$$
Plugging eq.(93) and eq.(94) into eq.(91) with  $d = \max \{d_1,d_2\}$  we get
$$
\triangle^ {w} \left(\widetilde {\omega} ^ {u}, \widetilde {\nu} ^ {u}\right) \leq \frac {2 m L ^ {2}}{\mu n} + \sqrt {d} \left(\sigma_ {1} + \sigma_ {2}\right) L + \frac {2 \sqrt {2} L ^ {2}}{\mu n}. \tag {95}
$$
With the noise scale  $\sigma_{1}$  and  $\sigma_{2}$  being equal to  $\frac{2(8\sqrt{2}L^{2}\ell^{3}\rho / \mu^{6} + 2\sqrt{2}L\ell^{2} / \mu^{3})m^{2}}{n^{2}\epsilon}\sqrt{2\log(2.5 / \delta)}$ , we can get our generalization guarantee in terms of population weak PD risk:
$$
\triangle^ {w} \left(\widetilde {\boldsymbol {\omega}} ^ {u}, \widetilde {\boldsymbol {\nu}} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n}\right). \tag {96}
$$
For the population strong PD risk, using an application of eq.(83) with Lemma 9, eq.(93), eq.(94) and the noise scales given in Theorem 2, we can get
$$
\triangle^ {s} \left(\widetilde {\boldsymbol {\omega}} ^ {u}, \widetilde {\boldsymbol {\nu}} ^ {u}\right) = \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \mu^ {6} + L ^ {2} \ell^ {2} / \mu^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\mu n} + \frac {L ^ {2} \ell}{\mu^ {2} n}\right). \tag {97}
$$
B.2.4 Proof of Theorem 4 (Deletion Capacity)
Proof. By the definition of deletion capacity, in order to ensure the population weak or strong PD risk derived in Lemma 3 is bounded by  $\gamma$ , it suffices to let  $m_{\epsilon, \delta, \gamma}^{A, \bar{A}}(d_1, d_2, n) \geq c \cdot \frac{n\sqrt{\epsilon}}{(d\log(1/\delta))^{1/4}}$ .
B.3 Efficient Online Unlearning Algorithm
To support successive unlearning requests, similar to the STL case Suriyakumar and Wilson [2022], we further provide an efficient and online minimax unlearning algorithm (denoted by  $\bar{A}_{\mathrm{online}}$ ). The pseudocode of  $\bar{A}_{\mathrm{online}}$  is given in Algorithm 4. Its certified minimax unlearning guarantee, generalization, and deletion capacity can be identically yielded as Algorithm 3, which are omitted here.
Algorithm 4 Efficient Online Certified Minimax Unlearning  $(\bar{A}_{\mathrm{online}})$
Input: Delete request  $z_{k} \in S$ , early delete requests  $U: \{z_{j}\}_{j=1}^{m} \subseteq S$ , output of  $A_{sc-sc}(S): (\omega_{S}^{*}, \nu_{S}^{*})$ , memory variables  $T(S): \{\mathsf{D}_{\omega \omega} F_{S}(\omega_{S}^{*}, \nu_{S}^{*}), \mathsf{D}_{\nu \nu} F_{S}(\omega_{S}^{*}, \nu_{S}^{*})\}$ , early unlearning variables:  $(\widetilde{\omega}_{U}^{u}, \widetilde{\nu}_{U}^{u})$ , loss function:  $f(\omega, \nu; z)$ , noise parameters:  $\sigma_{1}, \sigma_{2}$ .
1: Set  $\widetilde{\omega}_{\emptyset} = \omega_{S}^{*}$  and  $\widetilde{\nu}_{\emptyset} = \nu_{S}^{*}$ .
2: Compute
$$
\overline {{\boldsymbol {\omega}}} _ {U \cup \{k \}} = \widetilde {\boldsymbol {\omega}} _ {U} + \frac {1}{n} \left[ \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {k}\right), \tag {98}
$$
$$
\overline {{\boldsymbol {\nu}}} _ {U \cup \{k \}} = \widetilde {\boldsymbol {\nu}} _ {U} + \frac {1}{n} \left[ \mathbb {D} _ {\boldsymbol {\nu} \boldsymbol {\nu}} F _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \nabla_ {\boldsymbol {\nu}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {k}\right). \tag {99}
$$
3:  $\widetilde{\omega}_{U\cup \{k\}}^{u} = \overline{\omega}_{U\cup \{k\}} + \pmb{\xi}_{1}$ , where  $\pmb{\xi}_{1}\sim \mathcal{N}(0,\sigma_{1}\mathbf{I}_{d_{1}})$  and  $\widetilde{\pmb{\nu}}_{U\cup \{k\}}^{u} = \overline{\pmb{\nu}}_{U\cup \{k\}} + \pmb{\xi}_{2}$ , where  $\pmb{\xi}_{2}\sim \mathcal{N}(0,\sigma_{2}\mathbf{I}_{d_{2}})$
Output:  $(\widetilde{\omega}_{U\cup \{k\}}^u,\widetilde{\nu}_{U\cup \{k\}}^u)$
C Detailed Algorithm Descriptions and Missing Proofs in Section 5
C.1 Minimax Unlearning Algorithm for Smooth Convex-Concave Loss Function
In this section, we provide minimax learning and minimax unlearning algorithms for smooth convex-concave loss functions based on the counterpart algorithms for the SC-SC setting. Given the convex-concave loss function  $f(\omega, \nu; z)$ , we define the regularized loss function as  $\widetilde{f}(\omega, \nu; z) = f(\omega, \nu; z) + \frac{\lambda}{2} \| \omega \|^2 - \frac{\lambda}{2} \| \nu \|^2$ . Suppose the function  $f$  satisfies Assumption 1, then the function  $\widetilde{f}$  is  $\lambda$ -strongly convex in  $\omega$ ,  $\lambda$ -strongly concave in  $\nu$ ,  $(2L + \lambda \| \omega \| + \lambda \| \nu \|)$ -Lipschitz,  $\sqrt{2}(2\ell + \lambda)$ -gradient Lipschitz and  $\rho$ -Hessian Lipschitz. Thus, we can apply the minimax learning in Algorithm 1 and unlearning in Algorithm 2 to the regularized loss function with a properly chosen  $\lambda$ . We denote our learning algorithm by  $A_{c-c}$  and unlearning algorithm by  $A_{c-c}$ . The pseudocode is provided in Algorithm 5 and Algorithm 6, respectively. Additionally, we denote the regularized population loss as  $\widetilde{F}(\omega, \nu) := \mathbb{E}_{z \sim \mathcal{D}}[\widetilde{f}(\omega, \nu; z)]$  and regularized empirical loss as  $\widetilde{F}_S(\omega, \nu) := \frac{1}{n} \sum_{i=1}^{n} \widetilde{f}(\omega, \nu; z_i)$ .
Algorithm 5 Mimimax Learning Algorithm  $(A_{c - c})$
Input: Dataset  $S:\{z_i\}_{i = 1}^n\sim \mathcal{D}^n$  , loss function:  $f(\omega ,\nu ;z)$  , regularization parameter:  $\lambda$
1: Define
$$
\widetilde {f} (\boldsymbol {\omega}, \boldsymbol {\nu}; z) = f (\boldsymbol {\omega}, \boldsymbol {\nu}; z) + \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2} - \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2}. \tag {100}
$$
2: Run the algorithm  $A_{sc - sc}$  on the dataset  $S$  with loss function  $\widetilde{f}$ .
Output:  $(\omega_{S}^{*},\nu_{S}^{*},\mathsf{D}_{\omega \omega}\widetilde{F}_{S}(\omega_{S}^{*},\nu_{S}^{*}),\mathsf{D}_{\nu \nu}\widetilde{F}_{S}(\omega_{S}^{*},\nu_{S}^{*}))\gets A_{sc - sc}(S,\widetilde{f})$