SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 6 Certified Minimax Unlearning for Convex-Concave Loss  $(\bar{A}_{c - c})$
Input: Delete requests  $U: \{z_{j}\}_{j=1}^{m} \subseteq S$ , output of  $A_{c-c}(S): (\omega_{S}^{*}, \nu_{S}^{*})$ , memory variables  $T(S): \{\mathsf{D}_{\omega \omega} \widetilde{F}_{S}(\omega_{S}^{*}, \nu_{S}^{*}), \mathsf{D}_{\nu \nu} \widetilde{F}_{S}(\omega_{S}^{*}, \nu_{S}^{*})\}$ , loss function:  $f(\omega, \nu; z)$ , regularization parameter:  $\lambda$ , noise parameters:  $\sigma_{1}, \sigma_{2}$ .
$$
\widetilde {f} (\boldsymbol {\omega}, \boldsymbol {\nu}; z) = f (\boldsymbol {\omega}, \boldsymbol {\nu}; z) + \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2} - \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2}. \tag {101}
$$
2: Run the algorithm  $\bar{A}_{sc - sc}$  with delete requests  $U$ , learning variables  $(\boldsymbol{\omega}_S^*, \boldsymbol{\nu}_S^*)$ , memory variables  $T(S)$ , loss function  $\widetilde{f}$  and noise parameters  $\sigma_1$  and  $\sigma_2$ .
Output:  $(\pmb{\omega}^u, \pmb{\nu}^u) \gets \bar{A}_{sc - sc}(U, (\pmb{\omega}_S^*, \pmb{\nu}_S^*), T(S), \tilde{f}, \sigma_1, \sigma_2)$ .
C.2 Supporting Lemma
Lemma 11. Suppose the function  $f(\omega, \nu; z)$  is  $L$ -Lipschitz continuous. Define the function  $\widetilde{f}(\omega, \nu; z)$  as
$$
\widetilde {f} (\omega , \nu ; z) = f (\omega , \nu ; z) + \frac {\lambda}{2} \| \omega \| ^ {2} - \frac {\lambda}{2} \| \nu \| ^ {2}. \tag {102}
$$
Given a dataset  $S = \{z_i\}_{i=1}^n$  and denote  $(\omega_S^*, \nu_S^*) := \arg \min_{\boldsymbol{\omega} \in \mathcal{W}} \max_{\boldsymbol{\nu} \in \mathcal{V}} \{\widetilde{F}_S(\boldsymbol{\omega}, \boldsymbol{\nu}) := \frac{1}{n} \sum_{i=1}^n \widetilde{f}(\boldsymbol{\omega}, \boldsymbol{\nu}; z_i)\}$ . Then, the variables  $(\omega_S^*, \nu_S^*)$  satisfy  $\| \omega_S^* \| \leq L / \lambda$  and  $\| \nu_S^* \| \leq L / \lambda$ .
Proof. Due to the optimality of  $\omega_{S}^{*}$ , we have
$$
\nabla_ {\omega} \widetilde {F} _ {S} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z\right) = \frac {1}{n} \sum_ {z _ {i} \in S} \nabla_ {\omega} \widetilde {f} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) = 0. \tag {103}
$$
Plugging in the definition of the function  $\widetilde{f}$  in the above, we get that
$$
\frac {1}{n} \sum_ {z _ {i} \in S} \nabla_ {\boldsymbol {\omega}} f \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \lambda \boldsymbol {\omega} _ {S} ^ {*} = 0. \tag {104}
$$
Then, using the triangle inequality, we have
$$
\left\| \lambda \omega_ {S} ^ {*} \right\| = \left\| \frac {1}{n} \sum_ {z _ {i} \in S} \nabla_ {\omega} f \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}; z _ {i}\right) \right\| \leq \frac {1}{n} \sum_ {z _ {i} \in S} \left\| \nabla_ {\omega} f \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}; z _ {i}\right) \right\| \leq L, \tag {105}
$$
where the last inequality holds because the function  $f$  is  $L$ -Lipschitz continuous. Thus we have  $\| \boldsymbol{\omega}_S^* \| \leq L / \lambda$ . Similarly, we can get  $\| \pmb{\nu}_S^* \| \leq L / \lambda$ .
Lemma 11 implies that the empirical optimizer  $(\omega_{S}^{*},\nu_{S}^{*})$  returned by Algorithm 6 satisfies  $\| \boldsymbol{\omega}_{S}^{*}\| \leq L / \lambda$  and  $\| \pmb {\nu}_S^*\| \leq L / \lambda$ . Thus our domain of interest are  $\mathcal{W}\coloneqq \{\pmb {\omega}||\pmb {\omega}\| \leq L / \lambda \}$  and  $\mathcal{V}\coloneqq \{\pmb {\nu}||\pmb {\nu}\| \leq$ $L / \lambda \}$ . Over the set  $\mathcal{W}\times \mathcal{V}$ , the function  $\widetilde{f} (\omega ,\nu ;z)$  is  $4L$ -Lipschitz continuous. Also, with  $\lambda < \ell$ ,  $\widetilde{f} (\omega ,\nu ;z)$  has  $3\sqrt{2}\ell$ -Lipschitz gradients.
C.3 Proof of Theorem 5 (Certified Minimax Unlearning for Convex-Concave Loss Function)
Denote  $\widetilde{L} = 4L$  and  $\widetilde{\ell} = 3\sqrt{2}\ell$ , then the function  $\widetilde{f}$  is  $\widetilde{L}$ -Lipschitz continuous and has  $\widetilde{\ell}$ -Lipschitz gradients. Let  $(\boldsymbol{\omega}_{S^{\backslash}}^{*},\boldsymbol{\nu}_{S^{\backslash}}^{*})$  be the optimal solution of the loss function  $\widetilde{F}_{S^{\backslash}}(\boldsymbol{\omega},\boldsymbol{\nu})$  on the remaining dataset, i.e.,
$$
\left(\boldsymbol {\omega} _ {S \backslash} ^ {*}, \boldsymbol {\nu} _ {S \backslash} ^ {*}\right) := \arg \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \{\widetilde {F} _ {S \backslash} (\boldsymbol {\omega}, \boldsymbol {\nu}) := \frac {1}{n - m} \sum_ {z _ {i} \in S \backslash U} \widetilde {f} (\boldsymbol {\omega}, \boldsymbol {\nu}; z _ {i}) \}. \tag {106}
$$
Additionally, we have  $\| \mathbb{D}_{\omega \omega}\widetilde{F}_S(\pmb{\omega}_S^*,\pmb{\nu}_S^*)\| \geq \lambda$  and  $\| \mathbb{D}_{\pmb{\nu}\pmb{\nu}}\widetilde{F}_S(\pmb{\omega}_S^*,\pmb{\nu}_S^*)\| \geq \lambda$
Lemma 12. Under the settings of Theorem 5, for any  $\lambda >0$ , the population weak and strong PD risk for the minimax learning variables  $(\omega_{S}^{*},\nu_{S}^{*})$  returned by Algorithm 5 are
$$
\left\{ \begin{array}{l} \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\omega_ {S} ^ {*}, \boldsymbol {\nu}\right) \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \leq \frac {3 2 \sqrt {2} L ^ {2}}{\lambda n} + \frac {\lambda}{2} \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right), \\ \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} F \left(\omega_ {S} ^ {*}, \boldsymbol {\nu}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \leq \frac {3 8 4 \sqrt {2} L ^ {2} \ell}{\lambda^ {2} n} + \frac {\lambda}{2} \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right). \end{array} \right. \tag {107}
$$
Proof. For the function  $\widetilde{F} (\omega ,\nu)$ , an application of Lemma 8 gives that
$$
\max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} [ \widetilde {F} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}) ] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} [ \widetilde {F} (\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}) ] \leq \frac {3 2 \sqrt {2} L ^ {2}}{\lambda n}. \tag {108}
$$
By the assumption of bounded parameter spaces  $\mathcal{W}$  and  $\nu$  so that  $\max_{\omega \in \mathcal{W}}\| \boldsymbol {\omega}\| \leq B_{\omega}$  and  $\max_{\nu \in \nu}\| \pmb {\nu}\| \leq B_{\pmb{\nu}}$ , we have the following derivations for the population weak PD risk,
$$
\begin{array}{l} \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ = \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ \widetilde {F} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) - \frac {\lambda}{2} \| \boldsymbol {\omega} _ {S} ^ {*} \| ^ {2} + \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2} \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ \widetilde {F} \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) - \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2} + \frac {\lambda}{2} \| \boldsymbol {\nu} _ {S} ^ {*} \| ^ {2} \right] \\ \leq \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} \left[ \widetilde {F} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) + \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2} \right] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} \left[ \widetilde {F} \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) - \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2} \right] \tag {109} \\ \leq \max _ {\pmb {\nu} \in \mathcal {V}} \mathbb {E} \big [ \widetilde {F} (\pmb {\omega} _ {S} ^ {*}, \pmb {\nu}) \big ] - \min _ {\pmb {\omega} \in \mathcal {W}} \mathbb {E} \big [ \widetilde {F} (\pmb {\omega}, \pmb {\nu} _ {S} ^ {*}) \big ] + \max _ {\pmb {\nu} \in \mathcal {V}} \mathbb {E} \left[ \frac {\lambda}{2} \| \pmb {\nu} \| ^ {2} \right] + \max _ {\pmb {\omega} \in \mathcal {W}} \mathbb {E} \left[ \frac {\lambda}{2} \| \pmb {\omega} \| ^ {2} \right] \\ \leq \frac {3 2 \sqrt {2} L ^ {2}}{\lambda n} + \frac {\lambda}{2} \left(B _ {\omega} ^ {2} + B _ {\nu} ^ {2}\right). \\ \end{array}
$$
Similarly, an application of Lemma 9 gives that
$$
\mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \widetilde {F} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \widetilde {F} \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \leq \frac {1 2 8 \sqrt {2} L ^ {2} (2 \ell + \lambda)}{\lambda^ {2} n}. \tag {110}
$$
And we can get the population strong PD risk with the following conversions,
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} F \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} F \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \\ = \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \left(\widetilde {F} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) - \frac {\lambda}{2} \| \boldsymbol {\omega} _ {S} ^ {*} \| ^ {2} + \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \left(\widetilde {F} \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) - \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2} + \frac {\lambda}{2} \| \boldsymbol {\nu} _ {S} ^ {*} \| ^ {2}\right) \right] \\ \leq \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \left(\widetilde {F} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}) + \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \left(\widetilde {F} (\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}) - \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2}\right) \right] \\ = \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \left(\widetilde {F} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) + \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2}\right) \right] - \mathbb {E} \left[ \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \left(\widetilde {F} \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) - \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2}\right) \right] \\ \leq \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \widetilde {F} \left(\omega_ {S} ^ {*}, \boldsymbol {\nu}\right) + \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2} \right] + \mathbb {E} \left[ \max  _ {\boldsymbol {\omega} \in \mathcal {W}} (- \widetilde {F}) \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) + \max  _ {\boldsymbol {\omega} \in \mathcal {W}} \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2} \right] \\ = \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \widetilde {F} \left(\omega_ {S} ^ {*}, \boldsymbol {\nu}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \widetilde {F} \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] + \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \frac {\lambda}{2} \| \boldsymbol {\nu} \| ^ {2} \right] + \mathbb {E} \left[ \max  _ {\boldsymbol {\omega} \in \mathcal {W}} \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2} \right] \\ \leq \frac {1 2 8 \sqrt {2} L ^ {2} (2 \ell + \lambda)}{\lambda^ {2} n} + \frac {\lambda}{2} \left(B _ {\omega} ^ {2} + B _ {\nu} ^ {2}\right) \leq \frac {3 8 4 \sqrt {2} L ^ {2} \ell}{\lambda^ {2} n} + \frac {\lambda}{2} \left(B _ {\omega} ^ {2} + B _ {\nu} ^ {2}\right). \tag {111} \\ \end{array}
$$
Lemma 13 (Closeness Upper Bound). Under the settings of Theorem 5, we have the closeness bound between the intermediate variables  $(\widehat{\omega}, \widehat{\nu})$  in Algorithm 6 and  $(\omega_{S^{\backslash}}^{*}, \nu_{S^{\backslash}}^{*})$  in eq.(106):
$$
\left\{\| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\omega}} \|, \| \boldsymbol {\nu} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\nu}} \| \right\} \leq \frac {(8 \sqrt {2} \widetilde {L} ^ {2} \widetilde {\ell} ^ {3} \rho / \lambda^ {5} + 8 \widetilde {L} \widetilde {\ell} ^ {2} / \lambda^ {2}) m ^ {2}}{n (\lambda n - (\widetilde {\ell} + \widetilde {\ell} ^ {2} / \lambda) m)}. \tag {112}
$$
Proof. Since we now run the algorithms  $A_{sc - sc}$  and  $\bar{A}_{sc - sc}$  with the regularized loss function  $\tilde{f}$ , the proof is identical to that of Lemma 10.
Equipped with the supporting lemmas above, the proof of Theorem 5 can be separated into the proofs of the following three lemmas.
Lemma 14 (Minimax Unlearning Certification). Under the settings of Theorem 5, our minimax learning algorithm  $A_{c - c}$  and unlearning algorithm  $\bar{A}_{c - c}$  is  $(\epsilon, \delta)$ -certified minimax unlearning if we choose
$$
\sigma_ {1} \text {a n d} \sigma_ {2} = \frac {2 \left(8 \sqrt {2} \widetilde {L} ^ {2} \widetilde {\ell} ^ {3} \rho / \lambda^ {5} + 8 \widetilde {L} \widetilde {\ell} ^ {2} / \lambda^ {2}\right) m ^ {2}}{n (\lambda n - (\widetilde {\ell} + \widetilde {\ell} ^ {2} / \lambda) m) \epsilon} \sqrt {2 \log (2 . 5 / \delta)}. \tag {113}
$$
Proof. With the closeness upper bound in Lemma 13 and the given noise scales in eq.(113), the proof is identical to that of Theorem 6.  $\square$
Lemma 15 (Population Primal-Dual Risk). Under the settings of Theorem 5, the population weak and strong PD risk for  $(\omega^u,\nu^u)$  returned by Algorithm 6 are
$$
\left\{ \begin{array}{l} \triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \lambda^ {6} + L ^ {2} \ell^ {2} / \lambda^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \lambda \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right)\right), \\ \triangle^ {s} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \lambda^ {6} + L ^ {2} \ell^ {2} / \lambda^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \bar {\delta})}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \frac {L ^ {2} \ell}{\lambda^ {2} n} + \lambda \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right)\right). \end{array} \right. \tag {114}
$$
Proof. For the population weak PD risk, an application of eq.(77) together with Lemma 12 gives that
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathbb {E} \left[ L \| \boldsymbol {\omega} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \right] + \mathbb {E} \left[ L \| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \| \right] + \frac {3 2 \sqrt {2} L ^ {2}}{\lambda n} + \frac {\lambda}{2} \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right). \tag {115}
$$
According to Algorithm 6, we have the unlearning update step
$$
\boldsymbol {\omega} ^ {u} = \boldsymbol {\omega} _ {S} ^ {*} + \frac {1}{n - m} \left[ \mathbb {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} \widetilde {F} _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} \widetilde {f} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \boldsymbol {\xi} _ {1}, \tag {116}
$$
where  $\widetilde{F}_{S\setminus}(\pmb{\omega}_S^*,\pmb{\nu}_S^*) := \frac{1}{n - m}\sum_{z_i\in S\setminus U}\widetilde{f} (\pmb {\omega},\pmb {\nu};z_i)$ . From the relation above, we further get
$$
\begin{array}{l} \mathbb {E} [ \| \omega^ {u} - \omega_ {S} ^ {*} \| ] \\ = \mathbb {E} \left[ \left\| \frac {1}{n - m} \left[ \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} \widetilde {F} _ {S \backslash} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \cdot \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} \widetilde {f} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) + \boldsymbol {\xi} _ {1} \right\| \right] \\ \stackrel {(i)} {\leq} \frac {1}{n - m} \mathbb {E} \left[ \left\| \left[ \mathrm {D} _ {\boldsymbol {\omega} \boldsymbol {\omega}} \widetilde {F} _ {S ^ {\backslash}} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] ^ {- 1} \cdot \sum_ {z _ {i} \in U} \nabla_ {\boldsymbol {\omega}} \widetilde {f} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \right\| \right] + \mathbb {E} [ \| \boldsymbol {\xi} _ {1} \| ] \\ \stackrel {(i i)} {\leq} \frac {1}{n - m} \cdot \frac {n - m}{(\lambda n - \widetilde {\ell} (1 + \widetilde {\ell} / \lambda) m) ^ {2}} \mathbb {E} \left[ \left\| \sum_ {z _ {i} \in U} \nabla_ {\omega} \widetilde {f} \left(\omega_ {S} ^ {*}, \nu_ {S} ^ {*}; z _ {i}\right) \right\| \right] + \sqrt {\mathbb {E} [ \| \boldsymbol {\xi} _ {1} \| ^ {2} ]} \tag {117} \\ \stackrel {(i i i)} {=} \frac {1}{(\lambda n - \widetilde {\ell} (1 + \widetilde {\ell} / \lambda) m)} \mathbb {E} \left[ \left\| \sum_ {z _ {i} \in U} \nabla_ {\omega} \widetilde {f} \left(\omega_ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}\right) \right\| \right] + \sqrt {d _ {1}} \sigma_ {1} \\ \stackrel {(i v)} {\leq} \frac {1}{(\lambda n - \tilde {\ell} (1 + \tilde {\ell} / \lambda) m)} \mathbb {E} \left[ \sum_ {z _ {i} \in U} \left(\| \nabla_ {\boldsymbol {\omega}} f (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu} _ {S} ^ {*}; z _ {i}) \| + \lambda \| \boldsymbol {\omega} _ {S} ^ {*} \| + \lambda \| \boldsymbol {\nu} _ {S} ^ {*} \|\right) \right] + \sqrt {d _ {1}} \sigma_ {1} \\ \stackrel {(v i)} {\leq} \frac {3 m L}{(\lambda n - \widetilde {\ell} (1 + \widetilde {\ell} / \lambda) m)} + \sqrt {d _ {1}} \sigma_ {1}, \\ \end{array}
$$
where the inequality  $(i)$  uses the triangle inequality and the inequality  $(ii)$  follows from an application of eq.(71), together with the Jensen's inequality to bound  $\mathbb{E}[\|\pmb{\xi}_1\|]$ . The equality  $(iii)$  holds because the vector  $\pmb{\xi}_1 \sim \mathcal{N}(0, \sigma_1^2\mathbf{I}_{d_1})$  and thus we have  $\mathbb{E}[\|\pmb{\xi}_1\|^2] = d_1\sigma_1^2$ . The inequality  $(iv)$  uses the definition of the function  $\widetilde{f}$  and the triangle inequality. The inequality  $(vi)$  is due to the fact that  $f(\omega, \nu; z)$  is  $L$ -Lipshtz continuous and Lemma 11. Symmetrically, we have
$$
\mathbb {E} \left[ \left\| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \right\| \right] \leq \frac {3 m L}{(\lambda n - \widetilde {\ell} (1 + \widetilde {\ell} / \lambda) m)} + \sqrt {d _ {2}} \sigma_ {2}. \tag {118}
$$
Plugging eq.(117) and eq.(118) into eq.(115) with noise scales given in Lemma 14, we can get our generalization guarantee in terms of population weak PD risk:
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \lambda^ {6} + L ^ {2} \ell^ {2} / \lambda^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \lambda \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right)\right). \tag {119}
$$
Similarly, using an application of eq.(83) together with Lemma 12, Lemma 14, eq.(117) and eq.(118), we can get the following population strong PD risk:
$$
\triangle^ {s} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(L ^ {3} \ell^ {3} \rho / \lambda^ {6} + L ^ {2} \ell^ {2} / \lambda^ {3}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \frac {L ^ {2} \ell}{\lambda^ {2} n} + \lambda \left(B _ {\boldsymbol {\omega}} ^ {2} + B _ {\boldsymbol {\nu}} ^ {2}\right)\right). \tag {120}
$$
Lemma 16 (Deletion Capacity). Under the settings of Theorem 5, the deletion capacity of Algorithm 6 is
$$
m _ {\epsilon , \delta , \gamma} ^ {A, \bar {A}} \left(d _ {1}, d _ {2}, n\right) \geq c \cdot \frac {n \sqrt {\epsilon}}{\left(d \log (1 / \delta)\right) ^ {1 / 4}}, \tag {121}
$$
where the constant  $c$  depends on  $L, l, \rho, B_{\omega}$  and  $B_{\nu}$ .
Proof. By the definition of deletion capacity, in order to ensure the population PD risk derived in Lemma 15 is bounded by  $\gamma$ , it suffices to let  $m_{\epsilon, \delta, \gamma}^{A, \bar{A}}(d_1, d_2, n) \geq c \cdot \frac{n \sqrt{\epsilon}}{(d \log(1 / \delta))^{1/4}}$ .
C.4 Minimax Unlearning Algorithm for Smooth Convex-Strongly-Concave Loss Function
In this section, we briefly discuss the extension to the smooth C-SC setting. The SC-C setting is symmetric and thus omitted here.
Given the loss function  $f(\omega, \nu; z)$  that satisfies Assumption 1 with  $\mu_{\nu}$ -strong concavity in  $\nu$ , we define the regularized function as  $\widetilde{f}(\omega, \nu; z) = f(\omega, \nu; z) + \frac{\lambda}{2} \| \omega \|^2$ . Our minimax learning and minimax unlearning algorithms for C-SC loss function  $f$  denoted by  $A_{c - sc}$  and  $\bar{A}_{c - sc}$  are given in Algorithm 7 and Algorithm 8 respectively. Additionally, we denote the regularized population loss by  $\widetilde{F}(\omega, \nu) := \mathbb{E}_{z \sim \mathcal{D}}[\widetilde{f}(\omega, \nu; z)]$  and regularized empirical loss by  $\widetilde{F}_S(\omega, \nu) := \frac{1}{n} \sum_{i=1}^{n} \widetilde{f}(\omega, \nu; z_i)$ .
Algorithm 7 Mimimax Learning Algorithm ( $A_{c - sc}$ )
Input: Dataset  $S:\{z_i\}_{i = 1}^n\sim \mathcal{D}^n$  , loss function:  $f(\pmb {\omega},\pmb {\nu};z)$  , regularization parameter:  $\lambda$
$$
\widetilde {f} (\boldsymbol {\omega}, \boldsymbol {\nu}; z) = f (\boldsymbol {\omega}, \boldsymbol {\nu}; z) + \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2}. \tag {122}
$$
Algorithm 8 Certified Minimax Unlearning for Convex-Strongly-Concave Loss  $(\bar{A}_{c - sc})$
Input: Delete requests  $U: \{z_j\}_{j=1}^m \subseteq S$ , output of  $A_{c-sc}(S): (\omega_S^*, \nu_S^*)$ , memory variables  $T(S): \{\mathsf{D}\omega\omega\widetilde{F}_S(\boldsymbol{\omega}_S^*, \boldsymbol{\nu}_S^*), \mathsf{D}\nu\upsilon\widetilde{F}_S(\boldsymbol{\omega}_S^*, \boldsymbol{\nu}_S^*)\}$ , loss function:  $f(\boldsymbol{\omega}, \boldsymbol{\nu}; z)$ , regularization parameter:  $\lambda$ , noise parameters:  $\sigma_1, \sigma_2$ .
$$
\widetilde {f} (\boldsymbol {\omega}, \boldsymbol {\nu}; z) = f (\boldsymbol {\omega}, \boldsymbol {\nu}; z) + \frac {\lambda}{2} \| \boldsymbol {\omega} \| ^ {2}. \tag {123}
$$
2: Run the algorithm  $\bar{A}_{sc - sc}$  with delete requests  $U$ , learning variables  $(\omega_{S}^{*},\nu_{S}^{*})$ , memory variables  $T(S)$ , loss function  $\widetilde{f}$  and noise parameters  $\sigma_{1}$  and  $\sigma_{2}$ .
Note that the function  $\widetilde{f}(\omega, \nu; z)$  is  $\lambda$ -strongly convex in  $\omega$ ,  $\mu_{\nu}$ -strongly concave in  $\nu$ ,  $(\widetilde{L} := 2L + \lambda \| \omega \|$ -Lipschitz,  $(\widetilde{\ell} := \sqrt{2}(2\ell + \lambda))$ -gradient Lipschitz and  $\rho$ -Hessian Lipschitz. We also have  $\| \mathbb{D}_{\omega \omega} \widetilde{F}_S(\omega_S^*, \nu_S^*) \| \geq \lambda$ . Let  $(\omega_{S^{\setminus}}^*, \nu_{S^{\setminus}}^*)$  be the optimal solution of the loss function  $\widetilde{F}_{S^{\setminus}}(\omega, \nu)$  on the remaining dataset, i.e.,
$$
\left(\boldsymbol {\omega} _ {S \backslash} ^ {*}, \boldsymbol {\nu} _ {S \backslash} ^ {*}\right) := \arg \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \{\widetilde {F} _ {S \backslash} (\boldsymbol {\omega}, \boldsymbol {\nu}) := \frac {1}{n - m} \sum_ {z _ {i} \in S \backslash U} \widetilde {f} (\boldsymbol {\omega}, \boldsymbol {\nu}; z _ {i}) \}. \tag {124}
$$