SlowGuess/NeurIPS_2023_OCR huggingface dataset:
An application of Lemma 11 implies that the empirical optimizer  $(\omega_{S}^{*},\nu_{S}^{*})$  returned by Algorithm 8 satisfies  $\| \omega_S^*\| \leq L / \lambda$ . Thus our domain of interest are  $\mathcal{W}\coloneqq \{\pmb {\omega}||\pmb {\omega}||\leq L / \lambda \}$ . Over the set  $\mathcal{W}$ , the function  $\widetilde{f} (\pmb {\omega},\pmb {\nu};z)$  is  $3L$ -Lipschitz continuous. Suppose the strongly-convex regularization parameter  $\lambda$  satisfies  $\lambda < \ell$ , then  $\widetilde{f} (\pmb {\omega},\pmb {\nu};z)$  has  $3\sqrt{2}\ell$ -Lipschitz gradients.
The corresponding theoretical results are given below.
Lemma 17 (Closeness Upper Bound). Let Assumption 1 hold. Assume the function  $f(\omega, \nu; z)$  is  $\mu_{\nu}$ -strongly concave in  $\pmb{\nu}$  and  $\| \mathbb{D}_{\pmb{\nu}\pmb{\nu}}\widetilde{F}_S(\pmb{\omega}_S^*,\pmb{\nu}_S^*)\| \geq \mu_{\pmb{\nu}\pmb{\nu}}$ . Let  $\mu = \min \{\mu_{\nu},\mu_{\nu \nu}\}$ . Then, we have the closeness bound between the intermediate variables  $(\widehat{\omega},\widehat{\pmb{\nu}})$  in Algorithm 8 and  $(\omega_{S\setminus}^{*},\nu_{S\setminus}^{*})$  in eq.(124):
$$
\left\{ \begin{array}{l} \| \boldsymbol {\omega} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\omega}} \| \leq \left(\frac {8 \sqrt {2} \widetilde {L} ^ {2} \widetilde {\ell} ^ {3} \rho}{\lambda^ {2} \mu^ {3}} + \frac {8 \widetilde {L} \widetilde {\ell} ^ {2}}{\lambda \mu}\right) \cdot \frac {m ^ {2}}{n (\lambda n - (\widetilde {\ell} + \widetilde {\ell} ^ {2} / \mu) m)}, \\ \| \boldsymbol {\nu} _ {S ^ {\backslash}} ^ {*} - \widehat {\boldsymbol {\nu}} \| \leq \left(\frac {8 \sqrt {2} \widetilde {L} ^ {2} \widetilde {\ell} ^ {3} \rho}{\lambda^ {3} \mu^ {2}} + \frac {8 \widetilde {L} \widetilde {\ell} ^ {2}}{\lambda \mu}\right) \cdot \frac {m ^ {2}}{n (\mu n - (\widetilde {\ell} + \widetilde {\ell} ^ {2} / \lambda) m)}. \end{array} \right. \tag {125}
$$
Proof. Since we now run the algorithms  $A_{sc - sc}$  and  $\bar{A}_{sc - sc}$  with the regularized loss function  $\widetilde{f}$ , the proof is identical to that of Lemma 10.
Lemma 18 (Minimax Unlearning Certification). Under the settings of Lemma 17, our minimax learning algorithm  $A_{c - sc}$  and unlearning algorithm  $\bar{A}_{c - sc}$  is  $(\epsilon ,\delta)$ -certified minimax unlearning if
wechoose
$$
\left\{ \begin{array}{l} \sigma_ {1} = \left(\frac {8 \sqrt {2} \widetilde {L} ^ {2} \widetilde {\ell} ^ {3} \rho}{\lambda^ {2} \mu^ {3}} + \frac {8 \widetilde {L} \widetilde {\ell} ^ {2}}{\lambda \mu}\right) \cdot \frac {2 m ^ {2} \sqrt {2 \log (2 . 5 / \delta)}}{n (\lambda n - (\widetilde {\ell} + \widetilde {\ell} ^ {2} / \mu) m) \epsilon}, \\ \sigma_ {2} = \left(\frac {8 \sqrt {2} \widetilde {L} ^ {2} \widetilde {\ell} ^ {3} \rho}{\lambda^ {3} \mu^ {2}} + \frac {8 \widetilde {\ell} ^ {2}}{\lambda \mu}\right) \cdot \frac {2 m ^ {2} \sqrt {2 \log (2 . 5 / \delta)}}{n (\mu n - (\widetilde {\ell} + \widetilde {\ell} ^ {2} / \lambda) m) \epsilon}. \end{array} \right. \tag {126}
$$
Proof. With the closeness upper bound in Lemma 17 and the given noise scales in eq.(126), the proof is identical to that of Theorem 6.  $\square$
Lemma 19 (Population Weak PD Risk). Under the same settings of Lemma 17, suppose the parameter space  $\mathcal{W}$  is bounded so that  $\max_{\omega \in \mathcal{W}}\| \boldsymbol {\omega}\| \leq B_{\omega}$ , the population weak PD risk for the certified minimax unlearning variables  $(\boldsymbol{\omega}^u,\boldsymbol{\nu}^u)$  returned by Algorithm 8 is
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(\frac {L ^ {3} \ell^ {3} \rho}{\lambda^ {3} \mu^ {3}} + \frac {L ^ {2} \ell^ {2}}{\lambda^ {2} \mu} + \frac {L ^ {2} \ell^ {2}}{\lambda \mu^ {2}}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \frac {m L ^ {2}}{\mu n} + \lambda B _ {\boldsymbol {\omega}} ^ {2}\right), \tag {127}
$$
where  $d = \max \{d_1, d_2\}$ . In particular, by setting the regularization parameter  $\lambda$  as:
$$
\begin{array}{l} \lambda = \max  \left\{\frac {L}{B _ {\omega}} \sqrt {\frac {m}{n}}, \frac {L \ell m}{B _ {\omega} \mu n} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 2}, \left(\frac {L ^ {2} \ell^ {2} m ^ {2} \sqrt {d \log (1 / \delta)}}{B _ {\omega} ^ {2} \mu n ^ {2} \epsilon}\right) ^ {1 / 3}, \right. \tag {128} \\ \left. \left(\frac {L ^ {3} \ell^ {3} \rho m ^ {2} \sqrt {d \log (1 / \delta)}}{B _ {\omega} ^ {2} \mu^ {3} n ^ {2} \epsilon}\right) ^ {1 / 4} \right\}, \\ \end{array}
$$
we have the following population weak PD risk:
$$
\begin{array}{l} \triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(c _ {1} \sqrt {\frac {m}{n}} + c _ {2} \frac {m}{n} + c _ {3} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 2} \frac {m}{n} \right. \tag {129} \\ + c _ {4} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 3} \left(\frac {m}{n}\right) ^ {2 / 3} + c _ {5} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 4} \sqrt {\frac {m}{n}}), \\ \end{array}
$$
where  $c_{1}, c_{2}, c_{3}, c_{4}$  and  $c_{5}$  are constants that depend only on  $L, l, \rho, \mu$  and  $B_{\omega}$ .
Proof. An application of [Zhang et al., 2021, Theorem 1] gives that
$$
\max  _ {\boldsymbol {\nu} \in \mathcal {V}} \mathbb {E} [ \widetilde {F} (\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}) ] - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \mathbb {E} [ \widetilde {F} (\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}) ] \leq \frac {1 8 \sqrt {2} L ^ {2}}{n} \left(\frac {1}{\lambda} + \frac {1}{\mu}\right). \tag {130}
$$
Using the relation above with an application of eq.(77) and eq.(109), we have
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathbb {E} \left[ L \| \boldsymbol {\omega} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \| \right] + \mathbb {E} \left[ L \| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \| \right] + \frac {1 8 \sqrt {2} L ^ {2}}{n} \left(\frac {1}{\lambda} + \frac {1}{\mu}\right) + \frac {\lambda B _ {\boldsymbol {\omega}} ^ {2}}{2}. \tag {131}
$$
By an application of eq.(117), we further get
$$
\mathbb {E} \left[ \left\| \boldsymbol {\omega} ^ {u} - \boldsymbol {\omega} _ {S} ^ {*} \right\| \right] \leq \frac {2 m L}{\lambda n - \widetilde {\ell} (1 + \widetilde {\ell} / \mu) m} + \sqrt {d _ {1}} \sigma_ {1}, \tag {132}
$$
$$
\mathbb {E} \left[ \left\| \boldsymbol {\nu} ^ {u} - \boldsymbol {\nu} _ {S} ^ {*} \right\| \right] \leq \frac {2 m L}{\mu n - \widetilde {\ell} (1 + \widetilde {\ell} / \lambda) m} + \sqrt {d _ {2}} \sigma_ {2}. \tag {133}
$$
Plugging eq.(132) and eq.(133) into eq.(131) with noise scales given in Lemma 18, we can get our generalization guarantee:
$$
\triangle^ {w} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(\frac {L ^ {3} \ell^ {3} \rho}{\lambda^ {3} \mu^ {3}} + \frac {L ^ {2} \ell^ {2}}{\lambda^ {2} \mu} + \frac {L ^ {2} \ell^ {2}}{\lambda \mu^ {2}}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \frac {m L ^ {2}}{\mu n} + \lambda B _ {\boldsymbol {\omega}} ^ {2}\right), \tag {134}
$$
where  $d = \max \{d_1,d_2\}$
Lemma 20 (Population Strong PD Risk). Under the same settings of Lemma 19, the population strong PD risk for  $(\omega^u,\nu^u)$  returned by Algorithm 8 is
$$
\begin{array}{l} \triangle^ {s} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(\frac {L ^ {3} \ell^ {3} \rho}{\lambda^ {3} \mu^ {3}} + \frac {L ^ {2} \ell^ {2}}{\lambda^ {2} \mu} + \frac {L ^ {2} \ell^ {2}}{\lambda \mu^ {2}}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \frac {m L ^ {2}}{\mu n}\right. \tag {135} \\ \left. + \frac {L ^ {2} \ell}{\lambda^ {3 / 2} \mu^ {1 / 2} n} + \frac {L ^ {2} \ell}{\lambda^ {1 / 2} \mu^ {3 / 2} n} + \lambda B _ {\boldsymbol {\omega}} ^ {2}\right), \\ \end{array}
$$
$$
\begin{array}{l} \lambda = \max  \left\{\frac {L}{B _ {\omega}} \sqrt {\frac {m}{n}}, \frac {L \ell m}{B _ {\omega} \mu n} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 2}, \left(\frac {L ^ {2} \ell^ {2} m ^ {2} \sqrt {d \log (1 / \delta)}}{B _ {\omega} ^ {2} \mu n ^ {2} \epsilon}\right) ^ {1 / 3}, \right. \tag {136} \\ \left. \left(\frac {L ^ {3} \ell^ {3} \rho m ^ {2} \sqrt {d \log (1 / \delta)}}{B _ {\omega} ^ {2} \mu^ {3} n ^ {2} \epsilon}\right) ^ {1 / 4}, \left(\frac {L ^ {2} \ell}{B _ {\omega} ^ {2} \mu^ {1 / 2} n}\right) ^ {2 / 5}, \frac {1}{\mu} \left(\frac {L ^ {2} \ell}{B _ {\omega} ^ {2} n}\right) ^ {2 / 3} \right\}. \\ \end{array}
$$
we have the following population strong PD risk:
$$
\begin{array}{l} \triangle^ {s} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(c _ {1} \sqrt {\frac {m}{n}} + c _ {2} \frac {m}{n} + c _ {3} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 2} \frac {m}{n} + c _ {4} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 3} \left(\frac {m}{n}\right) ^ {2 / 3} \right. \\ + c _ {5} \left(\frac {\sqrt {d \log (1 / \delta)}}{\epsilon}\right) ^ {1 / 4} \sqrt {\frac {m}{n}} + c _ {6} \frac {1}{n ^ {2 / 5}} + c _ {7} \frac {1}{n ^ {2 / 3}}), \tag {137} \\ \end{array}
$$
where  $c_{1}, c_{2}, c_{3}, c_{4}, c_{5}, c_{6}$  and  $c_{7}$  are constants that depend only on  $L, l, \rho, \mu$  and  $B_{\omega}$ .
Proof. An application of Lemma 9 gives that
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {\boldsymbol {\nu} \in \mathcal {V}} \widetilde {F} \left(\boldsymbol {\omega} _ {S} ^ {*}, \boldsymbol {\nu}\right) - \min  _ {\boldsymbol {\omega} \in \mathcal {W}} \widetilde {F} \left(\boldsymbol {\omega}, \boldsymbol {\nu} _ {S} ^ {*}\right) \right] \leq \frac {3 6 \sqrt {2} L ^ {2} (2 \ell + \lambda)}{n} \left(\frac {1}{\lambda^ {3 / 2} \mu^ {1 / 2}} + \frac {1}{\lambda^ {1 / 2} \mu^ {3 / 2}}\right) \tag {138} \\ \leq \frac {1 0 8 \sqrt {2} L ^ {2} \ell}{n} \left(\frac {1}{\lambda^ {3 / 2} \mu^ {1 / 2}} + \frac {1}{\lambda^ {1 / 2} \mu^ {3 / 2}}\right). \\ \end{array}
$$
Using an application of eq.(83) and eq.(111), together with eq.(138), eq.(132), eq.(133) and the noise scales given in Lemma 18, we have
$$
\begin{array}{l} \triangle^ {s} \left(\boldsymbol {\omega} ^ {u}, \boldsymbol {\nu} ^ {u}\right) \leq \mathcal {O} \left(\left(\frac {L ^ {3} \ell^ {3} \rho}{\lambda^ {3} \mu^ {3}} + \frac {L ^ {2} \ell^ {2}}{\lambda^ {2} \mu} + \frac {L ^ {2} \ell^ {2}}{\lambda \mu^ {2}}\right) \cdot \frac {m ^ {2} \sqrt {d \log (1 / \delta)}}{n ^ {2} \epsilon} + \frac {m L ^ {2}}{\lambda n} + \frac {m L ^ {2}}{\mu n}\right. \tag {139} \\ \left. + \frac {L ^ {2} \ell}{\lambda^ {3 / 2} \mu^ {1 / 2} n} + \frac {L ^ {2} \ell}{\lambda^ {1 / 2} \mu^ {3 / 2} n} + \lambda B _ {\omega} ^ {2}\right). \\ \end{array}
$$
Lemma 21 (Deletion Capacity). Under the same settings as Lemma 19, the deletion capacity of Algorithm 3 is
$$
m _ {\epsilon , \delta , \gamma} ^ {A, \bar {A}} \left(d _ {1}, d _ {2}, n\right) \geq c \cdot \frac {n \sqrt {\epsilon}}{\left(d \log (1 / \delta)\right) ^ {1 / 4}}, \tag {140}
$$
where the constant  $c$  depends on  $L, l, \rho, \mu$  and  $B_{\omega}$  and  $d = \max \{d_1, d_2\}$ .
Proof. By the definition of deletion capacity, in order to ensure the population PD risk derived in Lemma 19 or Lemma 20 is bounded by  $\gamma$ , it suffices to let  $m_{\epsilon, \delta, \gamma}^{A, \bar{A}}(d_1, d_2, n) \geq c \cdot \frac{n\sqrt{\epsilon}}{(d\log(1 / \delta))^{1/4}}$ .