SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {G} \left(x ^ {r}, y ^ {r}\right) = \left[ \begin{array}{c} \frac {1}{\eta} \left(x ^ {r} - \mathcal {P} _ {\chi} \left(x ^ {r} - \eta \nabla_ {x} \mathcal {L} \left(x ^ {r}, y ^ {r}; \lambda^ {r}\right)\right)\right) \\ \nabla_ {y} \mathcal {L} \left(x ^ {r}, y ^ {r}; \lambda^ {r}\right) \end{array} \right]. \tag {216}
$$
For the block-  $x$  , we have
$$
\left\| \frac {1}{\eta} \left(x ^ {r} - \mathcal {P} _ {\mathcal {X}} (x ^ {r} - \eta \nabla_ {x} \mathcal {L} (x ^ {r}, y ^ {r}; \lambda^ {r}))\right) \right\|
$$
$$
\leq \frac {1}{\eta} \left(\| x ^ {r + 1} - x ^ {r} \| + \| x ^ {r + 1} - \mathcal {P} _ {\mathcal {X}} (x ^ {r} - \eta (\nabla_ {x} f (x ^ {r}, y ^ {r}) + \lambda^ {r} (\nabla_ {x} g (x ^ {r}, y ^ {r}) - \nabla_ {x} g ^ {*} (x ^ {r}))) \|\right)
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \frac {1}{\eta} \left(3 \| x ^ {r + 1} - x ^ {r} \| + \eta \| \nabla_ {x} f (x ^ {r}, y ^ {r}) - \nabla_ {x} f (x ^ {r}, y ^ {r + 1}) \| + \eta \lambda^ {r + 1} \| \nabla_ {x} g (x ^ {r}, y ^ {r}) - \nabla_ {x} g (x ^ {r}, y ^ {r + 1}) \|\right) \\ + \lambda^ {r + 1} \| \nabla_ {x} g \left(x ^ {r}, u ^ {r + 1}\right)) - \nabla_ {x} g ^ {*} \left(x ^ {r}\right) \| + \left| \lambda^ {r + 1} - \lambda^ {r} \right| \| \nabla_ {x} g \left(x ^ {r}, y ^ {r}\right) - \nabla_ {x} g ^ {*} \left(x ^ {r}\right) \| + p \| x ^ {r} - w ^ {r} \| \\ \leq \left(\frac {3}{\eta} + p\right) \| x ^ {r + 1} - x ^ {r} \| + \frac {p}{\beta} \| w ^ {r + 1} - w ^ {r} \| \\ + \left(L _ {f} + L _ {g} \lambda^ {r + 1}\right) \| y ^ {r + 1} - y ^ {r} \| + \lambda^ {r + 1} L _ {g} d _ {\mathcal {S} \left(x ^ {r}\right)} \left(u ^ {r + 1}\right) + 2 \ell_ {g} \left| \lambda^ {r + 1} - \lambda^ {r} \right| \tag {217} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {\leq} \left(\frac {3}{\eta} + p\right) \| x ^ {r + 1} - x ^ {r} \| + \frac {p}{\beta} \| w ^ {r + 1} - w ^ {r} \| + (L _ {f} + L _ {g} \lambda^ {r + 1}) \| y ^ {r + 1} - y ^ {r} \| \\ + 2 \ell_ {g} \tau | g \left(x ^ {r}, y ^ {r}\right) - g \left(x ^ {r}, u ^ {r + 1}\right) - \delta | + \lambda^ {r + 1} L _ {g} d _ {\mathcal {S} \left(x ^ {r}\right)} \left(u ^ {r + 1}\right) \tag {218} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(c)} {\leq} \left(\frac {3}{\eta} + p + \frac {2 \ell_ {g} ^ {2} \tau}{\eta (p - L)}\right) \| x ^ {r + 1} - x ^ {r} \| + \frac {p}{\beta} \| w ^ {r + 1} - w ^ {r} \| \\ + \left(L _ {f} + L _ {g} \lambda^ {r + 1} + \frac {\ell_ {g} ^ {2} \tau}{\alpha (p - L)}\right) \| y ^ {r + 1} - y ^ {r} \| \\ + 2 \ell_ {g} \tau | g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta | \\ + \left(2 \ell_ {g} \tau + \lambda^ {r + 1} L _ {g}\right) d _ {\mathcal {S} \left(x ^ {r}\right)} \left(u ^ {r + 1}\right) \tag {219} \\ \end{array}
$$
where in (a) we apply the following optimality condition of the  $x$ -subproblem
$$
\begin{array}{l} x ^ {r + 1} = \mathcal {P} _ {\mathcal {X}} \left[ x ^ {r + 1} - \eta \left(\nabla_ {x} f \left(x ^ {r}, y ^ {r + 1}\right) \right. \right. \\ \left. \left. + \lambda^ {r + 1} \left(\nabla_ {x} g \left(x ^ {r}, y ^ {r + 1}\right) - \nabla_ {x} g \left(x ^ {r}, u ^ {r + 1}\right)\right) + p \left(x ^ {r} - w ^ {r}\right) + \frac {1}{\eta} \left(x ^ {r + 1} - x ^ {r}\right)\right) \right], \\ \end{array}
$$
(b) results from the nonexpansiveness of the nonnegative projection operator, and (c) holds due to
$$
\begin{array}{l} \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g \left(x ^ {r}, y ^ {r}\right) \right| \\ \leq \frac {\ell_ {g}}{p - L} \left(\frac {1}{\alpha} \| y ^ {r + 1} - y ^ {r} \| + \frac {1}{\eta} \| x ^ {r + 1} - x ^ {r} \|\right), \tag {220} \\ \end{array}
$$
$$
\left| g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {r}\right) \right| \leq \frac {\ell_ {g}}{\eta (p - L)} \| x ^ {r + 1} - x ^ {r} \|. \tag {221}
$$
For the block-  $y$  , we have
$$
\begin{array}{l} \| \nabla_ {y} \mathcal {L} (x ^ {r}, y ^ {r}; \lambda^ {r}) \| \\ \leq \left\| \nabla_ {y} f \left(x ^ {r}, y ^ {r}\right) + \lambda^ {r} \nabla_ {y} g \left(x ^ {r}, y ^ {r}\right) \right\| (222) \\ \leq \frac {1}{\alpha} \| y ^ {r + 1} - y ^ {r} \| + \ell_ {g} \left| \lambda^ {r + 1} - \lambda^ {r} \right| + p \| y ^ {r} - z ^ {r} \| (223) \\ \stackrel {(7 \mathrm {c})} {\leq} \left(\frac {1}{\alpha} + p\right) \| y ^ {r + 1} - y ^ {r} \| + \ell_ {g} \left| \lambda^ {r + 1} - \lambda^ {r} \right| + \frac {p}{\beta} \| z ^ {r + 1} - z ^ {r} \| (224) \\ \stackrel {(b)} {\leq} \left(\frac {1}{\alpha} + p\right) \| y ^ {r + 1} - y ^ {r} \| + \ell_ {g} \tau | g \left(x ^ {r}, y ^ {r}\right) - g \left(x ^ {r}, u ^ {r + 1}\right) - \delta | + \frac {p}{\beta} \| z ^ {r + 1} - z ^ {r} \| (225) \\ \leq \left(\frac {1}{\alpha} + p + \frac {\ell_ {g} ^ {2} \tau}{\alpha (p - L)}\right) \| y ^ {r + 1} - y ^ {r} \| + \frac {2 \ell_ {g} ^ {2} \tau}{\eta (p - L)} \| x ^ {r + 1} - x ^ {r} \| + \frac {p}{\beta} \| z ^ {r + 1} - z ^ {r} \| \\ + \ell_ {g} \tau | g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta | \\ + \ell_ {g} \tau d _ {\mathcal {S} \left(x ^ {r}\right)} \left(u ^ {r + 1}\right) (226) \\ \end{array}
$$
where in (a) we apply the following optimality condition of the  $y$ -subproblem
$$
\nabla_ {y} f \left(x ^ {r}, y ^ {r}\right) + \lambda^ {r + 1} \nabla_ {y} g \left(x ^ {r}, y ^ {r}\right) + p \left(y ^ {r} - z ^ {r}\right) + \frac {1}{\alpha} \left(y ^ {r + 1} - y ^ {r}\right) = 0, \tag {227}
$$
and  $(b)$  comes from the update rule of the dual variable, i.e., the second line of Algorithm 1
Therefore, the primal optimality gap can be quantified as follows:
$$
\begin{array}{l} \| \mathcal {G} (x ^ {r}, y ^ {r}) \| ^ {2} \\ \leq 5 \left(\left(\frac {3}{\eta} + p + \frac {2 \ell_ {g} ^ {2} \tau}{\eta (p - L)}\right) ^ {2} + \left(\frac {2 \ell_ {g} ^ {2} \tau}{\eta (p - L)}\right) ^ {2}\right) \| x ^ {r + 1} - x ^ {r} \| ^ {2} \\ + 5 \left(\left(L _ {f} + L _ {g} \lambda^ {r + 1} + \frac {2 \ell_ {g} ^ {2} \tau}{\alpha (p - L)}\right) ^ {2} + \left(\frac {1}{\alpha} + p + \frac {\ell_ {g} ^ {2} \tau}{\alpha (p - L)}\right) ^ {2}\right) \| y ^ {r + 1} - y ^ {r} \| ^ {2} \\ + \frac {5 p ^ {2}}{\beta^ {2}} \| w ^ {r + 1} - w ^ {r} \| ^ {2} + \frac {5 p ^ {2}}{\beta^ {2}} \| z ^ {r + 1} - z ^ {r} \| ^ {2} \\ + 2 5 \ell_ {g} ^ {2} \tau^ {2} | g (x ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1}), y ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1})) - g ^ {*} (x ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1})) - \delta | ^ {2} \\ + 5 \left(\left(2 \ell_ {g} \tau + \lambda^ {r + 1} L _ {g}\right) ^ {2} + \ell_ {g} ^ {2} \tau^ {2}\right) d _ {\mathcal {S} \left(x ^ {r}\right)} ^ {2} \left(u ^ {r + 1}\right). \tag {228} \\ \end{array}
$$
From (144), we have the following three inequalities.
The first one is
$$
\min  \left\{\frac {1}{2 4 \eta}, \frac {1}{1 2 \alpha} \right\} \left(\| x ^ {r + 1} - x ^ {r} \| ^ {2} + \| y ^ {r + 1} - y ^ {r} \| ^ {2}\right) \leq \mathcal {Q} ^ {r} - \mathcal {Q} ^ {r + 1} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}}. \tag {229}
$$
The second one is
$$
\frac {p}{1 2 \beta} \left(\| w ^ {r + 1} - w ^ {r} \| ^ {2} + \| z ^ {r + 1} - z ^ {r} \| ^ {2}\right) \leq \mathcal {Q} ^ {r} - \mathcal {Q} ^ {r + 1} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}}. \tag {230}
$$
The third one is
$$
\begin{array}{l} \frac {\tau}{1 8 9} \left| \right. g \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right), y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\left. \right) - g ^ {*} \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \left. \right| ^ {2} \\ \leq \mathcal {Q} ^ {r} - \mathcal {Q} ^ {r + 1} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}}. \tag {231} \\ \end{array}
$$
$$
\rho_ {1} \triangleq \min  \left\{\frac {1}{2 4 \eta}, \frac {1}{1 2 \alpha} \right\} \tag {232a}
$$
$$
\begin{array}{l} \rho_ {2} \triangleq \max  \left\{5 \left(\left(\frac {3}{\eta} + p + \frac {2 \ell_ {g} ^ {2} \tau}{\eta (p - L)}\right) ^ {2} + \left(\frac {2 \ell_ {g} ^ {2} \tau}{\eta (p - L)}\right) ^ {2}\right), \right. \\ \left. \left. 5 \left(\left(L _ {f} + L _ {g} \lambda^ {r + 1} + \frac {2 \ell_ {g} ^ {2} \tau}{\alpha (p - L)}\right) ^ {2} + \left(\frac {1}{\alpha} + p + \frac {\ell_ {g} ^ {2} \tau}{\alpha (p - L)}\right) ^ {2}\right) \right\}. \right. \tag {232b} \\ \end{array}
$$
Plugging (229), (230), and (231) into (228) along with (232a) and (232b), we can have
$$
\begin{array}{l} \| \mathcal {G} (x ^ {r}, y ^ {r}) \| ^ {2} \leq \left(\frac {\rho_ {2}}{\rho_ {1}} + \frac {6 0 p}{\beta} + 2 5 \cdot 1 8 9 \ell_ {g} ^ {2} \tau\right) \left(\mathcal {Q} ^ {r} - \mathcal {Q} ^ {r + 1} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}}\right) \\ + 5 \left(\left(2 \ell_ {g} \tau + \lambda^ {r + 1} L _ {g}\right) ^ {2} + \ell_ {g} ^ {2} \tau^ {2}\right) d _ {\mathcal {S} (x ^ {r})} ^ {2} (u ^ {r + 1}). \\ \end{array}
$$
Note that according to (32) we have
$$
L _ {g} ^ {2} \left(\lambda^ {r + 1}\right) ^ {2} \left(1 - 2 \gamma \mu_ {g}\right) ^ {T r} D _ {\mathcal {S}} \leq \frac {L _ {g} ^ {2} D _ {\mathcal {S}}}{r ^ {2}}, \tag {233}
$$
when  $T^r \geq -2\log_{1 - 2\gamma \mu_g}\lambda^{r + 1}r$
Applying the telescoping sum over  $r = 1,\ldots ,T$  yields
$$
\begin{array}{l} \frac {1}{T} \sum_ {r = 1} ^ {T} \| \mathcal {G} (x ^ {r}) \| ^ {2} \\ \leq \frac {1}{T} \left(\frac {\rho_ {2}}{\rho_ {1}} + \frac {6 0 p}{\beta} + 2 5 \cdot 1 8 9 \ell_ {g} ^ {2} \tau\right) \sum_ {r = 1} ^ {T} \left(\mathcal {Q} ^ {r} - \mathcal {Q} ^ {r + 1}\right) \\ + \frac {1}{T} \left(\frac {\rho_ {2}}{\rho_ {1}} + \frac {6 0 p}{\beta} + 2 5 \cdot 1 8 9 \ell_ {g} ^ {2} \tau\right) \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \sum_ {r = 1} ^ {T} \frac {D _ {\mathcal {S}}}{r ^ {2}} + \left(1 0 + 4 5 \ell_ {g} ^ {2} \tau^ {2}\right) \sum_ {r = 1} ^ {T} \frac {D _ {\mathcal {S}}}{r ^ {2}} \\ \stackrel {(a)} {\leq} \frac {1}{T} \left(\frac {\rho_ {2}}{\rho_ {1}} + \frac {6 0 p}{\beta} + 2 5 \cdot 1 8 9 \ell_ {g} ^ {2} \tau\right) \left(\mathcal {Q} ^ {1} - \underline {{f}}\right) \\ + \frac {1}{T} \left(\left(\frac {\rho_ {2}}{\rho_ {1}} + \frac {6 0 p}{\beta} + 2 5 \cdot 1 8 9 \ell_ {g} ^ {2} \tau\right) \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) + \left(1 0 + 4 5 \ell_ {g} ^ {2} \tau^ {2}\right)\right) D _ {\mathcal {S}} \\ \end{array}
$$
where  $(a)$  results from the fact  $\sum_{r=1}^{T} r^{-2} \leq 1 + \int_{1}^{T} x^{-2} dx = 2 - T^{-1}$ .
We can obtain the convergence rate based on the following two cases: 1)  $\delta > 1$  and 2)  $\delta \leq 1$ .
If  $\delta > 1$ , we have  $\lambda^{r + 1} = \mathcal{O}(1)$ . We choose
$$
\eta , \alpha = \mathcal {O} \left(\frac {1}{\lambda^ {r + 1}}\right), p = \Theta \left(\lambda^ {r + 1}\right), \gamma , \tau = \mathcal {O} (1), \zeta , \beta = \mathcal {O} \left(\frac {\delta}{p}\right), T ^ {r} = \Omega (\log \lambda^ {r + 1}). \tag {234}
$$
such that the step-sizes satisfy (143). Note that as  $\zeta$  satisfies (169) and (173), it leads to  $\varrho = \mathcal{O}(\sqrt{\delta})$  and the strong error bound being true automatically.
If  $0 < \delta \leq 1$ , we choose
$$
\eta , \alpha = \mathcal {O} \left(\frac {1}{\lambda^ {r + 1}}\right), p = \Theta (\lambda^ {r + 1}), \zeta , \beta = \Theta (\delta^ {1. 5}), \tau = \mathcal {O} (1), T ^ {r} = \Omega (\log \lambda^ {r + 1}), \tag {235}
$$
such that the step-sizes satisfy (143), (169), (173) and (177), which leads to  $\varrho = \mathcal{O}(\sqrt{\delta})$  and the strong error bound being true again.
Finally, it can be easily checked that when  $\delta > 1$ , according to (234), we can have  $\frac{1}{T}\sum_{r=1}^{T}\|\mathcal{G}(x^r,y^r)\|^2 \leq \mathcal{O}(1/T)$ , when  $\delta \leq 1$ , we can obtain
$$
\frac {\rho_ {2}}{\rho_ {1}} = \mathcal {O} \left(\lambda^ {r + 1}\right), \quad \frac {p}{\beta} = \Theta \left(\frac {\lambda^ {r + 1}}{\delta^ {2 . 5}}\right), \quad \mathcal {Q} ^ {1} = \mathcal {O} (p) \tag {236}
$$
which gives  $\frac{1}{T}\sum_{r = 1}^{T}\| \mathcal{G}(x^r,y^r)\| ^2\leq \mathcal{O}(1 / (\delta^{2.5}T))$
Constraint Violation. Similarly, we can quantify
$$
\begin{array}{l} \left| g \left(x ^ {r}, y ^ {r}\right) - g ^ {*} \left(x ^ {r}\right) - \delta \right| _ {+} ^ {2} \\ \leq \left| g \left(x ^ {r}, y ^ {r}\right) - g \left(x ^ {r}, u ^ {r + 1}\right) + g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) - \delta \right| _ {+} ^ {2} \\ \stackrel {(a)} {\leq} 2 \left| g \left(x ^ {r}, y ^ {r}\right) - g \left(x ^ {r}, u ^ {r + 1}\right) - \delta \right| _ {+} ^ {2} + 2 \left| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \right| _ {+} ^ {2} \\ \leq \frac {2}{\tau^ {2}} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} + 2 | g (x ^ {r}, u ^ {r + 1}) - g ^ {*} (x ^ {r}) | _ {+} ^ {2} \\ \end{array}
$$
where  $(a)$  holds due to  $g(x^{r},u^{r + 1}) - g^{*}(x^{r})\geq 0$
From (144), we have
$$
\frac {2}{\tau^ {2}} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} \leq \frac {8}{\tau} \left(\mathcal {Q} ^ {r} - \mathcal {Q} ^ {r + 1} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}}\right).
$$
Applying the telescoping sum yields
$$
\begin{array}{l} \frac {1}{T} \sum_ {r = 1} ^ {T} \left| g (x ^ {r}, y ^ {r}) - g ^ {*} (x ^ {r}) - \delta \right| _ {+} ^ {2} \\ \leq \frac {8}{\tau} \frac {\mathcal {Q} ^ {1} - \underline {{f}} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) D _ {\mathcal {S}}}{T} + \frac {4 \ell_ {2} D _ {\mathcal {S}}}{T} \tag {237} \\ \end{array}
$$
where  $(a)$  results from the fact  $\sum_{r=1}^{T} r^{-2} \leq 1 + \int_{1}^{T} x^{-2} dx = 2 - T^{-1}$ . Therefore, when  $\delta > 1$ , we select step-sizes by (234), which gives  $\frac{1}{T} \sum_{r=1}^{T} |g(x^r, y^r) - g^*(x^r) - \delta|_+^2 = \mathcal{O}(1 / T)$ ; when  $0 < \delta \leq 1$ , we choose step-sizes by (235), which gives  $\frac{1}{T} \sum_{r=1}^{T} |g(x^r, y^r) - g^*(x^r) - \delta|_+^2 = \mathcal{O}(1 / (\sqrt{\delta} T))$ .
Slackness. If  $\lambda^r = 0$ , then it is trivial that  $|g(x^r, y^r) - g^*(x^r) - \delta| \lambda^r$  is zero. So, we only need to consider the case where  $\lambda^r > 0$  as follows.
Step 1. Note that
$$
\begin{array}{l} \left| g \left(x ^ {r}, y ^ {r}\right) - g ^ {*} \left(x ^ {r}\right) - \delta \right| ^ {2} \\ \leq 3 \left| g \left(x ^ {r}, y ^ {r}\right) - g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(x ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) \right| ^ {2} \\ + 3 \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| ^ {2} \\ + 3 \left| g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {r}\right) \right| ^ {2} (238) \\ \leq 6 \ell_ {g} ^ {2} \| y ^ {r} - y ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1}) \| ^ {2} + 3 \left(2 \ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) \| x ^ {r} - x ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1}) \| ^ {2} \\ + 3 \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(x ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| ^ {2} (239) \\ \leq \frac {6 \ell_ {g} ^ {2}}{\alpha^ {2} (p - L) ^ {2}} \| y ^ {r + 1} - y ^ {r} \| ^ {2} + \frac {3 (2 \ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2})}{\eta^ {2} (p - L) ^ {2}} \| x ^ {r + 1} - x ^ {r} \| ^ {2} \\ + 3 \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(x ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| ^ {2} (240) \\ \leq \underbrace{\frac{3}{(p - L)^{2}}\max\left\{\frac{2\ell_{g}^{2}}{\alpha^{2}},\frac{2\ell_{g}^{2} + \ell_{g}^{\prime 2}}{2\eta^{2}}\right\}}_{\triangleq_{\rho_{3}}}\big(\| x^{r + 1} - x^{r}\|^{2} + \| y^{r + 1} - y^{r}\|^{2}\big) \\ + 3 \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| ^ {2}. (241) \\ \end{array}
$$
Step 2. Then, we can get
$$
\begin{array}{l} \frac {1}{T} \sum_ {r = 1} ^ {T} \| g (x ^ {r + 1}, y ^ {r + 1}) - g ^ {*} (x ^ {r + 1}) - \delta \| ^ {2} \| \lambda^ {r + 1} \| ^ {2} \\ \leq \frac {1}{T} \sum_ {r = 1} ^ {T} \rho_ {3} \left(\| x ^ {r + 2} - x ^ {r + 1} \| ^ {2} + \| y ^ {r + 2} - y ^ {r + 1} \| ^ {2}\right) \| \lambda^ {r + 1} \| ^ {2} \\ + 3 \left| g \left(x ^ {*} \left(v ^ {r + 1}; \lambda^ {r + 2}\right), y ^ {*} \left(v ^ {r + 1}; \lambda^ {r + 2}\right)\right) - g ^ {*} \left(x ^ {*} \left(v ^ {r + 1}; \lambda^ {r + 2}\right)\right) - \delta \right| ^ {2} \| \lambda^ {r + 1} \| ^ {2} \tag {242} \\ \leq \left(\frac {\rho_ {3}}{\rho_ {1}} + \frac {3 \cdot 1 8 9}{\tau}\right) \max  \left\{\| \lambda^ {r + 1} \| ^ {2} \right\} \frac {\mathcal {Q} ^ {1} - \underline {{f}} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) D _ {\mathcal {S}}}{T} \\ \end{array}
$$
where  $(a)$  holds as when  $\delta > 1$  we have  $\lambda^{r + 1} = \mathcal{O}(1)$ , which gives  $\| g(x^{r + 1},y^{r + 1}) - g^{*}(x^{r + 1}) - \delta \| ^2\| \lambda^{r + 1}\| ^2 = \mathcal{O}(1 / T)$ ; when  $\delta \leq 1$ , we have  $\rho_3 / \rho_1 = \mathcal{O}(1 / \lambda^{r + 1}),\| \lambda^{r + 1}\| ^2 /\tau \leq \mathcal{O}(1 / \delta)$ , so we can obtain  $\| g(x^{r + 1},y^{r + 1}) - g^{*}(x^{r + 1}) - \delta \| ^2\| \lambda^{r + 1}\| ^2 = \mathcal{O}(1 / (\delta^{3 / 2}T))$ .
E Additional Numerical Results
In this section, we present additional numerical results obtained from the Fashion MNIST data set and compare the performance of SLM with other algorithms, such as BOME [15], V-PBGD [16], BVFSM [52], and ITD[13].
(b) Test accuracy
Figure 4: Convergence performance of SLM, BOME [15], and V-PBGD [16] on the neural network with the fashion MNIST data set.
E.1 Fashion MNIST Data Set
We also evaluate the performance of these algorithms using the fashion MNIST dataset. The neural network architecture employed is the same as in the previous case. For all the compared algorithms, we set the step-sizes of the block-  $x$  and block-  $y$  updates to 0.1. The step-size  $\gamma$  for the auxiliary block-  $u$  update is set to 0.01. Figure 4 illustrates that SLM with  $\delta = 1$  consistently outperforms the other algorithms in this example. This suggests that allowing for a certain tolerance level for the LL objective can enhance generalization performance. The convergence rates of all the algorithms are similar. However, it is important to note that both BOME and V-PBGD fail to achieve the optimal solution at the LL level since there exists a discrepancy of  $c_{\mathrm{error}} = 0.78$  for V-PBGD and  $c_{\mathrm{error}} = 0.42$  for BOME. Consequently, none of the methods can find the KKT points of problem (2) in this particular case. Nonetheless, SLM can attain the KKT solution of problem (1) with  $c_{\mathrm{error}}$  values below 1 and 0.5 respectively. This further underscores the advantages of utilizing this class of structured constrained problems compared to the BO formulation.
E.2 Comparison with BVFSM and ITD w.r.t. Runtime
In addition, we conduct a numerical performance evaluation of our problem formulation and SLM algorithm on a different data hypercleaning task described in [15]. In this particular setup, a clipping function is applied at the lower level, and the training data consists of 50,000 samples. We compare the performance of SLM with two other baseline algorithms: the bi-level value-function-based sequential minimization (BVFSM) algorithm introduced in [52], and the iterative differentiation (ITD) based bilevel optimizer as described in [13].
Figure 5: Convergence performance (test accuracy) of BOME, SLM, V-PBGD, ITD, and BVFSM.
The results of these experiments are presented
in Figure 5. It is important to note that in this presentation, we plot test accuracy against runtime to ensure a fair comparison. The outcomes of the evaluation demonstrate that SLM achieves higher test accuracy compared to these existing benchmarks. Furthermore, the convergence speed of SLM closely aligns with that of V-PBGD and BVFSM.
This work primarily focuses on the design of optimization algorithms and the theoretical analysis of convergence. It does not involve any negative societal or ethical issues.