SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} \\ = \left\| \lambda^ {r + 1} - \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) + \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) - \lambda^ {r} \right\| ^ {2} (127) \\ \geq \frac {\left\| \lambda_ {+} ^ {r + 1} \left(w ^ {r} , z ^ {r}\right) - \lambda^ {r} \right\| ^ {2}}{2} - \left\| \lambda^ {r + 1} - \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2}. (128) \\ \end{array}
$$
According to the definition of  $\lambda_{+}(w^{r},z^{r})$  (cf. (81a)), we have
$$
\left\| \lambda^ {r + 1} - \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \left\| \lambda^ {r} + \tau \left(g \left(x ^ {r}, y ^ {r}\right) - g \left(x ^ {r}, u ^ {r + 1}\right) - \delta\right) - \left[ \lambda^ {r + 1} + \tau \nabla_ {\lambda} K \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right), v ^ {r}; \lambda^ {r + 1}\right) \right] \right\| ^ {2} \\ \leq 2 \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} \\ + 2 \tau^ {2} \left(\| g (x ^ {r}, y ^ {r}) - g (x ^ {r}, u ^ {r + 1}) - \left(g \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right)\right) \| ^ {2}\right) \\ \leq 2 \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} + 6 \tau^ {2} \left(2 \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {r 2}\right) \| x ^ {r} - x ^ {*} (v ^ {r}; \lambda^ {r + 1}) \| ^ {2} + \ell_ {g} ^ {2} \| y ^ {r} - y ^ {*} (v ^ {r}; \lambda^ {r + 1}) \| ^ {2}\right) \\ + 6 \tau^ {2} \| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \| ^ {2} \tag {129} \\ \end{array}
$$
where  $(a)$  is true due to the nonexpansiveness of the nonnegative projection operator. Also, we have
$$
\begin{array}{l} \left\| \lambda^ {r} - \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \\ = \left\| \lambda^ {r} - \left[ \lambda^ {r + 1} + \tau \nabla_ {\lambda} K \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right), v ^ {r}; \lambda^ {r + 1}\right) \right] \right\| (130) \\ \geq \frac {\left\| \tau \nabla_ {\lambda} K \left(x ^ {*} \left(v ^ {r} ; \lambda^ {r + 1}\right) , y ^ {*} \left(v ^ {r} ; \lambda^ {r + 1}\right) , v ^ {r} ; \lambda^ {r + 1}\right) \right\| ^ {2}}{2} - \left\| \lambda^ {r + 1} - \lambda^ {r} \right\| ^ {2}. (131) \\ \end{array}
$$
Combining (128), (129), and (131) gives
$$
\begin{array}{l} \frac {7}{2} \left\| \lambda^ {r + 1} - \lambda^ {r} \right\| ^ {2} \\ \geq \frac {\| \tau \nabla_ {\lambda} K (x ^ {*} (v ^ {r} ; \lambda^ {r + 1}) , y ^ {*} (v ^ {r} ; \lambda^ {r + 1}) , v ^ {r} ; \lambda^ {r + 1}) \| ^ {2}}{4} \\ - 6 \tau^ {2} \left(2 \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) \| x ^ {r} - x ^ {*} (v ^ {r}; \lambda^ {r + 1}) \| ^ {2} + \ell_ {g} ^ {2} \| y ^ {r} - y ^ {*} (v ^ {r}; \lambda^ {r + 1}) \| ^ {2}\right) \\ - 6 \tau^ {2} \| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \| ^ {2}. \tag {132} \\ \end{array}
$$
Applying the primal error bounds (29a) and (29b), we can obtain
$$
\begin{array}{l} - \frac {1}{4 \tau} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} \\ \leq - \frac {\tau \| \nabla_ {\lambda} K (x ^ {*} (v ^ {r} ; \lambda^ {r + 1}) , y ^ {*} (v ^ {r} ; \lambda^ {r + 1}) , v ^ {r} ; \lambda^ {r + 1}) \| ^ {2}}{5 6} \\ + \frac {3 \tau}{7} \left(2 \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) \| x ^ {r} - x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right) \| ^ {2} + \ell_ {g} ^ {2} \| y ^ {r} - y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right) \| ^ {2} + \| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \| ^ {2}\right). (133) \\ \leq - \frac {\tau \| g \left(x ^ {*} \left(w ^ {r} , z ^ {r} ; \lambda^ {r + 1}\right) , y ^ {*} \left(w ^ {r} , z ^ {r} ; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r} , z ^ {r} ; \lambda^ {r + 1}\right)\right) - \delta \| ^ {2}}{5 6} \\ + \frac {3 \tau}{7} \left(\frac {2 \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) \sigma_ {3} ^ {2}}{\eta^ {2}} \| x ^ {r} - x ^ {r + 1} \| ^ {2} + \frac {\ell_ {g} ^ {2} \sigma_ {3} ^ {2}}{\alpha^ {2}} \| y ^ {r} - y ^ {r + 1} \| ^ {2} + \| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \| ^ {2}\right). (134) \\ \end{array}
$$
Step 3.) Substituting (134) and (126) back to (118) gives
$$
\begin{array}{l} \mathcal {Q} ^ {r + 1} - \mathcal {Q} ^ {r} \\ \leq - \left(\frac {1}{4 \eta} - \frac {1 7 \tau (\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}) \sigma_ {3} ^ {2}}{\eta^ {2}}\right) \| x ^ {r + 1} - x ^ {r} \| ^ {2} - \left(\frac {1}{2 \alpha} - \frac {9 \tau \ell_ {g} ^ {2} \sigma_ {3} ^ {2}}{\alpha^ {2}}\right) \| y ^ {r + 1} - y ^ {r} \| ^ {2} \\ \left. - p \left(\frac {1}{2 \beta} - \left(\frac {1}{\zeta} + \frac {2 p}{p - L}\right)\right) \| z ^ {r + 1} - z ^ {r} \| ^ {2} - p \left(\frac {1}{2 \beta} - \left(\frac {1}{\zeta} + \frac {2 p}{p - L}\right)\right) \| w ^ {r + 1} - w ^ {r} \| ^ {2} \right. \\ + 6 p \zeta \left(\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} + \| x ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r + 1}\right) \| ^ {2}\right) \\ - \frac {\tau}{5 6} \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| ^ {2} \\ + \frac {\mu_ {g} L _ {g} ^ {2} D _ {\mathcal {S}}}{r ^ {2}} + 2 \tau \| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \| ^ {2} - \frac {1}{4 \tau} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2}. \tag {135} \\ \end{array}
$$
Finally, by observing the weak dual error bound, we need to further quantify  $\| y^{*}(w^{r},z^{r};\lambda^{r + 1}) - \bar{y}^{*}(w^{r},z^{r})\|^{2} + \| x^{*}(w^{r},z^{r + 1};\lambda^{r + 1}) - \bar{x}^{*}(w^{r},z^{r + 1})\|^{2}$  as follows. Note that
$$
\begin{array}{l} \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} + \left\| x ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r + 1}\right) \right\| ^ {2} \\ \leq \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} + 3 \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \\ + 3 \| x ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \| ^ {2} + 3 \| \bar {x} ^ {*} \left(w ^ {r}, z ^ {r + 1}\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \tag {136} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} 2 \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} + 6 \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \\ + 2 \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) \right\| ^ {2} \\ + 6 \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) \right\| ^ {2} + 6 \sigma_ {1} ^ {2} \| z ^ {r + 1} - z ^ {r} \| ^ {2} \tag {137} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(b)} {\leq} 2 \| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} + 6 \| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \\ + 8 \sigma_ {2} ^ {2} \tau^ {2} \left\| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right\| ^ {2} \\ + 6 \sigma_ {1} ^ {2} \| z ^ {r + 1} - z ^ {r} \| ^ {2} \tag {138} \\ \end{array}
$$
where in  $(a)$  we use the primal error bounds (27a) and (27c),  $(b)$  holds as we first apply the primal error bounds (28a) and (28b) and then use the definition  $\lambda_{+}(w,z)$  as follows
$$
\begin{array}{l} \left\| \lambda^ {r + 1} - \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right) \right\| \\ = \left\| \lambda^ {r + 1} - P _ {\geq 0} \left[ \lambda^ {r + 1} + \tau \nabla_ {\lambda} K \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \right] \right\| (139) \\ \leq \tau \left\| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right\|. (140) \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {Q} ^ {r + 1} - \mathcal {Q} ^ {r} \\ \leq - \left(\frac {1}{4 \eta} - \frac {1 7 \tau \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) \sigma_ {3} ^ {2}}{\eta^ {2}}\right) \| x ^ {r + 1} - x ^ {r} \| ^ {2} - \left(\frac {1}{2 \alpha} - \frac {9 \tau \ell_ {g} ^ {2} \sigma_ {3} ^ {2}}{\alpha^ {2}}\right) \| y ^ {r + 1} - y ^ {r} \| ^ {2} \\ \left. - p \left(\frac {1}{2 \beta} - \left(\frac {1}{\zeta} + \frac {2 p}{p - L} + 3 6 \zeta \sigma_ {1} ^ {2}\right)\right) \| z ^ {r + 1} - z ^ {r} \| ^ {2} - p \left(\frac {1}{2 \beta} - \left(\frac {1}{\zeta} + \frac {2 p}{p - L}\right)\right) \| w ^ {r + 1} - w ^ {r} \| ^ {2} \right. \\ + 3 6 p \zeta \left(\| y ^ {*} (w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} (w ^ {r}, z ^ {r})) - \bar {y} ^ {*} (w ^ {r}, z ^ {r}) \| ^ {2} + \| x ^ {*} (w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} (w ^ {r}, z ^ {r})) - \bar {x} ^ {*} (w ^ {r}, z ^ {r}) \| ^ {2}\right) \\ - \tau \left(\frac {1}{5 6} - 4 8 p \zeta \sigma_ {2} ^ {2} \tau\right) | g (x ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1}), y ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1})) - g ^ {*} (x ^ {*} (w ^ {r}, z ^ {r}; \lambda^ {r + 1})) - \delta | ^ {2} \\ + \frac {\mu_ {g} L _ {g} ^ {2} D _ {\mathcal {S}}}{r ^ {2}} + 2 \tau \| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \| ^ {2} - \frac {1}{4 \tau} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} \tag {141} \\ \end{array}
$$
From Lemma 4, we know that  $d_{\mathcal{S}(x^r)}^2 (u^{r + 1}) \leq (1 - 2\gamma \mu_g)^{T^r}D_{\mathcal{S}}$ , so when  $T^r \geq -2\log_{(1 - 2\gamma \mu_g)}(\sqrt{\tau} r)$ ,
$$
2 \tau \| g \left(x ^ {r}, u ^ {r + 1}\right) - g ^ {*} \left(x ^ {r}\right) \| ^ {2} \leq \frac {2 \ell_ {g} ^ {2} D _ {\mathcal {S}}}{r ^ {2}}. \tag {142}
$$
Substituting (142) back to (141) gives the desired result.
B.5 Descent of Potential Function
Lemma 9.
Assume that A1-A5 are satisfied and that the local regularity condition holds. Suppose that the sequence  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r, \forall r\}$  is generated by SLM,  $p > L$ , and  $\lambda^r, y^r, \bar{y}^*(w^r, z^r), y^*(w^r, z^r; \lambda^{r+1})$  are bounded. When the step-sizes are chosen such that
$$
\gamma \leq \min \left\{\frac {1}{L _ {g}}, \frac {1}{2 \mu_ {g}} \right\}, \quad \eta \leq \min \left\{\frac {1}{L _ {K}}, \frac {(p - L) ^ {2}}{1 3 6 \tau (\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}) L _ {K} ^ {2}} \right\},
$$
$$
\alpha \leq \min  \left\{\frac {1}{L _ {K}}, \frac {(p - L) ^ {2}}{3 6 \tau \ell_ {g} ^ {2} L _ {K} ^ {2}} \right\}, \tau \leq \min  \left\{\frac {(p - L) ^ {2}}{3 4 5 6 p \zeta (p + L) ^ {2}}, \frac {1 / (1 2 6 \cdot 4 8)}{p \zeta \max  \left\{\sigma_ {\text {s t r o n g}} ^ {2} , \sigma_ {2} ^ {2} \right\}} \right\},
$$
$$
T ^ {r} \geq \max  \left\{- 2 \log_ {1 - 2 \gamma \mu_ {g}} (\lambda^ {r + 1} \sqrt {\eta} r), - 2 \log_ {1 - 2 \gamma \mu_ {g}} r \sqrt {\tau} \right\},
$$
$$
\beta \leq \frac {1 / 4}{\frac {1}{\zeta} + \frac {2 p}{p - L} + \frac {3 6 \zeta p ^ {2}}{(p - L) ^ {2}}} \leq 1, \quad \zeta \leq \sqrt {\frac {\varrho}{2}} \max  \left\{\frac {1}{\zeta_ {1}}, \frac {1}{\zeta_ {2}} \right\}, \quad \vartheta \leq \frac {\varrho}{\tau \zeta_ {3}}, \tag {143}
$$
then, we have
$$
\begin{array}{l} \mathcal {Q} ^ {r + 1} - \mathcal {Q} ^ {r} \\ \leq - \frac {1}{2 4 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2} - \frac {1}{1 2 \alpha} \| y ^ {r + 1} - y ^ {r} \| ^ {2} - \frac {p}{1 2 \beta} \| z ^ {r + 1} - z ^ {r} \| ^ {2} - \frac {p}{1 2 \beta} \| w ^ {r + 1} - w ^ {r} \| ^ {2} \\ - \frac {\tau}{1 8 9} \left| \right. g \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right), y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\left. \right) - g ^ {*} \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \left. \right| ^ {2} \\ - \frac {1}{4 \tau} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}} \tag {144} \\ \end{array}
$$
where  $\sigma_{\text{strong}} \triangleq \sigma_{\text{weak}} \sigma_{\text{aux}}$ , and  $\zeta_1, \zeta_2, \zeta_3$  are defined in (171) and (174).
Proof. From (104), it is clear that if we can select the step-sizes properly so that the coefficients in front of  $\| x^{r + 1} - x^r\| ^2$ ,  $\| y^{r + 1} - y^r\| ^2$ ,  $\| z^{r + 1} - z^r\| ^2$ ,  $\| w^{r + 1} - w^r\| ^2$  are strictly negative, then the potential function  $Q^r$  will be decreasing. To be more specific, the step-sizes are chosen as follows.
1) Selection of  $\eta$ . Given the condition of (41), we request
$$
\frac {1}{4 \eta} - \frac {1 7 \tau \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) \sigma_ {3} ^ {2}}{\eta^ {2}} = \frac {1}{4 \eta} - \frac {1 7 \tau \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right)}{\eta^ {2} (p - L) ^ {2}} > \frac {1}{8 \eta} > 0, \tag {145}
$$
$$
\eta <   \frac {(p - L) ^ {2}}{1 3 6 \tau \left(\ell_ {g} ^ {2} + \ell_ {g} ^ {\prime 2}\right) L _ {K} ^ {2}}. \tag {146}
$$
2) Selection of  $\alpha$
$$
\frac {1}{2 \alpha} - \frac {9 \tau \ell_ {g} ^ {2} \sigma_ {3} ^ {2}}{\alpha^ {2}} = \frac {1}{2 \alpha} - \frac {9 \tau \ell_ {g} ^ {2}}{\alpha^ {2} (p - L) ^ {2}} > \frac {1}{4 \alpha} > 0, \tag {147}
$$
$$
\alpha <   \frac {(p - L) ^ {2}}{3 6 \tau \ell_ {g} ^ {2} L _ {K} ^ {2}}. \tag {148}
$$
3) Selection of  $\beta$
$$
\frac {1}{2 \beta} - \left(\frac {1}{\zeta} + \frac {2 p}{p - L} + 3 6 \zeta \sigma_ {1} ^ {2}\right) > \frac {1}{4 \beta} > 0, \tag {149}
$$
$$
\beta <   \frac {1}{4} \frac {1}{\frac {1}{\zeta} + \frac {2 p}{p - L} + \frac {3 6 \zeta p ^ {2}}{(p - L) ^ {2}}}. \tag {150}
$$
4) Selection of  $\tau$
We need
$$
\frac {1}{5 6} - 4 8 p \zeta \sigma_ {2} ^ {2} \tau \geq \frac {1}{6 3} > 0, \tag {151}
$$
$$
\tau \leq \frac {(p - L) ^ {2}}{3 4 5 6 p \zeta (p + L) ^ {2}}. \tag {152}
$$
Then, consider the following two cases:
Case 1.
$$
\begin{array}{l} \frac {1}{3} \max  \left\{\frac {1}{8 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2}, \frac {1}{4 \alpha} \| y ^ {r + 1} - y ^ {r} \| ^ {2}, \frac {p}{4 \beta} \| z ^ {r + 1} - z ^ {r} \| ^ {2}, \frac {p}{4 \beta} \| w ^ {r + 1} - w ^ {r} \| ^ {2}, \right. \\ \left. \frac {\tau}{6 3} \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| ^ {2} \right\} \\ > \max  \left\{3 6 p \zeta \left(\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \right. \right. \\ + \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2}), \vartheta \tau \} \tag {153} \\ \end{array}
$$
where  $\vartheta$  is some parameter.
In this case, we can have
$$
\begin{array}{l} \mathcal {Q} ^ {r + 1} - \mathcal {Q} ^ {r} \leq - \frac {1}{2 4 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2} - \frac {1}{1 2 \alpha} \| y ^ {r + 1} - y ^ {r} \| ^ {2} - \frac {p}{1 2 \beta} \| z ^ {r + 1} - z ^ {r} \| ^ {2} - \frac {p}{1 2 \beta} \| w ^ {r + 1} - w ^ {r} \| ^ {2} \\ - \frac {\tau}{1 8 9} \left| \right. g \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right), y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\left. \right) - g ^ {*} \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \left. \right| ^ {2} \\ - \vartheta \tau - \frac {1}{4 \tau} \| \lambda^ {r + 1} - \lambda^ {r} \| ^ {2} + \left(\mu_ {g} L _ {g} ^ {2} + 2 \ell_ {g} ^ {2}\right) \frac {D _ {\mathcal {S}}}{r ^ {2}}, \tag {154} \\ \end{array}
$$
meaning that  $\mathcal{Q}^r$  is decreasing up to some constant at each step.
Case 2.
$$
\begin{array}{l} \frac {1}{3} \max  \left\{\frac {1}{8 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2}, \frac {1}{4 \alpha} \| y ^ {r + 1} - y ^ {r} \| ^ {2}, \frac {p}{4 \beta} \| z ^ {r + 1} - z ^ {r} \| ^ {2}, \frac {p}{4 \beta} \| w ^ {r + 1} - w ^ {r} \| ^ {2}, \right. \\ \left. \frac {\tau}{6 3} \left| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right| ^ {2} \right\} \\ \leq \max  \left\{3 6 p \zeta \left(\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \right. \\ + \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2}), \vartheta \tau \}. \tag {155} \\ \end{array}
$$
First, if  $36p\zeta \big(\| y^{*}(w^{r},z^{r};\lambda_{+}^{r + 1}(w^{r},z^{r})) - \bar{y}^{*}(w^{r},z^{r})\|^{2} + \| x^{*}(w^{r},z^{r};\lambda_{+}^{r + 1}(w^{r},z^{r})) - \bar{x}^{*}(w^{r},z^{r})\|^{2}\big)\geq \vartheta \tau$  , then we have the following results.
Recall the weak error bound
$$
\begin{array}{l} \left\| y ^ {*} (w, z; \lambda_ {+} (w, z)) - \bar {y} ^ {*} (w, z) \right\| ^ {2} + \left\| x ^ {*} (w, z; \lambda_ {+} (w, z)) - \bar {x} ^ {*} (w, z) \right\| ^ {2} \\ \leq \sigma_ {\text {w e a k}} \| \lambda - \lambda_ {+} (w, z) \| \| \lambda (w, z) - \lambda_ {+} (w, z) \| \tag {156} \\ \end{array}
$$
where  $\lambda (v)\in \arg \max_{0\leq \lambda \leq \Lambda}K(\bar{x}^{*}(v),\bar{y}^{*}(v),v;\lambda)$
We can get
$$
\begin{array}{l} \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} + \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \\ \leq \underbrace {\sigma_ {\text {w e a k}} 2 \Lambda} _ {\triangleq \bar {\sigma} _ {\mathrm {w}}} \tau \left\| g \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right) - g \left(x ^ {*} \left(v ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right\|, \tag {157} \\ \end{array}
$$
$$
\left\| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right\| \leq 3 \cdot 63 \cdot 3 6 p \zeta \bar {\sigma} _ {\mathrm {w}}. \tag {158}
$$
$$
\begin{array}{l} \frac {p}{4 \beta} \| z ^ {r + 1} - z ^ {r} \| ^ {2} \\ \leq 3 \cdot 36 p \zeta \left(\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} + \| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2}\right) \\ \leq 3 \cdot 3 6 p \zeta \bar {\sigma} _ {\mathrm {w}} \| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \| (159) \\ \leq 3 ^ {2} \cdot 6 3 \cdot 3 6 ^ {2} p ^ {2} \zeta^ {2} \bar {\sigma} _ {\mathrm {w}} ^ {2}. (160) \\ \end{array}
$$
$$
\left\{\frac {1}{8 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2}, \frac {1}{4 \alpha} \| y ^ {r + 1} - y ^ {r} \| ^ {2}, \frac {p}{4 \beta} \| w ^ {r + 1} - w ^ {r} \| ^ {2} \right\} \leq 3 ^ {2} \cdot 6 3 \cdot 3 6 ^ {2} p ^ {2} \zeta^ {2} \bar {\sigma} _ {\mathrm {w}} ^ {2}. \tag {161}
$$
These results imply that the iterates generated by SLM will converge to some point within a ball with a radius of  $\mathcal{O}(p^2\zeta^2\bar{\sigma}_{\mathrm{w}}^2)$ .
Further, note that
$$
\begin{array}{l} \left\| w ^ {r} - x ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} + \left\| z ^ {r} - y ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \\ = \| w ^ {r} - x ^ {r + 1} + x ^ {r + 1} - x ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} + \| z ^ {r} - y ^ {r + 1} + y ^ {r + 1} - y ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \\ \leq \frac {2}{\beta^ {2}} \left(\| w ^ {r + 1} - w ^ {r} \| ^ {2} + \| z ^ {r + 1} - z ^ {r} \| ^ {2}\right) + 2 \left(\| x ^ {r + 1} - x ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} + \| y ^ {r + 1} - y ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2}\right) \\ = \frac {2}{\beta^ {2}} \left(\| w ^ {r + 1} - w ^ {r} \| ^ {2} + \| z ^ {r + 1} - z ^ {r} \| ^ {2}\right) \\ + 2 \| x ^ {r + 1} - x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) + x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - x ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \\ + 2 \| y ^ {r + 1} - y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) + y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - y ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} (162) \\ \leq \frac {2}{\beta^ {2}} \left(\| w ^ {r + 1} - w ^ {r} \| ^ {2} + \| z ^ {r + 1} - z ^ {r} \| ^ {2}\right) \\ + 4 \| x ^ {r + 1} - x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \| ^ {2} + 4 \| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - x ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \\ + 4 \| y ^ {r + 1} - y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \| ^ {2} + 4 \| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - y ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2}. (163) \\ \end{array}
$$
By following the steps from (136) to (138), we have
$$
\begin{array}{l} \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} + \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \right\| ^ {2} \\ \leq 2 \| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} + 2 \| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \\ + 2 \left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) \right\| ^ {2} \\ + 2 \left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) \right\| ^ {2} (164) \\ \leq 2 \| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} + 2 \| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda_ {+} ^ {r + 1} \left(w ^ {r}, z ^ {r}\right)\right) - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r}\right) \| ^ {2} \\ + 4 \sigma_ {2} ^ {2} \tau^ {2} \left\| g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - \delta \right\| ^ {2}. (165) \\ \end{array}
$$
Combining (163) and (165) yields