SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on the assumptions listed in Table 2, we have that  $f, g$  are gradient Lipschitz continuous. Further, it has been shown in [34, Lemma A5] that  $\nabla g^{*}(x) = \nabla_{x}g(x,y^{*})$  for any  $y^{*}\in S(x)$  and  $g^{*}(x)$  is Lipschitz smooth with modulus  $L_{g} + L_{g}^{2} / (2\mu_{g})$ . Therefore, the Lagrangian  $\mathcal{L}(x,y;\lambda)$  is gradient Lipschitz continuous with parameter
$$
L \triangleq L _ {f} + \lambda \left(2 L _ {g} + \frac {L _ {g} ^ {2}}{2 \mu_ {g}}\right). \tag {25}
$$
Subsequently, function  $K(x,y,w,z;\lambda)$  is strongly convex of  $x$  and  $y$  with parameter  $p - L$  and gradient Lipschitz continuous with parameter
$$
L _ {K} \triangleq L + p. \tag {26}
$$
Given these property and the assumptions listed in Table 2, we can obtain the following primal error bounds that have been studied in [33, Lemma 5], [32, Lemma 3.5 and Lemma 3.10] and [36, Lemma
Table 2: Summary of Definitions. ("Lips": Lipschitz; "grad": gradient; "const." constant; "opt." optimal; "·" represents the gradient is taken w.r.t. either  $x$  or  $y$  ; "[  $x;y\rbrack$  " symbolizes the concatenation of  $x$  and  $y$  ; PEB: primal error bound; DEB: dual error bound.)
A1	Definition	Annotation
Lf	∇.f(x,y) - ∇.f(x',y) || ≤ Lf|| x - x' ||	grad. Lips. const. of f(x,y) w.r.t x
Lg	∇.f(x,y) - ∇.f(x,y') || ≤ Lf|| y - y' ||	grad. Lips. const. of f(x,y) w.r.t y
∇yg(x,y) - ∇yg(x,y') || ≤ Lg|| y - y' ||	grad. Lips. const. of g w.r.t. y
A2	Definition	Annotation
X	closed and bounded	imply (grad.) Lips. of f, g w.r.t. x
A5	Definition	Annotation
μg	∇yg(x,y)||2 ≥ 2μg(g(x,y) - g*(x))	PL condition
Parameter	Definition	Annotation
ds(x)(y)	miny'∈S(x) || y - y'||	min distance from y to S(x)
DS	maxu d2S(x)(u) (cf. (33))	upper bound of d2S(x)(u)
D(w,z;λ)	minx,y K(x,y,w,z;λ)	opt. solution of D(w,z;λ) w.r.t. x
x*(w,z;λ)	arg minx,y K(x,y,w,z;λ)	opt. solution of D(w,z;λ) w.r.t. y
y*(w,z;λ)	arg minx,y K(x,y,w,z;λ)	opt. solution of D(w,z;λ) w.r.t. y
P(w,z)	minx,y max0≤λ≤λ K(x,y,w,z;λ)	opt. solution of P(w,z) w.r.t. x
x*(w,z)	arg minx,y max0≤λ≤λ K(x,y,w,z;λ)	opt. solution of P(w,z) w.r.t. y
y*(w,z)	arg minx,y max0≤λ≤λ K(x,y,w,z;λ)	abbreviation of w,z
v	v ≡ (w,z)
Notation	Definition	Annotation
ℓf	|f(x,y) - f(x,y')| ≤ ℓf||y - y'||	Lips. const. of f(x,y) w.r.t. y
Lg	||∇.g(x,y) - ∇.g(x',y)|| ≤ Lg||x - x'||	grad. Lips. const. of g(x,y)
ℓg	||g(x,y) - g(x,y')|| ≤ ℓg||y - y'||	Lips. const. of g(x,y) w.r.t. y
||g(x,y) - g(x',y)|| ≤ ℓg||x - x'||	Lips. const. of g(x,y) w.r.t. x
ℓg'	||g*(x) - g*(x')|| ≤ ℓg'||x - x'|| (cf. [34])	Lips. const. of g*(x) w.r.t. x
L	||∇L(x,y;λ) - ∇L(x',y';λ)||	grad. Lips. const. of L() w.r.t. x,y
≤ L||[x;y] - [x';y'] || (cf. (25))
LK	L + p (cf. (26))	grad. Lips. const. of K() w.r.t. x,y
σ1	p(p-L)-1(cf. (27a))	const. of PEB w.r.t. v
σ2	(p+L)(p-L)-1(cf. (28a))	const. of PEB w.r.t. λ
σ3	(p-L)-1(cf. (29a))	const. of PEB w.r.t. x' or y'
σweak	(1+τ(2ℓg+ℓg')σ2)(τ(p-L))-1(cf. (80))	const. of weak DEB
σaux	(Lf + p + ΛLg)/√2μgδ (cf. (94))	const. of auxiliary DEB
σstrong	σweakσaux (cf. (100))	const. of strong DEB
B.2]. To be more specific, from [36, Lemma B.2] we have
$$
\left\| y ^ {*} (w, z; \lambda) - y ^ {*} \left(w, z ^ {\prime}; \lambda\right) \right\| \leq \underbrace {\frac {p}{p - L}} _ {\triangleq \sigma_ {1}} \| z - z ^ {\prime} \|, \tag {27a}
$$
$$
\left\| x ^ {*} \left(w, z; \lambda\right) - x ^ {*} \left(w ^ {\prime}, z; \lambda\right) \right\| \leq \sigma_ {1} \| w - w ^ {\prime} \|, \tag {27b}
$$
$$
\left\| \bar {y} ^ {*} (w, z) - \bar {y} ^ {*} \left(w, z ^ {\prime}\right) \right\| \leq \sigma_ {1} \| z - z ^ {\prime} \|, \tag {27c}
$$
$$
\left\| \bar {x} ^ {*} (w, z) - \bar {x} ^ {*} \left(w ^ {\prime}, z\right) \right\| \leq \sigma_ {1} \| w - w ^ {\prime} \|. \tag {27d}
$$
Similarly, following [36, Lemma B.2], we can also have
$$
\left\| y ^ {*} (w, z; \lambda) - y ^ {*} (w, z; \lambda^ {\prime}) \right\| \leq \underbrace {\frac {p + L}{p - L}} _ {\triangleq_ {\sigma_ {2}}} \| \lambda - \lambda^ {\prime} \|, \tag {28a}
$$
$$
\left\| x ^ {*} (w, z; \lambda) - x ^ {*} (w, z; \lambda^ {\prime}) \right\| \leq \sigma_ {2} \| \lambda - \lambda^ {\prime} \|, \tag {28b}
$$
and from [32, Lemma 3.10] or [33, Lemma 5] we can directly get
$$
\left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - y ^ {r} \right\| \leq \frac {1}{\alpha} \underbrace {\frac {1}{p - L}} _ {\triangleq \sigma_ {3}} \left\| y ^ {r + 1} - y ^ {r} \right\|, \tag {29a}
$$
$$
\left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - x ^ {r} \right\| \leq \frac {\sigma_ {3}}{\eta} \left\| x ^ {r + 1} - x ^ {r} \right\|, \tag {29b}
$$
$$
\left\| y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - y ^ {r + 1} \right\| \leq \frac {1 + \alpha (p - L)}{\alpha (p - L)} \left\| y ^ {r + 1} - y ^ {r} \right\|, \tag {29c}
$$
$$
\left\| x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - x ^ {r + 1} \right\| \leq \frac {1 + \eta (p - L)}{\eta (p - L)} \| x ^ {r + 1} - x ^ {r} \|, \tag {29d}
$$
where the primal error bounds (29b) and (29d) hold as  $\widehat{K}$  (which is used for updating  $x$ ) is also  $(p - L)$ -strongly convex and  $(p + L)$ -Lipschitz smooth.
A.4 Outline of the Proof
We will begin by presenting the contraction property of the block-  $u$  update in Lemma 4. Next, we will show the primal descent in Lemma 5, dual ascent in Lemma 6, and proximal descent in Lemma 7 by following the update rule of SLM, which will be used to construct the potential function. However, these lemmas alone are insufficient to establish the decrease of the potential function to achieve the  $\mathcal{O}(1 / \epsilon)$  convergence rate, as mentioned in the main text. Hence, we prove the strong dual error bound Lemma 3 by combining the weak dual error bound presented in Lemma 2 and the key auxiliary lemma in Lemma 8, which ensures the homogeneity of the dual error bound.
With the strong dual error bound established, we can demonstrate the sufficient decrease of the potential function in Lemma 9. This, in turn, leads to the final convergence results shown in Section Section D. Throughout the process of proving the lemmas and theorems, we initially assume the boundedness of the dual variable, as well as  $y^{r}, \bar{y}^{*}(w^{r}, z^{r}), y^{*}(w^{r}, z^{r}; \lambda^{r+1})$ . We then verify these assumptions through mathematical induction, which are shown in Lemma 11 and Lemma 12.
B Convergence Analysis
We now present the proofs, related results, and technical details that establish the lemmas and theorems of our convergence analysis.
We first show that the contraction property of  $u$ -update for the value function.
B.1 Contraction of Auxiliary Variable  $u$
Under A1, A2, and A5. Assume that the sequence  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r, \forall r\}$  is generated by SLM. When
$$
\gamma \leq \min  \left\{\frac {1}{L _ {g}}, \frac {1}{2 \mu_ {g}} \right\}, \tag {30}
$$
$$
d _ {\mathcal {S} \left(x ^ {r}\right)} ^ {2} \left(u ^ {r + 1}\right) \leq \frac {2}{\mu_ {g}} \left(1 - 2 \gamma \mu_ {g}\right) ^ {T ^ {r}} \left(g \left(x ^ {r}, u _ {0} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right)\right). \tag {31}
$$
Further, we can obtain
$$
d _ {\mathcal {S} \left(x ^ {r}\right)} ^ {2} \left(u ^ {r + 1}\right) \leq \left(1 - 2 \gamma \mu_ {g}\right) ^ {T ^ {r}} D _ {\mathcal {S}} \quad \forall r \tag {32}
$$
$$
D _ {\mathcal {S}} \triangleq 2 \mu_ {g} ^ {- 1} \left(\max  g (x, u _ {0}) - \min  g ^ {*} (x)\right) \forall x \in \mathcal {X}, u _ {0} \in \mathcal {B} (R). \tag {33}
$$
Proof. According to the gradient Lipschitz continuity of function  $g(x,y)$  w.r.t.  $y$  (cf. A1), we have
$$
g \left(x ^ {r}, u _ {t + 1} ^ {r}\right) \leq g \left(x ^ {r}, u _ {t} ^ {r}\right) - \gamma \left\langle \nabla_ {u} g \left(x ^ {r}, u _ {t} ^ {r}\right), u _ {t + 1} ^ {r} - u _ {t} ^ {r} \right\rangle + \frac {L _ {g} \gamma^ {2}}{2} \| \nabla_ {u} g \left(x ^ {r}, u _ {t} ^ {r}\right) \| ^ {2} \tag {34}
$$
$$
\stackrel {(a)} {\leq} g \left(x ^ {r}, u _ {t} ^ {r}\right) - \gamma \| \nabla_ {u} g \left(x, u _ {t} ^ {r}\right) \| ^ {2} + \frac {L _ {g} \gamma^ {2}}{2} \| \nabla_ {u} g \left(x ^ {r}, u _ {t} ^ {r}\right) \| ^ {2} \tag {35}
$$
$$
\stackrel {(b)} {\leq} g \left(x ^ {r}, u _ {t} ^ {r}\right) - \frac {\gamma}{2} \| \nabla_ {u} g \left(x, u _ {t} ^ {r}\right) \| ^ {2} \tag {36}
$$
where in  $(a)$  we apply (5),  $(b)$  holds when  $\gamma \leq 1 / L_{g}$ .
Under A5, we have
$$
g \left(x ^ {r}, u _ {t + 1} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right) \leq g \left(x ^ {r}, u _ {t} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right) - 2 \gamma \mu_ {g} \left(g \left(x ^ {r}, u _ {t} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right)\right) \tag {37}
$$
$$
\leq \left(1 - 2 \gamma \mu_ {g}\right) \left(g \left(x ^ {r}, u _ {t} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right)\right). \tag {38}
$$
From [35, Theorem 2], we know that the PL condition implies the error bound with the same constant, i.e.,  $d_{S(x^r)}^2 (u_t^r) \leq 2 / \mu_g(g(x^r,u_t^r) - g^* (x^r))$ , which yields
$$
d _ {\mathcal {S} (x ^ {r})} ^ {2} \left(u ^ {r + 1}\right) \stackrel {(a)} {=} d _ {\mathcal {S} (x ^ {r})} ^ {2} \left(u _ {T ^ {r}} ^ {r}\right)
$$
$$
\leq \frac {2}{\mu_ {g}} \left(g \left(x ^ {r}, u _ {t + 1} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right)\right) \tag {39}
$$
$$
\leq \frac {2}{\mu_ {g}} \left(1 - 2 \gamma \mu_ {g}\right) ^ {T ^ {r}} \left(g \left(x ^ {r}, u _ {0} ^ {r}\right) - g ^ {*} \left(x ^ {r}\right)\right) \tag {40}
$$
where  $(a)$  holds due to the line 7 of Algorithm 1. Combining the facts that  $\mathcal{X}$  is compact and  $u^r$  is initialized within a ball gives this lemma directly.
B.2 Descent Lemmas and Dual Ascent
B.2.1 Primal Descent Lemma
Lemma 5.
(Primal Descent) Under A1-A5, suppose that the sequence  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r, \forall r\}$  is generated by SLM. When
$$
0 <   \eta , \alpha \leq \frac {1}{L _ {K}}, \quad 0 <   \beta \leq 1, \quad a n d T ^ {r} \geq - 2 \log_ {(1 - 2 \gamma \mu_ {g})} (\lambda^ {r + 1} \sqrt {\eta} r), \tag {41}
$$
then the primal descent inequality holds, namely,
$$
\begin{array}{l} K \left(x ^ {r + 1}, y ^ {r + 1}, w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - K \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r}\right) \\ \leq - \frac {1}{4 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2} + \langle g (x ^ {r}, y ^ {r}) - g ^ {*} (x ^ {r}) - \delta , \lambda^ {r + 1} - \lambda^ {r} \rangle \\ - \frac {1}{2 \alpha} \| y ^ {r + 1} - y ^ {r} \| ^ {2} - \frac {p}{2 \beta} \| z ^ {r + 1} - z ^ {r} \| ^ {2} - \frac {p}{2 \beta} \| w ^ {r + 1} - w ^ {r} \| ^ {2} + \frac {\mu_ {g} L _ {g} ^ {2} D _ {\mathcal {S}}}{r ^ {2}}. \tag {42} \\ \end{array}
$$
Proof.  $x$ -update: First, recall
$$
\begin{array}{l} \nabla_ {x} \widehat {K} (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}) \\ \triangleq \nabla_ {x} f \left(x ^ {r}, y ^ {r + 1}\right) + \lambda^ {r + 1} \left(\nabla_ {x} g \left(x ^ {r}, y ^ {r + 1}\right) - \nabla_ {x} g \left(x ^ {r}, u ^ {r + 1}\right)\right) + p \left(x ^ {r} - w ^ {r}\right). \tag {43} \\ \end{array}
$$
From (8a), one step of the projected gradient descent step gives
$$
K \left(x ^ {r + 1}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - K \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right)
$$
$$
\begin{array}{l} \stackrel {(a)} {\leq} \left\langle \nabla K \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), x ^ {r + 1} - x ^ {r} \right\rangle + \frac {L _ {K}}{2} \| x ^ {r + 1} - x ^ {r} \| ^ {2} (44) \\ \leq \langle \nabla \widehat {K} (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}), x ^ {r + 1} - x ^ {r} \rangle + \frac {L _ {K}}{2} \| x ^ {r + 1} - x ^ {r} \| ^ {2} \\ + \left\langle \nabla K \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - \nabla \widehat {K} \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}\right), x ^ {r + 1} - x ^ {r} \right\rangle (45) \\ \end{array}
$$
where  $(a)$  is because gradient Lipschitz continuity of  $K(x,y,w,z;\lambda)$  is with parameter  $L_{K}$ .
From the optimality condition of (8a), we have
$$
\langle \nabla_ {x} \widehat {K} \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}\right), x ^ {r + 1} - x ^ {r} \rangle \leq - \frac {1}{\eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2}. \tag {46}
$$
Regarding the last term at the right hand side (RHS) of (45), we have
$$
\begin{array}{l} \langle \nabla K (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}) - \nabla \widehat {K} (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}), x ^ {r + 1} - x ^ {r} \rangle \\ \leq \eta \| \nabla K (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}) - \nabla \widehat {K} (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}) \| ^ {2} + \frac {1}{4 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2} \\ \end{array}
$$
where we apply Young's inequality with parameter 2.
For the first term at the RHS of the above inequality, we can further have
$$
\begin{array}{l} \| \nabla K (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}) - \nabla \widehat {K} (x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}, u ^ {r + 1}; \lambda^ {r + 1}) \| ^ {2} \\ \leq \left(\lambda^ {r + 1}\right) ^ {2} \left\| \nabla g ^ {*} \left(x ^ {r}\right) - \nabla g \left(x ^ {r}, u ^ {r + 1}\right) \right\| ^ {2} (47) \\ \leq \left(\lambda^ {r + 1}\right) ^ {2} L _ {g} ^ {2} d _ {\mathcal {S} (x ^ {r})} ^ {2} \left(u ^ {r + 1}\right) (48) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(3 2)} {\leq} \left(\lambda^ {r + 1}\right) ^ {2} L _ {g} ^ {2} \mu_ {g} \left(1 - 2 \gamma \mu_ {g}\right) ^ {T ^ {r}} D _ {\mathcal {S}} (49) \\ \stackrel {(a)} {\leq} \frac {\mu_ {g} L _ {g} ^ {2} D _ {\mathcal {S}}}{\eta r ^ {2}} (50) \\ \end{array}
$$
where in (a) we take  $T^r \geq -2\log_{(1 - 2\gamma \mu_g)}(\lambda^{r + 1}\sqrt{\eta} r)$ .
Substituting (46) and (50) back to (45) gives
$$
K \left(x ^ {r + 1}, y ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - K \left(x ^ {r}, y ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) \leq - \frac {1}{4 \eta} \| x ^ {r + 1} - x ^ {r} \| ^ {2} + \frac {\mu_ {g} L _ {g} ^ {2} D _ {\mathcal {S}}}{r ^ {2}} \tag {51}
$$
where we select  $\eta \leq 1 / L_K$ .
$w$ -update: From (6), we have
$$
\begin{array}{l} K \left(x ^ {r + 1}, y ^ {r + 1}, w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - K \left(x ^ {r + 1}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \\ = \frac {p}{2} \left(\| x ^ {r + 1} - w ^ {r + 1} \| ^ {2} - \| x ^ {r + 1} - w ^ {r} \| ^ {2}\right) (52) \\ = \frac {p}{2} \left\langle w ^ {r} - w ^ {r + 1}, x ^ {r + 1} - w ^ {r + 1} + x ^ {r + 1} - w ^ {r} \right\rangle (53) \\ \end{array}
$$
$$
\stackrel {(a)} {\leq} - \frac {p}{2 \beta} \| w ^ {r + 1} - w ^ {r} \| ^ {2} \tag {54}
$$
where  $(a)$  holds due to
$$
\begin{array}{l} \left\langle w ^ {r} - w ^ {r + 1}, x ^ {r + 1} - w ^ {r + 1} + x ^ {r + 1} - w ^ {r} \right\rangle \\ = \left\langle w ^ {r} - w ^ {r + 1}, x ^ {r + 1} - w ^ {r} + w ^ {r} - w ^ {r + 1} + x ^ {r + 1} - w ^ {r} \right\rangle (55) \\ = \left(1 - \frac {2}{\beta}\right) \| w ^ {r + 1} - w ^ {r} \| ^ {2} (56) \\ \end{array}
$$
and (8b) for  $\beta \leq 1$
$\lambda$  -update: According to the dual variable update (7a), we have
$$
K \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - K \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r}\right) = \langle g \left(x ^ {r}, y ^ {r}\right) - g ^ {*} \left(x ^ {r}\right) - \delta , \lambda^ {r + 1} - \lambda^ {r} \rangle . \tag {57}
$$
$y$ -update: The update of  $y$  shown in (7b) is the standard gradient descent, which gives
$$
\begin{array}{l} K \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - K \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \\ \leq \left\langle \nabla K \left(x ^ {r}, y ^ {r}, w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {r + 1} - y ^ {r} \right\rangle + \frac {L _ {K}}{2} \| y ^ {r + 1} - y ^ {r} \| ^ {2} (58) \\ \leq - \frac {1}{2 \alpha} \| y ^ {r + 1} - y ^ {r} \| ^ {2} (59) \\ \end{array}
$$
where  $\alpha \leq 1 / L_K$
$z$ -update: Similar to the  $w$ -update. From (7c), we have
$$
K \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - K \left(x ^ {r}, y ^ {r + 1}, w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \leq - \frac {p}{2 \beta} \| z ^ {r + 1} - z ^ {r} \| ^ {2}. \tag {60}
$$
B.2.2 Dual Ascent Lemma
Lemma 6.
(Dual Ascent) Under A1-A5, suppose that the sequence  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r, \forall r\}$  is generated by SLM. When  $p > L$ , the dual ascent inequality holds, namely,
$$
\begin{array}{l} D \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - D \left(w ^ {r}, z ^ {r}; \lambda^ {r}\right) \\ \geq \left\langle \lambda^ {r + 1} - \lambda^ {r}, g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \delta\right) \right\rangle \\ + \frac {p}{2} \left\langle z ^ {r + 1} - z ^ {r}, z ^ {r + 1} + z ^ {r} - 2 y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \right\rangle \\ + \frac {p}{2} \langle w ^ {r + 1} - w ^ {r}, w ^ {r + 1} + w ^ {r} - 2 x ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) \rangle . \tag {61} \\ \end{array}
$$
$$
K (x, y, w, z; \lambda) \triangleq \mathcal {L} (x, y; \lambda) + \frac {p}{2} \| x - w \| ^ {2} + \frac {p}{2} \| y - z \| ^ {2}. \tag {62}
$$
Then, we can have
$$
\begin{array}{l} D \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - D \left(w ^ {r}, z ^ {r}; \lambda^ {r}\right) \\ = K \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \\ - K \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r}\right), w ^ {r}, z ^ {r}; \lambda^ {r}\right) \tag {63} \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a)} {\geq} K \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \\ - K \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r}; \lambda^ {r}\right) (64) \\ = \left\langle \lambda^ {r + 1} - \lambda^ {r}, g \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right)\right) - g ^ {*} \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) - \delta\right) \right\rangle (65) \\ \end{array}
$$
where in (a) we use the definition of  $y^{*}(w^{r},z^{r};\lambda^{r + 1})$  for  $p > L$
$$
\begin{array}{l} D \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - D \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \\ = K \left(x ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \\ - K \left(x ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \tag {66} \\ \end{array}
$$
$$
\begin{array}{l} \geq K \left(x ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \\ - K \left(x ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r}; \lambda^ {r + 1}\right) \tag {67} \\ \end{array}
$$
$$
\begin{array}{l} = \frac {p}{2} \left(\| y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - z ^ {r + 1} \| ^ {2} - \| y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) - z ^ {r} \| ^ {2}\right) (68) \\ = \frac {p}{2} \left\langle z ^ {r + 1} - z ^ {r}, z ^ {r + 1} + z ^ {r} - 2 y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \right\rangle , (69) \\ \end{array}
$$
$$
\begin{array}{l} D \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - D \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \\ = K \left(x ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right), w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) \\ - K \left(x ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \tag {70} \\ \end{array}
$$
$$
\begin{array}{l} \geq K \left(x ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right), w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) \\ - K \left(x ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right), y ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right), w ^ {r}, z ^ {r + 1}; \lambda^ {r + 1}\right) \tag {71} \\ \end{array}
$$
$$
\begin{array}{l} = \frac {p}{2} \left(\| x ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - w ^ {r + 1} \| ^ {2} - \| x ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) - w ^ {r} \| ^ {2}\right) (72) \\ = \frac {p}{2} \left\langle w ^ {r + 1} - w ^ {r}, w ^ {r + 1} + w ^ {r} - 2 x ^ {*} \left(w ^ {r + 1}, z ^ {r + 1}; \lambda^ {r + 1}\right) \right\rangle , (73) \\ \end{array}
$$
B.2.3 Proximal Descent Lemma
(Proximal Descent) Under A1-A5, suppose that the sequence  $\{x^r, y^r, w^r, z^r, u^r, \lambda^r, \forall r\}$  is generated by SLM. Assume that  $y^r$  is bounded and  $p > L$ , then the proximal descent inequality holds, namely,
$$
\begin{array}{l} P \left(w ^ {r + 1}, z ^ {r + 1}\right) - P \left(w ^ {r}, z ^ {r}\right) \\ \leq p \left(z ^ {r + 1} - z ^ {r}\right) ^ {T} \left(z ^ {r} - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right)\right) + p \left(w ^ {r + 1} - w ^ {r}\right) ^ {T} \left(w ^ {r} - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r + 1}\right)\right) \\ + \frac {p}{2} \left(\frac {p}{p - L} + 1\right) \left(\| z ^ {r + 1} - z ^ {r} \| ^ {2} + \| w ^ {r + 1} - w ^ {r} \| ^ {2}\right). \tag {74} \\ \end{array}
$$
Proof. First, note that  $K(x,y,w,z;\lambda)$  is strongly convex w.r.t.  $x$  and  $y$  jointly with parameter  $p - L$ . Under A2, A5 and the assumption that  $y^{r}$  is bounded, we can obtain that  $\nabla_{z}P(w^{r},z^{r}) = p(z^{r} - \bar{y}^{*}(w^{r},z^{r}))$  by applying the Danskin's theorem in the convex analysis [48]. Then, using the primal error bound (27c), we can show that  $\nabla_{z}P(x^{r})$  has a Lipschitz constant, i.e.,
$$
\left\| \nabla P \left(w ^ {r}, z ^ {r + 1}\right) - \nabla P \left(w ^ {r}, z ^ {r}\right) \right\| \leq p \left(\frac {p}{p - L} + 1\right) \| z ^ {r + 1} - z ^ {r} \|. \tag {75}
$$
Therefore, it is straightforward that
$$
P \left(w ^ {r}, z ^ {r + 1}\right) - P \left(w ^ {r}, z ^ {r}\right) \leq p \left(z ^ {r + 1} - z ^ {r}\right) ^ {T} \left(z ^ {r} - \bar {y} ^ {*} \left(w ^ {r}, z ^ {r}\right)\right) + \frac {p}{2} \left(\frac {p}{p - L} + 1\right) \| z ^ {r + 1} - z ^ {r} \| ^ {2}. \tag {76}
$$
$$
\left\| \nabla_ {w} P \left(w ^ {r + 1}, z ^ {r + 1}\right) - \nabla_ {w} P \left(w ^ {r}, z ^ {r + 1}\right) \right\| \leq p \left(\frac {p}{p - L} + 1\right) \| w ^ {r + 1} - w ^ {r} \|, \tag {77}
$$
$$
\begin{array}{l} P \left(w ^ {r + 1}, z ^ {r + 1}\right) - P \left(w ^ {r}, z ^ {r + 1}\right) \\ \leq p \left(w ^ {r + 1} - w ^ {r}\right) ^ {T} \left(w ^ {r} - \bar {x} ^ {*} \left(w ^ {r}, z ^ {r + 1}\right)\right) + \frac {p}{2} \left(\frac {p}{p - L} + 1\right) \| w ^ {r + 1} - w ^ {r} \| ^ {2}. \tag {78} \\ \end{array}
$$
B.3 Dual Error Bounds
B.3.1 Proof of Weak Dual Error Bound (Lemma 2)
Lemma 2 (Formal)
Under A1-A5, suppose that  $\lambda, \bar{y}^{*}(w,z)$  are bounded and  $p > L$ , then the weak error bound holds, namely,
$$
\begin{array}{l} \left\| y ^ {*} (w, z; \lambda_ {+} (w, z)) - \bar {y} ^ {*} (w, z) \right\| ^ {2} + \left\| x ^ {*} (w, z; \lambda_ {+} (w, z)) - \bar {x} ^ {*} (w, z) \right\| ^ {2} \\ \leq \sigma_ {\text {w e a k}} \| \lambda - \lambda_ {+} (w, z) \| \| \lambda (w, z) - \lambda_ {+} (w, z) \| \tag {79} \\ \end{array}
$$