SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hence, Adding (11) and (13) yields:
$$
\begin{array}{l} \mathbb {E} \left[ F \left(\boldsymbol {\alpha} ^ {t + 1}, \mathbf {w} ^ {t + 1}\right) - F \left(\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}\right) \right] \\ \geq \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {L ^ {2} \eta}{2}\right) \mathbb {E} \| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \| ^ {2} - \left(1 6 (1 - \beta) ^ {2} L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {L ^ {2} \eta}{2}\right) \mathbb {E} \| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \| ^ {2} \\ + \left(\frac {\mu}{2} - \frac {3}{2 \eta} - \frac {1}{4 \eta (1 - \beta) ^ {2}}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} + \left(\mu - \frac {1}{2 \eta} - \frac {\eta L ^ {2}}{2}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} \\ - \sum_ {j = 1} ^ {N} (1 - \beta) ^ {2} \left(4 \alpha^ {t} (j) L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right) \right\| ^ {2} \\ - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B}, \\ \end{array}
$$
Lemma 8. For Algorithm 1, under the assumptions of Theorem 2, define the following auxiliary quantity  $\hat{F}^{t + 1}$ :
$$
\begin{array}{l} \hat {F} ^ {t + 1} := F \left(\boldsymbol {\alpha} ^ {t + 1}, \mathbf {w} ^ {t + 1}\right) + s ^ {t + 1} \\ \left. - \left(\frac {1}{4 \gamma} + 4 L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {\eta L ^ {2}}{2} + \frac {1 9 2 N G _ {g} ^ {2}}{\mu \eta \beta} + \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta}\right) \| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \| ^ {2} \right. \\ \left. - \left(\frac {1}{8 \gamma} + \frac {9 6 N G _ {g} ^ {2}}{\mu \eta \beta}\right) \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \left(\frac {7}{2 \eta} + \mu - \frac {\eta L ^ {2}}{2}\right) \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \right. \\ \end{array}
$$
$$
s ^ {t + 1} := - \frac {2}{\eta^ {2} \mu} \| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \| ^ {2}.
$$
Then the following statement holds:
$$
\begin{array}{l} \mathbb {E} \left[ \hat {F} ^ {t + 1} - \hat {F} ^ {t} \right] \geq C _ {1} \mathbb {E} \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} \\ + C _ {2} \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \\ - (1 - \beta) ^ {2} \left(4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right\| ^ {2} \\ - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B} - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}, \\ \end{array}
$$
$$
C _ {1} = \frac {1}{4 \gamma} - \frac {L}{2} - \frac {3}{2} \eta L ^ {2} - 1 6 L _ {g} ^ {2} G _ {f} ^ {4} \gamma - \frac {1 9 2 N G _ {g} ^ {2} G _ {f} ^ {2}}{\mu^ {2} \eta \beta} - \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta},
$$
$$
C _ {2} = \frac {1}{\eta} + \frac {3}{2} \mu - \frac {\eta L ^ {2}}{2} - \frac {1}{4 \eta (1 - \beta) ^ {2}} - \frac {2 L ^ {2}}{\mu}.
$$
Proof. According to (6) and (7):
$$
\frac {1}{\eta} \left\langle \Delta^ {t + 1}, \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t + 1} \right\rangle \geq \left\langle \mathbf {g} _ {\alpha} ^ {t} - \mathbf {g} _ {\alpha} ^ {t - 1}, \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\rangle .
$$
If we define  $\mathbf{g}_{\alpha}(z,\pmb {\alpha}) = [g(z_1),\dots,g(z_N)] + 2C\mathbf{M}\pmb {\alpha}.$
$$
\begin{array}{l} \frac {1}{\eta} \left\langle \Delta^ {t + 1}, \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t + 1} \right\rangle \geq \left\langle \mathbf {g} _ {\alpha} ^ {t} - \mathbf {g} _ {\alpha} \left(z ^ {t}, \boldsymbol {\alpha} ^ {t}\right), \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\rangle \\ + \left\langle \mathbf {g} _ {\alpha} \left(z ^ {t}, \boldsymbol {\alpha} ^ {t}\right) - \mathbf {g} _ {\alpha} ^ {t - 1}, \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\rangle \\ \geq \left\langle \mathbf {g} _ {\alpha} ^ {t} - \mathbf {g} _ {\alpha} \left(z ^ {t}, \boldsymbol {\alpha} ^ {t}\right), \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\rangle \\ + \left\langle \mathbf {g} _ {\alpha} (z ^ {t}, \boldsymbol {\alpha} ^ {t}) - \mathbf {g} _ {\alpha} ^ {t - 1}, \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} - (\boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t}) \right\rangle \\ + \left\langle \mathbf {g} _ {\alpha} \left(z ^ {t}, \boldsymbol {\alpha} ^ {t}\right) - \mathbf {g} _ {\alpha} ^ {t - 1}, \left(\boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1}\right) \right\rangle \\ \geq - \frac {G _ {g} ^ {2}}{\mu} \sum_ {j = 1} ^ {N} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2} - \frac {\mu}{4} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \\ - \frac {\eta L ^ {2}}{2} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} - \frac {1}{2 \eta} \left\| \Delta^ {t + 1} \right\| ^ {2} + \mu \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
Since  $\frac{1}{\eta}\left\langle \Delta^{t + 1},\pmb{\alpha}^t -\pmb{\alpha}^{t + 1}\right\rangle = \frac{1}{2\eta}\left\| \pmb {\alpha}^t -\pmb{\alpha}^{t - 1}\right\| ^2 -\frac{1}{2\eta}\left\| \pmb {\alpha}^{t + 1} - \pmb {\alpha}^t\right\| ^2 -\frac{1}{2\eta}\left\| \Delta^{t + 1}\right\| ^2$  Hence we have:
$$
\begin{array}{l} \frac {1}{2 \eta} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} - \frac {1}{2 \eta} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \geq - \frac {G _ {g} ^ {2}}{\mu} \sum_ {j = 1} ^ {N} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2} - \frac {\mu}{4} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \\ - \frac {\eta L ^ {2}}{2} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} + \mu \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} \\ \end{array}
$$
Multiplying both sides with  $\frac{4}{\mu\eta}$  yields:
$$
\begin{array}{l} \frac {2}{\mu \eta^ {2}} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} - \frac {2}{\mu \eta^ {2}} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \geq - \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2} - \frac {1}{\eta} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \\ - \frac {2 L ^ {2}}{\mu} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} + \frac {4}{\eta} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2}, \tag {14} \\ \end{array}
$$
and so, putting all together,
$$
\begin{array}{l} \mathbb {E} \left[ F \left(\boldsymbol {\alpha} ^ {t + 1}, \mathbf {w} ^ {t + 1}\right) - F \left(\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}\right) \right] \\ \geq \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {L ^ {2} \eta}{2}\right) \mathbb {E} \| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \| ^ {2} - \left(1 6 (1 - \beta) ^ {2} L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {L ^ {2} \eta}{2}\right) \mathbb {E} \| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \| ^ {2} \\ + \left(\frac {\mu}{2} - \frac {3}{2 \eta} - \frac {1}{4 \eta (1 - \beta) ^ {2}}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} + \left(\mu - \frac {1}{2 \eta} - \frac {\eta L ^ {2}}{2}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} \\ - \sum_ {j = 1} ^ {N} (1 - \beta) ^ {2} \left(4 \alpha^ {t} (j) L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\hat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right\| ^ {2} \\ - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B}. \\ \end{array}
$$
Now evoking Lemma 7 together with (14) yields:
$$
\begin{array}{l} \mathbb {E} \left[ F \left(\boldsymbol {\alpha} ^ {t + 1}, \mathbf {w} ^ {t + 1}\right) + s ^ {t + 1} - \left(F \left(\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}\right) + s ^ {t}\right) \right] \\ \geq \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {L ^ {2} \eta}{2}\right) \mathbb {E} \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \right\| ^ {2} - \left(1 6 (1 - \beta) ^ {2} L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {L ^ {2} \eta}{2}\right) \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} \\ + \left(\frac {\mu}{2} - \frac {3}{2 \eta} - \frac {1}{4 \eta (1 - \beta) ^ {2}} - \frac {1}{\eta}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} + \left(\mu - \frac {1}{2 \eta} - \frac {\eta L ^ {2}}{2} + \frac {4}{\eta} - \frac {2 L ^ {2}}{\mu}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} \\ - \sum_ {j = 1} ^ {N} (1 - \beta) ^ {2} \left(4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right) \right\| ^ {2} \\ - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B} - \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2}. \\ \end{array}
$$
Now we plug in Lemma 3 with the fact that  $\frac{1}{\beta} \geq 1$  and get:
$$
\begin{array}{l} \mathbb {E} \left[ F \left(\boldsymbol {\alpha} ^ {t + 1}, \mathbf {w} ^ {t + 1}\right) + h ^ {t + 1} - \left(F \left(\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}\right) + h ^ {t}\right) \right] \\ \geq \left(\frac {1}{2 \gamma} - \frac {L}{2} - \frac {L ^ {2} \eta}{2}\right) \mathbb {E} \| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \| ^ {2} \\ - \left(1 6 L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {L ^ {2} \eta}{2} + \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2}}{\mu^ {2} \eta \beta} + \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta}\right) \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} - \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2}}{\mu^ {2} \eta \beta} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} \\ + \left(\frac {\mu}{2} - \frac {5}{2 \eta} - \frac {1}{4 \eta (1 - \beta) ^ {2}}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} + \left(\mu + \frac {7}{2 \eta} - \frac {\eta L ^ {2}}{2} - \frac {2 L ^ {2}}{\mu}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t} - \boldsymbol {\alpha} ^ {t - 1} \right\| ^ {2} \\ \left. - (1 - \beta) ^ {2} \left(4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\hat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right\| ^ {2} \right. \\ - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B} - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
Recall our definition of potential function  $\hat{F}^{t + 1}$
$$
\begin{array}{l} \hat {F} ^ {t + 1} := F \left(\boldsymbol {\alpha} ^ {t + 1}, \mathbf {w} ^ {t + 1}\right) + h ^ {t + 1} - \left(\frac {1}{4 \gamma} + 1 6 L _ {g} ^ {2} G _ {f} ^ {4} \gamma + \frac {\eta L ^ {2}}{2} + \frac {1 9 2 N G _ {g} ^ {2} G _ {f} ^ {2}}{\mu^ {2} \eta \beta} + \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta}\right) \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \right\| ^ {2} \\ \left. \right. - \left(\frac {1}{8 \gamma} + \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2}}{\mu^ {2} \eta \beta}\right)\left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \left(\frac {7}{2 \eta} + \mu - \frac {\eta L ^ {2}}{2} - \frac {2 L ^ {2}}{\mu}\right)\left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2}. \\ \end{array}
$$
We conclude that:
$$
\begin{array}{l} \mathbb {E} \left[ \hat {F} ^ {t + 1} - \hat {F} ^ {t} \right] \geq \left(\frac {1}{4 \gamma} - \frac {L}{2} - \frac {3}{2} \eta L ^ {2} - 1 6 L _ {g} ^ {2} G _ {f} ^ {4} \gamma - \frac {1 9 2 N G _ {g} ^ {2} G _ {f} ^ {2}}{\mu^ {2} \eta \beta} - \frac {9 6 N G _ {g} ^ {2} G _ {f} ^ {2} \beta}{\mu^ {2} \eta}\right) \mathbb {E} \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \right\| ^ {2} \\ + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} \\ + \left(\frac {1}{\eta} + \frac {3}{2} \mu - \frac {\eta L ^ {2}}{2} - \frac {1}{4 \eta (1 - \beta) ^ {2}} - \frac {2 L ^ {2}}{\mu}\right) \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \\ - (1 - \beta) ^ {2} \left(4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right\| ^ {2} \\ \left. \right. - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B} - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
Lemma 9. For Algorithm 1, under the assumptions of Theorem 2, define the following auxiliary quantity  $\tilde{F}^{t + 1}$ :
$$
\tilde {F} ^ {t + 1} := \hat {F} ^ {t + 1} - \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - (f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})) \right\| ^ {2} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2}.
$$
Let  $C1, C2$  be defined in Lemma 8. If the following conditions hold
$$
\frac {1}{8 \gamma} - 4 \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} N + C _ {2} \eta^ {2} G _ {g} ^ {2} N ^ {2} + N\right) (1 - \beta) ^ {2} G _ {f} ^ {2} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \frac {4 8}{\beta} G _ {f} ^ {2} \geq 0, \tag {15}
$$
$$
1 - \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} + C _ {2} \eta^ {2} G _ {g} ^ {2} + 4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2} + 1\right) (1 - \beta) ^ {2} \geq 0, \tag {16}
$$
$$
\frac {1}{8 \gamma} - \frac {\left(1 - \frac {\beta}{2}\right) ^ {2}}{1 - \left(1 - \frac {\beta}{2}\right) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \frac {2 4}{\beta} G _ {f} ^ {2} \geq 0, \tag {17}
$$
then the following statement hold:
Proof. According to Lemma 8:
$$
\begin{array}{l} \mathbb {E} [ \hat {F} ^ {t + 1} - \hat {F} ^ {t} ] \\ \geq C _ {1} \mathbb {E} \left\| \mathbf {w} ^ {t + 1} - \mathbf {w} ^ {t} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} + C _ {2} \mathbb {E} \left\| \boldsymbol {\alpha} ^ {t + 1} - \boldsymbol {\alpha} ^ {t} \right\| ^ {2} \\ - (1 - \beta) ^ {2} \left(4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\hat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right\| ^ {2} \\ - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B} - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}, \\ \end{array}
$$
Now we plug in Lemma 5:
$$
\begin{array}{l} \mathbb {E} [ \hat {F} ^ {t + 1} - \hat {F} ^ {t} ] \\ \geq C _ {1} \left(\frac {1}{2} \gamma^ {2} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} - 2 \gamma^ {2} G _ {g} ^ {2} \frac {\sigma^ {2}}{B} - 4 \gamma^ {2} G _ {f} ^ {2} L _ {g} ^ {2} \sum_ {j = 1} ^ {N} \alpha^ {t} (j) \mathbb {E} \left\| z _ {j} ^ {t + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - \nabla f _ {j} (\mathbf {w} ^ {t})\right) \right\| ^ {2}\right) \\ + C _ {2} \left(\frac {1}{2} \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} - \eta^ {2} G _ {g} ^ {2} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})\right) \right\| ^ {2}\right) \\ - (1 - \beta) ^ {2} \left(4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} \left(\mathbf {w} ^ {t - 1}\right) - f _ {j} \left(\mathbf {w} ^ {t - 1}\right)\right) \right\| ^ {2} - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma\right) \frac {\sigma^ {2}}{B} \\ - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \hat {F} ^ {t + 1} - \hat {F} ^ {t} ] \geq \frac {C _ {1}}{2} \gamma^ {2} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} + C _ {2} \frac {1}{2} \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ - C _ {1} 4 \gamma^ {2} G _ {f} ^ {2} L _ {g} ^ {2} \sum_ {j = 1} ^ {N} \alpha^ {t} (j) \left((1 - \beta) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\hat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right) \right\| ^ {2} + 4 (1 - \beta) ^ {2} G _ {f} ^ {2} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + 2 \beta^ {2} \frac {\sigma^ {2}}{B}\right) \\ - C _ {2} \eta^ {2} G _ {g} ^ {2} \sum_ {j = 1} ^ {N} \left((1 - \beta) ^ {2} \mathbb {E} \left\| z _ {j} ^ {t} - (f _ {\widetilde {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})) \right\| ^ {2} + 4 (1 - \beta) ^ {2} G _ {f} ^ {2} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + 2 \beta^ {2} \frac {\sigma^ {2}}{B}\right) \\ - \left(1 - \beta\right) ^ {2} \left(4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widetilde {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right) \right\| ^ {2} \\ - \left(2 \gamma G _ {g} ^ {2} + 8 \beta^ {2} L _ {g} ^ {2} G _ {f} ^ {2} \gamma + 2 C _ {1} \gamma^ {2} G _ {g} ^ {2}\right) \frac {\sigma^ {2}}{B} \\ - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2}. \\ \end{array}
$$
Rearranging the terms yields:
$$
\begin{array}{l} \mathbb {E} [ \hat {F} ^ {t + 1} - \hat {F} ^ {t} ] \\ \geq \frac {C _ {1}}{2} \gamma^ {2} \mathbb {E} \left\| \nabla_ {\mathbf {w}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} + \frac {C _ {2}}{2} \eta^ {2} \mathbb {E} \left\| \nabla_ {\boldsymbol {\alpha}} G (\boldsymbol {\alpha} ^ {t}, \mathbf {w} ^ {t}) \right\| ^ {2} \\ \left. + \left(\frac {1}{8 \gamma} - (4 C _ {1} \gamma^ {2} L _ {g} ^ {2} + C _ {2} \eta^ {2} G _ {g} ^ {2} N) 4 (1 - \beta) ^ {2} G _ {f} ^ {2}\right) \mathbb {E} \left\| \mathbf {w} ^ {t} - \mathbf {w} ^ {t - 1} \right\| ^ {2} \right. \\ \left. \right. - \left(4 C _ {1} \gamma^ {2} L _ {g} ^ {2} + C _ {2} \eta^ {2} G _ {g} ^ {2} + 4 L _ {g} ^ {2} G _ {f} ^ {2} \gamma + \eta G _ {g} ^ {2}\right) (1 - \beta) ^ {2} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - \left(f _ {\widehat {\mathcal {T}}} (\mathbf {w} ^ {t - 1}) - f _ {j} (\mathbf {w} ^ {t - 1})\right)\right\| ^ {2} \\ + \frac {1}{8 \gamma} \mathbb {E} \left\| \mathbf {w} ^ {t - 1} - \mathbf {w} ^ {t - 2} \right\| ^ {2} - \left(8 C _ {1} \gamma^ {2} G _ {f} ^ {2} L _ {g} ^ {2} \beta^ {2} + 2 C _ {2} \eta^ {2} G _ {g} ^ {2} \beta^ {2} + 2 \gamma G _ {g} ^ {2} + 8 \gamma G _ {f} ^ {2} L _ {g} ^ {2} \beta^ {2} + 2 C _ {1} \gamma^ {2} G _ {g} ^ {2}\right) \frac {\sigma^ {2}}{B} \\ - \left(1 - \frac {\beta}{2}\right) ^ {2} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t} - z _ {j} ^ {t - 1} \right\| ^ {2}. \\ \end{array}
$$
Recall our definition of potential function  $\tilde{F}^{t + 1}$ :
$$
\tilde {F} ^ {t + 1} := \hat {F} ^ {t + 1} - \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - (f _ {\hat {\mathcal {T}}} (\mathbf {w} ^ {t}) - f _ {j} (\mathbf {w} ^ {t})) \right\| ^ {2} - \frac {(1 - \frac {\beta}{2}) ^ {2}}{1 - (1 - \frac {\beta}{2}) ^ {2}} \frac {4 G _ {g} ^ {2}}{\mu^ {2} \eta} \sum_ {j = 1} ^ {N} \mathbb {E} \left\| z _ {j} ^ {t + 1} - z _ {j} ^ {t} \right\| ^ {2}
$$