SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {2}{\mu_ {\boldsymbol {x}} (t + 1) ^ {c _ {\boldsymbol {x}}}} \left(F _ {S} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) - F _ {S} (\boldsymbol {x}, \boldsymbol {y} ^ {t})\right) \leq \left(1 - \frac {1}{(t + 1) ^ {c _ {\boldsymbol {x}}}}\right) \| \boldsymbol {x} - \boldsymbol {x} ^ {t} \| ^ {2} - \| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} \| ^ {2} + \frac {G ^ {2}}{\mu_ {\boldsymbol {x}} ^ {2} (t + 1) ^ {2 c _ {\boldsymbol {x}}}} \\ + \frac {2}{\mu_ {\boldsymbol {x}} (t + 1) ^ {c _ {\boldsymbol {x}}}} \langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) \rangle \\ \end{array}
$$
$$
\begin{array}{l} F _ {S} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) - F _ {S} \left(\boldsymbol {x}, \boldsymbol {y} ^ {t}\right) \leq \frac {\mu_ {\boldsymbol {x}}}{2} \left(\left(t + 1\right) ^ {c _ {\boldsymbol {x}}} - 1\right) \| \boldsymbol {x} - \boldsymbol {x} ^ {t} \| ^ {2} - \frac {\mu_ {\boldsymbol {x}}}{2} \left(t + 1\right) ^ {c _ {\boldsymbol {x}}} \| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} \| ^ {2} + \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}} \left(t + 1\right) ^ {c _ {\boldsymbol {x}}}} \\ + \left\langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) \right\rangle \\ \end{array}
$$
Taking summation on the above inequality from  $t = 0$  to  $t = T - 1$  and making use of the convexity of  $F_{S}$  on the second parameter that:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \left[ F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) - F _ {\mathcal {S}} (\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}) \right] \\ \leq \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}}} \sum_ {t = 0} ^ {T - 1} \frac {1}{(t + 1) ^ {c _ {\boldsymbol {x}}}} + \sum_ {t = 0} ^ {T - 1} \langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) - \nabla_ {\boldsymbol {x}} F _ {S} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) \rangle \\ \leq \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}}} \sum_ {t = 0} ^ {T - 1} \frac {1}{(t + 1) ^ {c _ {\boldsymbol {x}}}} + \sum_ {t = 0} ^ {T - 1} \langle \boldsymbol {x} ^ {t}, \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) - \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) \rangle \\ + \| \boldsymbol {x} \| \left\| \sum_ {t = 0} ^ {T - 1} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right)\right) \right\| \\ \end{array}
$$
Next we will take expectation on the randomness of the algorithm on both sides and proceed in the same way as we do in case a. that:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathcal {A}} \left[ F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) - \inf _ {\boldsymbol {x} \in \mathcal {X}} F _ {\mathcal {S}} (\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}) \right] \\ \leq \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}}} \sum_ {t = 0} ^ {T - 1} \frac {1}{(t + 1) ^ {c _ {\boldsymbol {x}}}} + 2 C _ {\boldsymbol {x}} G L \sum_ {t = 0} ^ {T - 1} \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s} \\ + C _ {\boldsymbol {x}} \sqrt {4 G ^ {2} T + 4 G ^ {2} L ^ {2} \sum_ {t \neq t ^ {\prime}} \left(\sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s}\right) \left(\sum_ {s = 0} ^ {t ^ {\prime} - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s}\right)} \\ \end{array}
$$
Without loss of generalization, we assume  $\eta_t^{max} = \eta_{\pmb{x},t} = \frac{1}{\mu_{\pmb{x}}(t + 1)^{c_{\pmb{x}}}}$ . Therefore the summation  $\sum_{s=0}^{t-1} \eta_s^{max} \lambda^{t-1-s}$  comes out to be  $\sum_{s=0}^{t-1} \frac{\lambda^{t-1-s}}{\mu_{\pmb{x}}(t+1)^{c_{\pmb{x}}}} \leq \frac{C_{\lambda}}{\mu_{\pmb{x}} t^{c_{\pmb{x}}}}$ . Then we have to analyse in different categories for the value of  $c_{\pmb{x}}$ .
First when  $c_{\pmb{x}} = 1$ , the summation result turns out to be  $\sum_{t=0}^{T-1} \frac{1}{t+1} \leq 1 + \ln T$ ,
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathcal {A}} \left[ F _ {S} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) \right] - \inf  _ {\boldsymbol {x} \in \mathcal {X}} F _ {S} \left(\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}\right) \leq \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}}} \frac {1 + \ln T}{T} + \frac {4 G L C _ {\boldsymbol {x}} C _ {\lambda} \ln T}{\mu_ {\boldsymbol {x}} T} + \frac {2 G C _ {\boldsymbol {x}}}{\sqrt {T}}
$$
When  $0 < c_{\pmb{x}} < 1$ ,  $\sum_{t=0}^{T-1} \frac{1}{(t+1)^{c_{\pmb{x}}}} \leq \frac{T^{1-c_{\pmb{x}}}}{1-c_{\pmb{x}}}$ , then we have:
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathcal {A}} \left[ F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) \right] - \inf  _ {\boldsymbol {x} \in \mathcal {X}} F _ {\mathcal {S}} (\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}) \leq \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}} (1 - c _ {\boldsymbol {x}}) T ^ {c _ {\boldsymbol {x}}}} + \frac {4 G L C _ {\boldsymbol {x}} C _ {\lambda}}{\mu_ {\boldsymbol {x}} (1 - c _ {\boldsymbol {x}}) T ^ {c _ {\boldsymbol {x}}}} + \frac {2 G C _ {\boldsymbol {x}}}{\sqrt {T}}
$$
It is in a similar way to get the symmetric result on the other hand and combining both results we can obtain the strong primal-dual empirical risk:
$$
\mathbb {E} _ {\mathcal {A}} \left[ \sup  _ {\boldsymbol {y} \in \mathcal {Y}} F _ {\mathcal {S}} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y}\right) - \inf  _ {\boldsymbol {x} \in \mathcal {X}} F _ {\mathcal {S}} \left(\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}\right) \right] \leq \frac {2 G \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right)}{\sqrt {T}} + T _ {\boldsymbol {x}} + T _ {\boldsymbol {y}} + T _ {\max }
$$
$$
T _ {\boldsymbol {x}} = \left\{ \begin{array}{c c} \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}}} \frac {1 + \ln T}{T} & c _ {\boldsymbol {x}} = 1 \\ \frac {G ^ {2}}{2 \mu_ {\boldsymbol {x}} (1 - c _ {\boldsymbol {x}}) T ^ {c _ {\boldsymbol {x}}}} & 0 <   c _ {\boldsymbol {x}} <   1 \end{array} ; \right. T _ {\boldsymbol {y}} = \left\{ \begin{array}{c c} \frac {G ^ {2}}{2 \mu_ {\boldsymbol {y}}} \frac {1 + \ln T}{T} & c _ {\boldsymbol {y}} = 1 \\ \frac {G ^ {2}}{2 \mu_ {\boldsymbol {y}} (1 - c _ {\boldsymbol {y}}) T ^ {c _ {\boldsymbol {y}}}} & 0 <   c _ {\boldsymbol {y}} <   1 \end{array} \right.
$$
$$
c _ {m i n} = \min  \{c _ {\boldsymbol {x}}, c _ {\boldsymbol {y}} \}; \quad \mu = \min  \{\mu_ {\boldsymbol {x}}, \mu_ {\boldsymbol {y}} \}; \quad T _ {m a x} = \left\{ \begin{array}{l l} \frac {4 G L C _ {\lambda} (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) \ln T}{\mu T} & c _ {m i n} = 1 \\ \frac {4 G L C _ {\lambda} (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\mu (1 - c _ {m i n}) T ^ {c _ {m i n}}} & 0 <   c _ {m i n} <   1 \end{array} \right.
$$
E.3 Proof of Strong/Weak Primal-Dual Population Risk
In this part, we are going to prove the strong and weak primal-dual population risk of the algorithm D-SGDA. Actually this is an obvious result as a summary of above lemmas and theorems.
Proof of Theorem 3. As we introduced the population risk (see Def. 2), we decompose the population risk into generalization gap and empirical risk that:
$$
\Delta^ {s} (\pmb {x} _ {a v e} ^ {T}, \pmb {y} _ {a v e} ^ {T}) = (\Delta^ {s} (\pmb {x} _ {a v e} ^ {T}, \pmb {y} _ {a v e} ^ {T}) - \Delta_ {S} ^ {s} (\pmb {x} _ {a v e} ^ {T}, \pmb {y} _ {a v e} ^ {T})) + \Delta_ {S} ^ {s} (\pmb {x} _ {a v e} ^ {T}, \pmb {y} _ {a v e} ^ {T})
$$
Notice that we use the average iterate  $\left(\boldsymbol{x}_{ave}^{T},\boldsymbol{y}_{ave}^{T}\right)$  instead of the last iterate to denote the output of D-SGDA. The first term is the averaged version of the strong primal-dual generalization gap and we will make some adjustments.
First we have the argument stability bound of D-SGDA that  $\mathbb{E}_{\mathcal{A}}\left\| \left( \begin{array}{c}\boldsymbol{x}^T -\dot{\boldsymbol{x}}^T\\ \boldsymbol{y}^T -\dot{\boldsymbol{y}}^T \end{array} \right)\right\| \leq \epsilon_{sta}^{arg}(\mathcal{A})$  where  $(\pmb {x}^T,\pmb {y}^T)$  and  $(\dot{\pmb{x}}^T,\dot{\pmb{y}}^T)$  denote the  $T$  -th output when D-SGDA is executed on the neighboring dataset
respectively. So we can make use of the convexity of the norm and obtain the argument stability of averaged-version:
$$
\mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{l} \boldsymbol {x} _ {a v e} ^ {T} - \dot {\boldsymbol {x}} _ {a v e} ^ {T} \\ \boldsymbol {y} _ {a v e} ^ {T} - \dot {\boldsymbol {y}} _ {a v e} ^ {T} \end{array} \right) \right\| \leq \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{l} \boldsymbol {x} ^ {T} - \dot {\boldsymbol {x}} ^ {T} \\ \boldsymbol {y} ^ {T} - \dot {\boldsymbol {y}} ^ {T} \end{array} \right) \right\| \leq \epsilon_ {s t a} ^ {\arg} (\mathcal {A}) \tag {15}
$$
Then the strong primal-dual generalization gap should be  $G\sqrt{2 + \frac{2L^2}{\mu^2}}\epsilon_{sta}^{arg}(\mathcal{A})$  following Thm. 1 when each  $f_i$  is strongly convex w.r.t.  $x$  and strongly concave w.r.t.  $y$ .
At last, the strong primal-dual empirical risk  $\mathbb{E}_{\mathcal{A}}[\Delta_{\mathcal{S}}^{s}(\pmb{x}_{ave}^{T},\pmb{y}_{ave}^{T})]$  is studied in Thm. 7. So we can combine above two bounds to analyze the strong primal-dual population risk in different categories when learning rates are fixed:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \Delta^ {s} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) \right] \\ \leq G \sqrt {2 + \frac {2 L ^ {2}}{\mu^ {2}}} \left(2 G \frac {L + \mu}{\eta^ {\min} L \mu} \left(\frac {2 (\eta^ {\max}) ^ {2} L}{1 - \lambda} + \frac {\eta^ {\max}}{n}\right)\right) + \frac {C _ {\boldsymbol {x}} ^ {2} + C _ {\boldsymbol {y}} ^ {2}}{2 \eta^ {\min} T} + \eta^ {\max} G ^ {2} \\ + \frac {4 \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right) G L \eta^ {\max }}{1 - \lambda} + \frac {2 \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right) G}{\sqrt {T}} \\ \end{array}
$$
And when learning rates are varying that  $\eta_t^{min} = \frac{1}{\mu(t + 1)}$  and  $\eta_t^{max} = \frac{1}{\mu(t + 1)^c}, c = 1$ , requiring  $2c > \frac{L}{L + \mu} + 1$ :
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \Delta^ {s} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) \right] \\ \leq G \sqrt {2 + \frac {2 L ^ {2}}{\mu^ {2}}} \left(\frac {2 G}{\mu n T ^ {\frac {L}{L + \mu}}} \sum_ {k = 0} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} + \frac {4 G L}{\mu^ {2} T ^ {\frac {L}{L + \mu}}} \sum_ {k = 1} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} \frac {C _ {\lambda}}{k ^ {c}}\right) \\ + \frac {2 G \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right)}{\sqrt {T}} + \frac {G ^ {2}}{\mu} \frac {1 + \ln T}{T} + \frac {4 G L C _ {\lambda} \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right) \ln T}{\mu T} \\ \leq G \sqrt {2 + \frac {2 L ^ {2}}{\mu^ {2}}} \left(\frac {2 G}{\mu \frac {L}{L + \mu}} \frac {1}{n} + \frac {4 G L C _ {\lambda}}{\mu^ {2} (1 - \frac {L}{L + \mu})} \frac {1}{T ^ {\frac {L}{L + \mu}}}\right) + \frac {2 G (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\sqrt {T}} + \frac {G ^ {2}}{\mu} \frac {1 + \ln T}{T} \\ + \frac {4 G L C _ {\lambda} \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right) \ln T}{\mu T} \\ \end{array}
$$
When  $c < 1$  and requiring  $2c > \frac{L}{L + \mu} + 1$ :
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \Delta^ {s} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) \right] \\ \leq G \sqrt {2 + \frac {2 L ^ {2}}{\mu^ {2}}} \left(\frac {2 G}{\mu n T ^ {\frac {L}{L + \mu}}} \sum_ {k = 0} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} + \frac {4 G L}{\mu^ {2} T ^ {\frac {L}{L + \mu}}} \sum_ {k = 1} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} \frac {C _ {\lambda}}{k ^ {c}}\right) \\ + \frac {2 G (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\sqrt {T}} + \frac {G ^ {2}}{2 \mu} \left(\frac {1 + \ln T}{T} + \frac {1}{(1 - c) T ^ {c}}\right) + \frac {4 G L C _ {\lambda} (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\mu (1 - c) T ^ {c}} \\ \leq G \sqrt {2 + \frac {2 L ^ {2}}{\mu^ {2}}} \left(\frac {2 G}{\mu (1 - c + \frac {L}{L + \mu})} \frac {T ^ {1 - c}}{n} + \frac {4 G L C _ {\lambda}}{\mu^ {2} (2 c - \frac {L}{L + \mu} - 1)} \frac {1}{T ^ {\frac {L}{L + \mu}}}\right) + \frac {2 G (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\sqrt {T}} \\ + \frac {G ^ {2}}{2 \mu} \left(\frac {1 + \ln T}{T} + \frac {1}{(1 - c) T ^ {c}}\right) + \frac {4 G L C _ {\lambda} \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right)}{\mu (1 - c) T ^ {c}} \\ \end{array}
$$
When  $c < 1$  and  $2c = \frac{L}{L + \mu} + 1$ :
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \Delta^ {s} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) \right] \\ \leq G \sqrt {2 + \frac {2 L ^ {2}}{\mu^ {2}}} \left(\frac {2 G}{\mu n T ^ {\frac {L}{L + \mu}}} \sum_ {k = 0} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} + \frac {4 G L}{\mu^ {2} T ^ {\frac {L}{L + \mu}}} \sum_ {k = 1} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} \frac {C _ {\lambda}}{k ^ {c}}\right) \\ + \frac {2 G (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\sqrt {T}} + \frac {G ^ {2}}{2 \mu} \left(\frac {1 + \ln T}{T} + \frac {1}{(1 - c) T ^ {c}}\right) + \frac {4 G L C _ {\lambda} (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\mu (1 - c) T ^ {c}} \\ \leq G \sqrt {2 + \frac {2 L ^ {2}}{\mu^ {2}}} \left(\frac {2 G}{c \mu} \frac {T ^ {1 - c}}{n} + \frac {4 G L C _ {\lambda}}{\mu^ {2}} \frac {\ln T}{T ^ {\frac {L}{L + \mu}}}\right) + \frac {2 G (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\sqrt {T}} + \frac {G ^ {2}}{2 \mu} (\frac {1 + \ln T}{T} + \frac {1}{(1 - c) T ^ {c}}) \\ + \frac {4 G L C _ {\lambda} \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right)}{\mu (1 - c) T ^ {c}} \\ \end{array}
$$
F Proof in the Convex-Concave Case
In this section, we will provide corresponding proof for argument stability, optimization error and weak primal-dual population risk in the C-C condition.
F.1 Proof of Stability
Proof of Theorem 4. Analogous to Eq. (9) in the proof for SC-SC (see Appendix E.1), considering C-C a special case for  $\mu_{\pmb{x}}\mathrm{SC} - \mu_{\pmb{y}}\mathrm{SC}$  when  $\mu_{\pmb{x}} = 0, \mu_{\pmb{y}} = 0$
Thus we can get the result:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{c} \mathcal {A} _ {\boldsymbol {x}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {x}} (\mathcal {S} ^ {\prime}) \\ \mathcal {A} _ {\boldsymbol {y}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {y}} (\mathcal {S} ^ {\prime}) \end{array} \right) \right\| \\ \leq \frac {2 G}{n} \sum_ {k = 0} ^ {T - 1} \eta_ {k} ^ {\max } + 4 G L \sum_ {k = 1} ^ {T - 1} \left(\eta_ {k} ^ {\max } \sum_ {s = 0} ^ {k - 1} \eta_ {s} ^ {\max } \lambda^ {k - 1 - s}\right) \\ \end{array}
$$
F.2 Proof of the Empirical Risk
Similar to Thm. 7, considering C-C as a special case of SC-SC for  $\mu_{\pmb{x}} = \mu_{\pmb{y}} = 0$ , we can get the weak primal-dual empirical risk in the following corollary, where we use the Jensen's inequality that  $\Delta_S^w (\pmb {x},\pmb {y})\leq \Delta_S^s (\pmb {x},\pmb {y})$
Corollary 1. Under assumption 1,2,3 and the restriction that  $\sup_{\boldsymbol{x} \in \mathcal{X}} \| \boldsymbol{x} \| \leq C_{\boldsymbol{x}}$  and  $\sup_{\boldsymbol{y} \in \mathcal{Y}} \| \boldsymbol{y} \| \leq C_{\boldsymbol{y}}$ , each local function is  $f_{i}$  is  $C-C$ . We have the weak primal-dual empirical risk over the dataset  $\mathcal{S}$  on the average output in  $T$  iterations as following for fixed learning rates:
$$
\Delta_ {\mathcal {S}} ^ {w} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}) \leq \frac {C _ {\boldsymbol {x}} ^ {2} + C _ {\boldsymbol {y}} ^ {2}}{2 \eta^ {m i n} T} + \eta^ {m a x} G ^ {2} + \frac {4 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G L \eta^ {\max}}{1 - \lambda} + \frac {2 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G}{\sqrt {T}}.
$$
F.3 Proof of Weak Primal-Dual Population Risk
Proof of Theorem 5. When each local function  $f_{i}$  is not strongly convex or strongly concave, we can not get access to the strong primal-dual generalization gap but weak primal-dual generalization gap. Following the same step in above proof that:
$$
\Delta^ {w} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}) = \left(\Delta^ {w} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}) - \Delta_ {\mathcal {S}} ^ {w} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T})\right) + \Delta_ {\mathcal {S}} ^ {w} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T})
$$
Analogously we have the weak primal-dual generalization gap according to Thm. 1 that:
$$
\Delta^ {w} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) - \Delta_ {S} ^ {w} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) \leq \sqrt {2} G \epsilon_ {s t a} ^ {\arg} (\mathcal {A})
$$
where  $\epsilon_{sta}^{arg}(\mathcal{A})\leq \frac{2G\eta^{max}T}{n} +\frac{4GL(\eta^{max})^2T}{1 - \lambda}$  follows Thm. 4 when learning rates are fixed.
In the case without strong convexity or strong concavity, we select the fixed learning rates and following the Thm. 7, we can bound the weak primal-dual empirical risk as:
$$
\Delta_ {\mathcal {S}} ^ {w} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) \leq \frac {C _ {\boldsymbol {x}} ^ {2} + C _ {\boldsymbol {y}} ^ {2}}{2 \eta^ {\min } T} + \eta^ {\max } G ^ {2} + \frac {4 \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right) G L \eta^ {\max }}{1 - \lambda} + \frac {2 \left(C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}\right) G}{\sqrt {T}}
$$
Finally we combine above patterns and we can get the weak primal-dual population risk:
$$
\begin{array}{l} \Delta^ {w} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}\right) \leq \sqrt {2} G \left(\frac {2 G \eta^ {\max } T}{n} + \frac {4 G L (\eta^ {\max }) ^ {2} T}{1 - \lambda}\right) \\ + \frac {C _ {\boldsymbol {x}} ^ {2} + C _ {\boldsymbol {y}} ^ {2}}{2 \eta^ {m i n} T} + \eta^ {m a x} G ^ {2} + \frac {4 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G L \eta^ {\mathrm {m a x}}}{1 - \lambda} + \frac {2 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G}{\sqrt {T}} \\ \end{array}
$$
G Proof in Nonconvex-Nonconcave Case
In this section, we will provide proof for weak stability and therefore we can derive the weak primal-dual generalization gap following Thm. 1 in the NC-NC condition.
G.1 Important Lemmas
Before we present the proof for the stability bound in nonconvex-nonconcave case, we should first introduce an important lemma which describes the fact that D-SGDA will run several iterations before encountering the different samples. We extend the Lemma 3.11 in [13] and make adjustments on Lemma F.1 in [19] to fit our decentralized setting.
Lemma 5. Let  $\mathcal{S} = \{\mathcal{S}_1, \dots, \mathcal{S}_m\}$  and  $\mathcal{S}' = \{\mathcal{S}_1', \dots, \mathcal{S}_m'\}$  be any arbitrary neighboring datasets,  $(\boldsymbol{x}^t, \boldsymbol{y}^t)$  and  $(\dot{\boldsymbol{x}}^t, \dot{\boldsymbol{y}}^t)$  represent output in t-th iteration under dataset  $\mathcal{S}$  and  $\mathcal{S}'$  respectively. We further require each local function is bounded that  $|f_i(\boldsymbol{x}, \boldsymbol{y}; \xi)| \leq B, \forall \boldsymbol{x} \in \mathcal{X}, \boldsymbol{y} \in \mathcal{Y}$ . Denoting  $\delta_t = \left\| \begin{pmatrix} \boldsymbol{x}^t - \dot{\boldsymbol{x}}^t \\ \boldsymbol{y}^t - \dot{\boldsymbol{y}}^t \end{pmatrix} \right\|$ , then we have:
$$
\mathbb {E} _ {\mathcal {A}} [ \pmb {f} (\pmb {x} ^ {t}, \pmb {y} ^ {\prime}; \pmb {\xi}) - \pmb {f} (\dot {\pmb {x}} ^ {t}, \pmb {y} ^ {\prime}; \pmb {\xi}) + \pmb {f} (\pmb {x} ^ {\prime}, \pmb {y} ^ {t}; \pmb {\xi}) - \pmb {f} (\pmb {x} ^ {\prime}, \dot {\pmb {y}} ^ {t}; \pmb {\xi}) ] \leq \sqrt {2} G \mathbb {E} _ {\mathcal {A}} \left[ \delta_ {t} \Bigg | \delta_ {t _ {0}} = 0 \right] + \frac {B m t _ {0}}{n}.
$$
Proof. First according to the property of Lipschitz continuity (see Assumption 1):
$$
\begin{array}{l} \pmb {f} (\pmb {x} ^ {t}, \pmb {y} ^ {\prime}; \pmb {\xi}) - \pmb {f} (\dot {\pmb {x}} ^ {t}, \pmb {y} ^ {\prime}; \pmb {\xi}) + \pmb {f} (\pmb {x} ^ {\prime}, \pmb {y} ^ {t}; \pmb {\xi}) - \pmb {f} (\pmb {x} ^ {\prime}, \dot {\pmb {y}} ^ {t}; \pmb {\xi}) \\ \leq G \| \boldsymbol {x} ^ {t} - \dot {\boldsymbol {x}} ^ {t} \| + G \| \boldsymbol {y} ^ {t} - \dot {\boldsymbol {y}} ^ {t} \| \\ \leq G \sqrt {2} \delta_ {t} \\ \end{array}
$$
Then we decompose the expectation by the law of total expectation:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \boldsymbol {f} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\dot {\boldsymbol {x}} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) + \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {t}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \dot {\boldsymbol {y}} ^ {t}; \boldsymbol {\xi}\right) \right] \\ = \mathbb {P} \left(\delta_ {t _ {0}} = 0\right) \mathbb {E} _ {\mathcal {A}} \left[ \left. \boldsymbol {f} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\dot {\boldsymbol {x}} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) + \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {t}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \dot {\boldsymbol {y}} ^ {t}; \boldsymbol {\xi}\right) \right| \delta_ {t _ {0}} = 0 \right] \\ + \mathbb {P} \left(\delta_ {t _ {0}} \neq 0\right) \mathbb {E} _ {\mathcal {A}} \left[ \left. \boldsymbol {f} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\dot {\boldsymbol {x}} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) + \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {t}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \dot {\boldsymbol {y}} ^ {t}; \boldsymbol {\xi}\right) \right| \delta_ {t _ {0}} \neq 0 \right] \\ \leq \sqrt {2} G \mathbb {E} _ {\mathcal {A}} \left[ \delta_ {t} \mid \delta_ {t _ {0}} = 0 \right] + B \mathbb {P} \left(\delta_ {t _ {0}} \neq 0\right) \\ \end{array}
$$
While the event that  $\delta_{t_0} \neq 0$  means the training process has already encountered the different samples before  $t_0$ :
$$
\mathbb {P} \left(\delta_ {t _ {0}} \neq 0\right) \leq \sum_ {t = 1} ^ {t _ {0}} \sum_ {k = 1} ^ {m} C _ {m} ^ {k} (\frac {1}{n}) ^ {k} \left(1 - \frac {1}{n}\right) ^ {m - k} = t _ {0} \left(1 - \left(1 - \frac {1}{n}\right) ^ {m}\right) \leq \frac {m t _ {0}}{n}
$$
Combining above inequalities and we can prove the Lemma.
G.2 Proof of Stability
Proof of Theorem 6. We are under the same setting as in the proof for SC-SC (see Appendix E.1) that  $(\pmb{x}^t,\pmb{y}^t)$  and  $(\dot{\pmb{x}}^t,\dot{\pmb{y}}^t)$  representing the  $t$ -th output over neighboring dataset  $S$  and  $S'$  respectively. And we assume each local dataset  $S_i'$  in  $S' = \{S_1',\dots,S_m'\}$  differs from  $S$  by the last sample without loss of generalization, i.e.,  $S_i = \{\xi_{i,1},\dots,\xi_{i,n}\}$  while  $S_i' = \{\xi_{i,1},\dots,\xi_{i,n - 1},\xi_{i,n}'\}$ .