SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Analogous to the decomposition equality (7), we will bound the term  $I_1$  and  $I_2'$  without convexity or concavity. Referring to Lemma 1, we have:
$$
\| I _ {1} \| \leq \frac {m - m _ {0}}{m} (1 + \eta_ {t} ^ {m i n} L) \left\| \left( \begin{array}{c} \boldsymbol {x} ^ {t} - \dot {\boldsymbol {x}} ^ {t} \\ \boldsymbol {y} ^ {t} - \dot {\boldsymbol {y}} ^ {t} \end{array} \right) \right\|
$$
$$
\| I _ {2} \| \leq \frac {m _ {0}}{m} \left[ (1 + \eta_ {t} ^ {m i n} L) \left\| \left( \begin{array}{c} {\pmb x ^ {t} - \dot {\pmb x} ^ {t}} \\ {\pmb y ^ {t} - \dot {\pmb y} ^ {t}} \end{array} \right) \right\| + 2 \eta_ {t} ^ {m a x} G \right]
$$
where we use the same denotation that  $\eta_t^{min} = \min \{\eta_{\pmb{x},t},\eta_{\pmb{y},t}\}$  and  $\eta_t^{max} = \max \{\eta_{\pmb{x},t},\eta_{\pmb{y},t}\}$ . So we can get the similar result as the inequality (8):
$$
\begin{array}{l} \mathbb{E}_{\mathcal{A}}\left[\left|\left( \begin{array}{c}\boldsymbol{x}^{t + 1} - \dot{\boldsymbol{x}}^{t + 1}\\ \boldsymbol{y}^{t + 1} - \dot{\boldsymbol{y}}^{t + 1} \end{array} \right)\right|\bigg|\delta_{t_{0}} = 0\right] \\ \leq \sum_{m_{0} = 0}^{m}C_{m}^{m_{0}}(1 - \frac{1}{n})^{m - m_{0}}(\frac{1}{n})^{m_{0}}\left(\left(1 + \eta_{t}^{min}L\right)\mathbb{E}_{\mathcal{A}}\left[\left\| \left( \begin{array}{c}\boldsymbol{x}^{t} - \dot{\boldsymbol{x}}^{t}\\ \boldsymbol{y}^{t} - \dot{\boldsymbol{y}}^{t} \end{array} \right)\right\| \right|\delta_{t_{0}} = 0\right] \\ \left. + \frac {m _ {0}}{m} 2 \eta_ {t} ^ {\max } G + 4 \eta_ {t} ^ {\max } L G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s}\right) \\ \leq \left(1 + \eta_ {t} ^ {m i n} L\right) \mathbb {E} _ {\mathcal {A}} \left[ \left\| \left( \begin{array}{c} \boldsymbol {x} ^ {t} - \dot {\boldsymbol {x}} ^ {t} \\ \boldsymbol {y} ^ {t} - \dot {\boldsymbol {y}} ^ {t} \end{array} \right) \right\| \right| \delta_ {t _ {0}} = 0 \biggr ] + 4 \eta_ {t} ^ {m a x} L G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {m a x} \lambda^ {t - 1 - s} \\ + \sum_ {m _ {0} = 0} ^ {m} C _ {m} ^ {m _ {0}} \left(1 - \frac {1}{n}\right) ^ {m - m _ {0}} \left(\frac {1}{n}\right) ^ {m _ {0}} \frac {m _ {0}}{m} 2 \eta_ {t} ^ {\max } G \\ = \left(1 + \eta_ {t} ^ {m i n} L\right) \mathbb {E} _ {\mathcal {A}} \left[ \left| \left| \left( \begin{array}{c} \boldsymbol {x} ^ {t} - \dot {\boldsymbol {x}} ^ {t} \\ \boldsymbol {y} ^ {t} - \dot {\boldsymbol {y}} ^ {t} \end{array} \right) \right| \right| \delta_ {t _ {0}} = 0 \right] + 4 \eta_ {t} ^ {m a x} L G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {m a x} \lambda^ {t - 1 - s} + \frac {2 \eta_ {t} ^ {m a x} G}{n} \\ \end{array}
$$
Recursively applying above inequalities from  $t = t_0$  to  $t = T - 1$ :
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \right.\left\|\left(\begin{array}{c}\mathcal {A} _ {\boldsymbol {x}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {x}} (\mathcal {S} ^ {\prime})\\\mathcal {A} _ {\boldsymbol {y}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {y}} (\mathcal {S} ^ {\prime})\end{array}\right)\right\|\left. \right| \delta_ {t _ {0}} = 0 \left. \right] \\ \leq \frac {2 G}{n} \sum_ {k = t _ {0}} ^ {T - 1} \eta_ {k} ^ {\max } \prod_ {s = k + 1} ^ {T - 1} \left(1 + \eta_ {s} ^ {\min } L\right) + 4 G L \sum_ {k = t _ {0}} ^ {T - 1} \left(\eta_ {k} ^ {\max } \sum_ {s = 0} ^ {k - 1} \eta_ {s} ^ {\max } \lambda^ {k - 1 - s}\right) \prod_ {j = k + 1} ^ {T - 1} \left(1 + \eta_ {j} ^ {\min } L\right) \tag {16} \\ \end{array}
$$
where we use the fact that  $\delta_{t_0} = \left\| \left( \begin{array}{c}\boldsymbol{x}^t -\dot{\boldsymbol{x}}^t\\ \boldsymbol{y}^t -\boldsymbol{y}^t \end{array} \right)\right\| = 0$
When learning rates are fixed that  $\eta_{\pmb{x},t} = \eta_{\pmb{x}}$  and  $\eta_{\pmb{y},t} = \eta_{\pmb{y}}$ , then we have:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \delta_ {t} \Bigg | \delta_ {t _ {0}} = 0 \right] \\ \leq \frac {2 G \eta^ {m a x} ((1 + \eta^ {m i n} L) ^ {T - t _ {0}} - 1)}{n \eta^ {m i n} L} + \frac {4 G L (\eta^ {m a x}) ^ {2} ((1 + \eta^ {m i n} L) ^ {T - t _ {0}} - 1)}{(1 - \lambda) \eta^ {m i n} L} \\ \end{array}
$$
Then combining with Lemma 5, we have the following result:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \boldsymbol {f} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\dot {\boldsymbol {x}} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) + \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {t}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \dot {\boldsymbol {y}} ^ {t}; \boldsymbol {\xi}\right) \right] \\ \leq \sqrt {2} G \left(\frac {2 G \eta^ {\max}}{n} + \frac {4 G L \eta^ {\max 2}}{1 - \lambda}\right) (T - t _ {0}) + \frac {B m t _ {0}}{n} \\ \end{array}
$$
where it can obtain the optimal of  $2\sqrt{2} G^2 \left( \frac{\eta^{max} T}{n} + \frac{2L \eta^{max} 2 T}{1 - \lambda} \right)$  when  $t_0 = 0$ .
When learning rates are varying that  $\eta_t^{min} = \frac{1}{t + 1}$  and  $\eta_t^{max} = \frac{1}{(t + 1)^c}$ ,  $c \leq 1$ , then we can simplify the production by  $1 \pm a \leq \exp \{a\}$ :
$$
\begin{array}{l} \prod_ {j = k + 1} ^ {T - 1} \left(1 + \eta_ {j} ^ {\min } L\right) = \prod_ {j = k + 1} ^ {T - 1} \left(1 + \frac {L}{j + 1}\right) \leq \prod_ {j = k + 1} ^ {T - 1} \exp \left\{\frac {L}{j + 1} \right\} = \exp \left\{\sum_ {j = k + 1} ^ {T - 1} \frac {L}{j + 1} \right\} \\ \leq \exp \left\{L \ln \frac {T}{k + 1} \right\} = (\frac {T}{k + 1}) ^ {L} \\ \end{array}
$$
Then back to the inequality (16), we can obtain:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \delta_ {t} \middle | \delta_ {t _ {0}} = 0 \right] \\ \leq \frac {2 G}{n} \sum_ {k = t _ {0}} ^ {T - 1} \frac {1}{(k + 1) ^ {c}} \left(\frac {T}{k + 1}\right) ^ {L} + 4 G L \sum_ {k = t _ {0}} ^ {T - 1} \frac {1}{(k + 1) ^ {c}} \frac {C _ {\lambda}}{k ^ {c}} \left(\frac {T}{k + 1}\right) ^ {L} \tag {17} \\ = \frac {2 G T ^ {L}}{n} \sum_ {k = t _ {0}} ^ {T - 1} \frac {1}{(k + 1) ^ {c + L}} + 4 G L C _ {\lambda} T ^ {L} \sum_ {k = t _ {0}} ^ {T - 1} \frac {1}{(k + 1) ^ {c + L} k ^ {c}} \\ \end{array}
$$
Requiring  $c + L > 1$  for convergence and combining with Lemma 5, we have the following inequality:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \boldsymbol {f} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\dot {\boldsymbol {x}} ^ {t}, \boldsymbol {y} ^ {\prime}; \boldsymbol {\xi}\right) + \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {t}; \boldsymbol {\xi}\right) - \boldsymbol {f} \left(\boldsymbol {x} ^ {\prime}, \dot {\boldsymbol {y}} ^ {t}; \boldsymbol {\xi}\right) \right] \\ \leq \sqrt {2} G \left(\frac {2 G T ^ {L}}{n} \sum_ {k = t _ {0}} ^ {T - 1} \frac {1}{(k + 1) ^ {c + L}} + 4 G L C _ {\lambda} T ^ {L} \sum_ {k = t _ {0}} ^ {T - 1} \frac {1}{(k + 1) ^ {c + L} k ^ {c}}\right) + \frac {B m t _ {0}}{n} \\ \leq \frac {2 \sqrt {2} G ^ {2} T ^ {L}}{(c + L - 1) n} \frac {1}{t _ {0} ^ {c + L - 1}} + \frac {4 \sqrt {2} G ^ {2} L C _ {\lambda} T ^ {L}}{(2 c + L - 1)} \frac {1}{t _ {0} ^ {2 c + L - 1}} + \frac {B m t _ {0}}{n} \\ \leq (c + L) (c + L - 1) ^ {\frac {1}{c + L}} (\frac {2 \sqrt {2} G ^ {2} T ^ {L}}{(c + L - 1) n} + \frac {4 \sqrt {2} G ^ {2} L C _ {\lambda} T ^ {L}}{2 c + L - 1}) ^ {\frac {1}{c + L}} (\frac {B m}{n}) ^ {1 - \frac {1}{c + L}} \\ \end{array}
$$
where for the last inequality, when  $t_0 = \left(\frac{(c + L - 1)\left(\frac{2\sqrt{2}G^2T^L}{(c + L - 1)n} + \frac{4\sqrt{2}G^2LC_{\lambda}T^L}{2c + L - 1}\right)}{\frac{Bm}{n}}\right)^{\frac{1}{c + L}}$ , it can obtain minimal.
Taking supremum over parameters  $x, y$  separately and over the random sample, we can get the weak stability for D-SGDA in the NC-NC condition.
H Primal Metric
In addition to the primal-dual population risk (see Def. 2) as well as the corresponding generalization gap (see Def. 3), primal measure extends the relative concept in single variable minimization problem.
The excess primal population risk is defined as  $\Delta^{ex}(\pmb{x}) = \sup_{\pmb{y}' \in \mathcal{Y}} F(\pmb{x}, \pmb{y}') - \inf_{\pmb{x}' \in \mathcal{X}} \sup_{\pmb{y}' \in \mathcal{Y}} F(\pmb{x}', \pmb{y}')$ ; and the excess primal empirical risk is  $\Delta_S^{ex}(\pmb{x}) = \sup_{\pmb{y}' \in \mathcal{Y}} F_S(\pmb{x}, \pmb{y}') - \inf_{\pmb{x}' \in \mathcal{X}} \sup_{\pmb{y}' \in \mathcal{Y}} F_S(\pmb{x}', \pmb{y}')$ , for a randomized model  $(\pmb{x}, \pmb{y})$ .
There are two ways to define the corresponding generalization gap. One is to directly subtract the excess primal empirical risk from the excess primal population risk, defined as the excess primal generalization gap:  $\epsilon_{gen}^{ex}(\pmb{x}) = \Delta^{ex}(\pmb{x}) - \Delta_S^{ex}(\pmb{x})$  (called primal gap in [30]). Another one is to neglect the difference between the saddle point of  $F$  and  $F_S$ , defined as the primal generalization gap:  $\epsilon_{gen}^{pr}(\pmb{x}) = \sup_{\pmb{y}' \in \mathcal{Y}} F(\pmb{x}, \pmb{y}') - \sup_{\pmb{y}' \in \mathcal{Y}} F_S(\pmb{x}, \pmb{y}')$ .
Remark 11. Our definition of strong primal-dual population risk (see Def. 2) has included the expectation inside for assistance with the definition of weak primal-dual population risk (see Def. 2). Although the definitions in [19] hold different forms, which do not contain the expectation inside, while our theoretical result aims at the same value.
Ozdaglar et al. [30] points out that the excess primal generalization gap  $\epsilon_{gen}^{ex}$  can act as a better metric to characterize the generalizability in the nonconvex condition. It is our limitation that we do not
calculate the corresponding excess primal generalization gap and population risk for nonconvex case in our paper.
While we omit the primal generalization gap  $\epsilon_{gen}^{pr}$  and excess primal population risk  $\Delta^{ex}$  under C-C condition in the main text, for it can be derived from the corresponding proof of strong primal-dual risk. And we will illustrate them in the following as a corollary.
Corollary 2. For an  $\epsilon$ -argument stable decentralized algorithm  $\mathcal{A}$ , under Assumption 1, 2, when each  $f_{i}$  is  $\mu_{\pmb{y}}$ -strongly concave on the second parameter, we have the primal generalization gap:
$$
\mathbb {E} _ {\mathcal {A}, \mathcal {S}} [ \epsilon_ {g e n} ^ {p r} (\mathcal {A} _ {\boldsymbol {x}} (\mathcal {S})) ] \leq G \sqrt {1 + \frac {L ^ {2}}{\mu_ {y} ^ {2}}} \epsilon .
$$
Proof of Corollary 2.  $\epsilon_{gen}^{pr}(\mathcal{A}_x(S))$  is exactly the first counterpart of the strong primal-dual generalization gap  $\epsilon_{gen}^s (\mathcal{A}_x(S),\mathcal{A}_y(S))$  in Eq. (6). And referring to the proof for case b. in Thm. 1 (see Appendix D), we can get the result as above.
Corollary 3. Under Assumption 1,2,3, when each  $f_{i}$  is  $\mu_{\mathbf{x}}SC - \mu_{\mathbf{y}}SC$ , we have the excess primal population risk as follows, where  $\eta_t^{\text{max}} \triangleq \max \{\eta_{\mathbf{x},t}, \eta_{\mathbf{y},t}\}$ ,  $\eta_t^{\text{min}} \triangleq \min \{\eta_{\mathbf{x},t}, \eta_{\mathbf{y},t}\}$ ,  $\mu = \min \{\mu_{\mathbf{x}}, \mu_{\mathbf{y}}\}$ , and  $(\pmb{x}_{ave}^{T}, \pmb{y}_{ave}^{T})$  is defined in Eq. (3):
$$
\begin{array}{l} \mathbb {E} [ \Delta^ {p r} (\boldsymbol {x} _ {a v e} ^ {T}) ] \leq 2 G \sqrt {1 + \frac {L ^ {2}}{\mu^ {2}} (2 G \frac {L + \mu}{\eta^ {m i n} L \mu} (\frac {2 (\eta^ {m a x}) ^ {2} L}{1 - \lambda} + \frac {\eta^ {m a x}}{n}))} + \frac {C _ {\boldsymbol {x}} ^ {2} + C _ {\boldsymbol {y}} ^ {2}}{2 \eta^ {m i n} T} \\ + \eta^ {m a x} G ^ {2} + \frac {4 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G L \eta^ {\max}}{1 - \lambda} + \frac {2 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G}{\sqrt {T}}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ \Delta^ {p r} \left(\boldsymbol {x} _ {a v e} ^ {T}\right) \right] \\ \leq 2 G \sqrt {1 + \frac {L ^ {2}}{\mu^ {2}}} \left(\frac {2 G}{\mu (1 - c + \frac {L}{L + \mu})} \frac {T ^ {1 - c}}{n} + \frac {4 G L C _ {\lambda}}{\mu^ {2} T ^ {\frac {L}{L + \mu}}} \left(\frac {\mathbf {1} _ {2 c \neq L / (L + \mu) + 1}}{2 c - \frac {L}{L + \mu} - 1} + \ln T \cdot \mathbf {1} _ {2 c = L / (L + \mu) + 1}\right)\right) \\ + \frac {2 G (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\sqrt {T}} + \frac {G ^ {2}}{2 \mu} (\frac {1 + \ln T}{T} + \frac {\mathbf {1} _ {c \neq 1}}{(1 - c) T ^ {c}} + \frac {(1 + \ln T) \mathbf {1} _ {c = 1}}{T}) + \frac {4 G L C _ {\lambda} (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\mu T ^ {c}} (\frac {\mathbf {1} _ {c \neq 1}}{1 - c} + \ln T \cdot \mathbf {1} _ {c = 1}). \\ \end{array}
$$
Proof of Corollary 3. Firstly we already know the argument stability bound for D-SGDA (denoted as  $\mathcal{A}$ ) on the last iterate. Then the averaged output follows due to the convexity of the norm:
$$
\mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{c} \boldsymbol {x} _ {a v e} ^ {T} - \dot {\boldsymbol {x}} _ {a v e} ^ {T} \\ \boldsymbol {y} _ {a v e} ^ {T} - \dot {\boldsymbol {y}} _ {a v e} ^ {T} \end{array} \right) \right\| \leq \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{c} \boldsymbol {x} ^ {T} - \dot {\boldsymbol {x}} ^ {T} \\ \boldsymbol {y} ^ {T} - \dot {\boldsymbol {y}} ^ {T} \end{array} \right) \right\| \leq \epsilon_ {s t a} ^ {a r g} (\mathcal {A})
$$
Then we have the primal generalization gap according to Corollary 2 that  $\mathbb{E}_{\mathcal{A},\mathcal{S}}[\epsilon_{gen}^{pr}(\pmb{x}_{ave}^T)] \leq G\sqrt{1 + \frac{L^2}{\mu_y^2}}\epsilon_{sta}^{arg}(\mathcal{A})$ .
We decompose the primal population risk for the averaged output  $(\pmb{x}_{ave}^{T},\pmb{y}_{ave}^{T})$  as follows:
$$
\begin{array}{l} \sup_{\boldsymbol {y}^{\prime}\in \mathcal{Y}}F(\boldsymbol{x}_{ave}^{T},\boldsymbol{y}^{\prime}) - \inf_{\boldsymbol{x}^{\prime}\in \mathcal{X}}\sup_{\boldsymbol {y}^{\prime}\in \mathcal{Y}}F(\boldsymbol{x}^{\prime},\boldsymbol{y}^{\prime}) \\ = \left(\sup  _ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y}} F \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} ^ {\prime}\right) - \sup  _ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y}} F _ {S} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} ^ {\prime}\right)\right) + \left(\sup  _ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y}} F _ {S} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} ^ {\prime}\right) - \inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} F _ {S} \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} _ {a v e} ^ {T}\right)\right) \\ + \left(\inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} _ {a v e} ^ {T}) - \inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} F (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} _ {a v e} ^ {T})\right) + \left(\inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} F (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} _ {a v e} ^ {T}) - \inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} \sup  _ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y}} F (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime})\right) \\ \leq \left(\sup  _ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y}} F (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} ^ {\prime}) - \sup  _ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y}} F _ {\mathcal {S}} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} ^ {\prime})\right) + \left(\sup  _ {\boldsymbol {y} ^ {\prime} \in \mathcal {Y}} F _ {\mathcal {S}} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} ^ {\prime}) - \inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} _ {a v e} ^ {T})\right) \\ + \left(\inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} _ {a v e} ^ {T}) - \inf  _ {\boldsymbol {x} ^ {\prime} \in \mathcal {X}} F (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} _ {a v e} ^ {T})\right) \\ \end{array}
$$
where the inequality is due to:  $\inf_{\pmb{x}^{\prime}\in \mathcal{X}}F(\pmb{x}^{\prime},\mathcal{A}_{\pmb{y}}(\mathcal{S}))\leq \inf_{\pmb{x}^{\prime}\in \mathcal{X}}\sup_{\pmb{y}^{\prime}\in \mathcal{Y}}F(\pmb{x}^{\prime},\pmb{y}^{\prime})$
The first term is the primal generalization gap. And notice that the third term is analogous to the contrast side of the primal generalization gap. So both of them can be bounded by  $\epsilon_{gen}^{pr}(\boldsymbol{x}_{ave}^T)$ .
While the second term is the strong primal-dual empirical risk referring to the proof of Thm. 7 (see Appendix E.2).
Overall we can get the excess primal population risk, almost the same with the strong primal-dual population risk for SC-SC case (see Appendix E.3) except for a  $\sqrt{2}$ -times factor in the argument stability error. And our bound analysis for the excess primal population risk is consistent with strong primal-dual population risk (see Remark 6 below Thm. 3).
In this paper, we include two experiments including solving the AUC problem on svmguide and w5a by the decentralized SGD to verify the conclusions for the Convex-Concave case and the generative adversarial network training for the Nonconvex-Nonconcave Case on MNIST.
I.1 General setup.
Different from the stability and generalization analysis of the way in [19], we need to deal with learning in a decentralized manner. In our experiments, we denote the total number of clients as  $N_{c}$  and  $S = \{S_{1}, S_{2}, \dots, S_{N_{c}}\}$  as the set of samples, where  $S_{i}$  represents the observations stored in the  $i$ -th client. And we let  $N_{S_{i}}$  denote the size of  $S_{i}$ . We follow the same experimental setting as outlined in [13, 19] to build a neighboring/perturbing dataset  $S'$ , which is constructed by individually changing one observation on each node. That is to say, for each  $S_{i}$ ,  $S_{i}'$  is constructed by randomly changing one element in  $S_{i}$ . Then, we deploy the totally same sub-model on each client and initialize them to the same starting point. Then, each sub-model is trained on its local data. After each iteration, each sub-model is communicated with some other clients as per a predefined communication topology. To evaluate the distance between two models trained on  $S$  and  $S'$ , after finishing training, we obtain an ensemble model by averaging all sub-models collected from all clients.
I.2 Detailed implementations.
For the AUC problem, we get two model squences  $\{(w,v)\}$  and  $\{(w',v')\}$ . Then, we calculate the Euclidean distance  $\Delta = (||w - w'||_2^2 + ||v - v'||_2^2)^{1/2}$ . For training each model on each client, the algorithm we used in our experiment is SOLAM [41], which is the SGDA designed for the minimax AUC problem. We repeat the experiments 10 times to report the average results as well as the standard deviation.
For the generative adversarial learning problem, we just take the vanilla GAN structure, of which the generator and the discriminator comprise 4 fully connected layers, respectively. The leaky ReLU is taken before the output layer. Following [19], we ignore all forms of regularization such as the weight decay or dropout, and data augmentation tricks. We take 3 different seeds and 3 different ways to construct  $S'$ , which means changing different observations (total 9 runs). To evaluate the model distance, we also take the Euclidean distance between the generator and discriminator separately.
Our implementation is highly based on the two source codes[23].
$^{2}$ https://github.com/zhenhuan-yang/minimax-stability
<sup>3</sup>https://github.com/Raiden-Zhu/Generalization-of-DSGD