SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Ron Bekkerman, Mikhail Bilenko, and John Langford. Scaling up machine learning: Parallel and distributed approaches. Cambridge University Press, 2011.
[2] Aleksandr Beznosikov, Gesualdo Scutari, Alexander Rogozin, and Alexander Gasnikov. Distributed saddle-point problems under data similarity. Advances in Neural Information Processing Systems, 34:8172-8184, 2021.
[3] Aleksandr Beznosikov, Pavel Dvurechenskii, Anastasiia Koloskova, Valentin Samokhin, Sebastian U Stich, and Alexander Gasnikov. Decentralized local stochastic extra-gradient for variational inequalities. Advances in Neural Information Processing Systems, 35:38116-38133, 2022.
[4] Olivier Bousquet and Andr 'e Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[5] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning, 3(1):1-122, 2011.
[6] Lesi Chen, Boyuan Yao, and Luo Luo. Faster stochastic algorithms for minimax optimization under polyak- $\{\backslash \mathrm{L}\}$  ojasiewicz condition. Advances in Neural Information Processing Systems, 35:13921-13932, 2022.
[7] Lesi Chen, Haishan Ye, and Luo Luo. A simple and efficient stochastic algorithm for decentralized nonconvex-strongly-concave minimax optimization. arXiv preprint arXiv:2212.02387, 2022.
[8] Andre Elisseeff, Theodoros Evgeniou, Massimiliano Pontil, and Leslie Pack Kaelbing. Stability of randomized learning algorithms. Journal of Machine Learning Research, 6(1), 2005.
[9] Farzan Farnia and Asuman Ozdaglar. Train simultaneously, generalize better: Stability of gradient-based minimax learners. In International Conference on Machine Learning, pages 3174-3185. PMLR, 2021.
[43] Xin Zhang, Zhuqing Liu, Jia Liu, Zhengyuan Zhu, and Songtao Lu. Taming communication and sample complexities in decentralized policy evaluation for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:18825-18838, 2021.
[44] Tongtian Zhu, Fengxiang He, Lan Zhang, Zhengyang Niu, Mingli Song, and Dacheng Tao. Topology-aware generalization of decentralized sgd. In International Conference on Machine Learning, pages 27479-27503. PMLR, 2022.
A Decentralized Optimization 14
B Notation 14
C Important Lemmas 15
D Proof of the Connection 17
E Proof in the Strongly-Convex-Strongly-Concave Case 19
E.1 Proof of Stability 19
E.2 Proof of the Empirical Risk 23
E.3 Proof of Strong/Weak Primal-Dual Population Risk 28
F Proof in the Convex-Concave Case 30
F.1 Proof of Stability 30
F.2 Proof of the Empirical Risk 30
F.3 Proof of Weak Primal-Dual Population Risk 30
G Proof in Nonconvex-Nonconcave Case 31
G.1 Important Lemmas 31
G.2 Proof of Stability 32
H Primal Metric 33
I Additional Experiments 35
I.1 General setup. 35
I.2 Detailed implementations. 35
A Decentralized Optimization
ring
Figure 3: Visualization of different typologies. Note that the lines without arrows mean undirected, such as the fully connected, the grid, the ring, and the star graph. While lines with arrows denote the direction, which shows how information flows from one node to another. Our study objective is undirected graph in this paper, with the exponential graph an illustrative example of directed graph.
1.  $\pmb{x} \in \mathbb{R}^{d_x}$  and  $\pmb{y} \in \mathbb{R}^{d_y}$ , we use  $\left( \begin{array}{c} \pmb{x} \\ \pmb{y} \end{array} \right) \in \mathbb{R}^{d_x + d_y}$  to denote the concatenation.
2.  $\mathbf{X}^t \triangleq (\mathbf{x}_1^t, \dots, \mathbf{x}_m^t)^T \in \mathbb{R}^{m \times d_x}; \mathbf{Y}^t \triangleq (\mathbf{y}_1^t, \dots, \mathbf{y}_m^t)^T \in \mathbb{R}^{m \times d_y}$ ; and we use  $[\mathbf{X}, \mathbf{Y}]^t \in \mathbb{R}^{m \times (d_x + d_y)}$  to denote the concatenation  $\left[ \begin{pmatrix} \mathbf{x}_1^t \\ \mathbf{y}_1^t \end{pmatrix}, \begin{pmatrix} \mathbf{x}_2^t \\ \mathbf{y}_2^t \end{pmatrix}, \dots, \begin{pmatrix} \mathbf{x}_m^t \\ \mathbf{y}_m^t \end{pmatrix} \right]^T$ .
3. We use  $\pmb{\xi}^t$  to denote the samples collected in the  $t$ -th iteration, i.e.,  $\{\xi_{1,j_t(1)},\dots,\xi_{m,j_t(m)}\}$ .
4. We denote the gradients in the  $t$ -th iteration as follows:
$$
\nabla_ {\boldsymbol {x}} \boldsymbol {f} (\boldsymbol {X} ^ {t}, \boldsymbol {Y} ^ {t}; \boldsymbol {\xi} ^ {t}) \triangleq \left(\nabla_ {\boldsymbol {x}} f _ {1} (\boldsymbol {x} _ {1} ^ {t}, \boldsymbol {y} _ {1} ^ {t}; \xi_ {1, j _ {t} (1)}), \dots , \nabla_ {\boldsymbol {x}} f _ {m} (\boldsymbol {x} _ {m} ^ {t}, \boldsymbol {y} _ {m} ^ {t}; \xi_ {m, j _ {t} (m)})\right) ^ {T} \in \mathbb {R} ^ {m \times d _ {\boldsymbol {x}}}
$$
$$
\nabla_ {\boldsymbol {y}} \boldsymbol {f} (\boldsymbol {X} ^ {t}, \boldsymbol {Y} ^ {t}; \boldsymbol {\xi} ^ {t}) \triangleq \left(\nabla_ {\boldsymbol {y}} f _ {1} \left(\boldsymbol {x} _ {1} ^ {t}, \boldsymbol {y} _ {1} ^ {t}; \xi_ {1, j _ {t} (1)}\right), \dots , \nabla_ {\boldsymbol {y}} f _ {m} \left(\boldsymbol {x} _ {m} ^ {t}, \boldsymbol {y} _ {m} ^ {t}; \xi_ {m, j _ {t} (m)}\right)\right) ^ {T} \in \mathbb {R} ^ {m \times d _ {\boldsymbol {y}}}
$$
$$
\left( \begin{array}{c} \nabla_ {\boldsymbol {x}} \boldsymbol {f} (\boldsymbol {X} ^ {t}, \boldsymbol {Y} ^ {t}; \boldsymbol {\xi} ^ {t}) \\ \nabla_ {\boldsymbol {y}} \boldsymbol {f} (\boldsymbol {X} ^ {t}, \boldsymbol {Y} ^ {t}; \boldsymbol {\xi} ^ {t}) \end{array} \right) \triangleq \left[ \left( \begin{array}{c} \nabla_ {\boldsymbol {x}} f _ {1} (\boldsymbol {x} _ {1} ^ {t}, \boldsymbol {y} _ {1} ^ {t}; \xi_ {1, j _ {t} (1)}) \\ \nabla_ {\boldsymbol {y}} f _ {1} (\boldsymbol {x} _ {1} ^ {t}, \boldsymbol {y} _ {1} ^ {t}; \xi_ {1, j _ {t} (1)}) \end{array} \right), \dots , \left( \begin{array}{c} \nabla_ {\boldsymbol {x}} f _ {m} (\boldsymbol {x} _ {m} ^ {t}, \boldsymbol {y} _ {m} ^ {t}; \xi_ {m, j _ {t} (m)}) \\ \nabla_ {\boldsymbol {y}} f _ {m} (\boldsymbol {x} _ {m} ^ {t}, \boldsymbol {y} _ {m} ^ {t}; \xi_ {m, j _ {t} (m)}) \end{array} \right) \right] ^ {T} \in \mathbb {R} ^ {m \times (d _ {\boldsymbol {x}} + d _ {\boldsymbol {y}})}.
$$
5. We use  $\mathbf{P} \in \mathbb{R}^{m \times m}$  to denote the matrix whose elements are all  $\frac{1}{m}$ ,  $\mathbb{I}$  to denote the identity matrix, and  $\mathbb{1}_m \in \mathbb{R}^m$  to denote the vector with all elements equal to 1.
C Important Lemmas
In this section, we provide some important lemmas as fundamentals of the following proof.
Lemma 1. We define  $G_{g,\eta}$  as
$$
G _ {g, \eta} \left( \begin{array}{c} \boldsymbol {x} \\ \boldsymbol {y} \end{array} \right) = \left( \begin{array}{c} P _ {\mathcal {X}} (\boldsymbol {x} - \eta_ {\boldsymbol {x}} \nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y})) \\ P _ {\mathcal {Y}} (\boldsymbol {y} + \eta_ {\boldsymbol {y}} \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y})) \end{array} \right)
$$
with  $\eta^{max} \triangleq \max \{\eta_{x}, \eta_{y}\}$  and  $\eta^{min} \triangleq \min \{\eta_{x}, \eta_{y}\}$ . Under the assumption that  $g$  is  $L$ -Lipschitz smooth, then we have:
a.  $G_{g,\eta}$  is  $(1 + \eta^{max}L)$ -expansive.
b. When  $g$  is  $\mu_{\mathbf{x}}$ -strongly-convex w.r.t.  $\mathbf{x}$  and  $\mu_{\mathbf{y}}$ -strongly-concave w.r.t.  $\mathbf{y}$ , letting  $\frac{L + \mu}{2} (\eta^{max})^2 \leq \eta^{min} \leq \frac{L + \mu}{2} \frac{1}{L\mu}$  where  $\mu \triangleq \min \{\mu_{\mathbf{x}}, \mu_{\mathbf{y}}\}$ , then  $G_{g,\eta}$  is  $(1 - \eta^{min} \frac{L\mu}{L + \mu})$ -expansive.
Proof. For case a. where we do not require strong convexity or strong concavity, we have:
$$
\begin{array}{l} \left\| G _ {g, \eta} \left( \begin{array}{c} \boldsymbol {x} \\ \boldsymbol {y} \end{array} \right) - G _ {g, \eta} \left( \begin{array}{c} \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} ^ {\prime} \end{array} \right) \right\| \leq \left\| \left( \begin{array}{c} \boldsymbol {x} - \boldsymbol {x} ^ {\prime} - \eta_ {\boldsymbol {x}} \left(\nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime})\right) \\ \boldsymbol {y} - \boldsymbol {y} ^ {\prime} - \eta_ {\boldsymbol {y}} \left(\nabla_ {\boldsymbol {y}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y})\right) \end{array} \right) \right\| \\ = \left\| \left( \begin{array}{c} \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} - \boldsymbol {y} ^ {\prime} \end{array} \right) \right\| + \left\| \left( \begin{array}{c} \eta_ {\boldsymbol {x}} (\nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime})) \\ \eta_ {\boldsymbol {y}} (\nabla_ {\boldsymbol {y}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y})) \end{array} \right) \right\| \\ \leq \left(1 + \eta^ {m a x} L\right) \left\| \left( \begin{array}{c} \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} - \boldsymbol {y} ^ {\prime} \end{array} \right) \right\| \\ \end{array}
$$
When function  $g$  is further  $\mu_{\pmb{x}}$ -strongly convex and  $\mu_{\pmb{y}}$ -strongly concave in case b., thus we have:
$$
\begin{array}{l} \left\| G _ {g, \eta} \left( \begin{array}{c} \boldsymbol {x} \\ \boldsymbol {y} \end{array} \right) - G _ {g, \eta} \left( \begin{array}{c} \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} ^ {\prime} \end{array} \right) \right\| ^ {2} \\ = \left\| \left( \begin{array}{c} \boldsymbol {x} - \boldsymbol {x} ^ {\prime} - \eta_ {\boldsymbol {x}} (\nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime})) \\ \boldsymbol {y} - \boldsymbol {y} ^ {\prime} - \eta_ {\boldsymbol {y}} (\nabla_ {\boldsymbol {y}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y})) \end{array} \right) \right\| ^ {2} \\ = \left\| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} - \eta_ {\boldsymbol {x}} \left(\nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime})\right) \right\| ^ {2} + \left\| \boldsymbol {y} - \boldsymbol {y} ^ {\prime} - \eta_ {\boldsymbol {y}} \left(\nabla_ {\boldsymbol {y}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y})\right) \right\| ^ {2} \\ = \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| ^ {2} + \eta_ {\boldsymbol {x}} ^ {2} \| \nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right)) \| ^ {2} - 2 \eta_ {\boldsymbol {x}} \langle \boldsymbol {x} - \boldsymbol {x} ^ {\prime}, \nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) \rangle \\ + \| \boldsymbol {y} - \boldsymbol {y} ^ {\prime} \| ^ {2} + \eta_ {\boldsymbol {y}} ^ {2} \| \nabla_ {\boldsymbol {y}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y}) \| ^ {2} - 2 \eta_ {\boldsymbol {y}} \langle \boldsymbol {y} - \boldsymbol {y} ^ {\prime}, \nabla_ {\boldsymbol {y}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y}) \rangle \tag {4} \\ \end{array}
$$
Recalling the co-coercivity of a L-Lipschitz smooth and  $\mu$ -strongly convex function  $f(x)$  that [13]:
$$
\langle \nabla f (\boldsymbol {x}) - \nabla f (\boldsymbol {y}), \boldsymbol {x} - \boldsymbol {y} \rangle \geq \frac {1}{L + \mu} \| \nabla f (\boldsymbol {x}) - \nabla f (\boldsymbol {y}) \| ^ {2} + \frac {L \mu}{L + \mu} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2}
$$
So we can get:
$$
\begin{array}{l} \langle \boldsymbol {x} - \boldsymbol {x} ^ {\prime}, \nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) \rangle + \langle \boldsymbol {y} - \boldsymbol {y} ^ {\prime}, \nabla_ {\boldsymbol {y}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y}) \rangle \\ = \left\langle \left( \begin{array}{c} \nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}) \\ \nabla_ {\boldsymbol {y}} g (\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y}) \end{array} \right), \left( \begin{array}{c} \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} - \boldsymbol {y} ^ {\prime} \end{array} \right) \right\rangle \\ \geq \frac {1}{L + \mu} \left\| \left( \begin{array}{l} \nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) \\ \nabla_ {\boldsymbol {y}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y}) \end{array} \right) \right\| ^ {2} + \frac {L \mu}{L + \mu} \left\| \left( \begin{array}{l} \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} - \boldsymbol {y} ^ {\prime} \end{array} \right) \right\| ^ {2} \tag {5} \\ = \frac {1}{L + \mu} \| \nabla_ {\boldsymbol {x}} g (\boldsymbol {x}, \boldsymbol {y}) - \nabla_ {\boldsymbol {x}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) \| ^ {2} + \frac {1}{L + \mu} \| \nabla_ {\boldsymbol {y}} g \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) - \nabla_ {\boldsymbol {y}} g (\boldsymbol {x}, \boldsymbol {y}) \| ^ {2} \\ + \frac {L \mu}{L + \mu} \| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \| ^ {2} + \frac {L \mu}{L + \mu} \| \boldsymbol {y} - \boldsymbol {y} ^ {\prime} \| ^ {2} \\ \end{array}
$$
Combining above inequalities (4) and (5), we can get:
$$
\left\| G _ {g, \eta} \left( \begin{array}{c} \boldsymbol {x} \\ \boldsymbol {y} \end{array} \right) - G _ {g, \eta} \left( \begin{array}{c} \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} ^ {\prime} \end{array} \right) \right\| \leq (1 - \eta^ {m i n} \frac {L \mu}{L + \mu}) \left\| \left( \begin{array}{c} \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \\ \boldsymbol {y} - \boldsymbol {y} ^ {\prime} \end{array} \right) \right\|
$$
when  $\frac{L + \mu}{2} (\eta^{max})^2\leq \eta^{min}\leq \frac{L + \mu}{2}\frac{1}{L\mu}$  satisfies.
Lemma 2. Letting  $\mathbf{P} \in \mathbb{R}^{m \times m}$  denote the matrix whose elements are all  $\frac{1}{m}$ , following the update rule of D-SGDA (see Algorithm 1), we have:
$$
\left\| (\mathbb {I} - \mathbf {P}) [ \boldsymbol {X}, \boldsymbol {Y} ] ^ {t} \right\| \leq 2 \sqrt {m} G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s}
$$
where  $\eta_s^{max} = \max \{\eta_{\boldsymbol{x},s},\eta_{\boldsymbol{y},s}\}$
Proof. The Lipschitz continuity (see Assumption 1) implies that for any  $\pmb{x} \in \mathcal{X}$  and  $\pmb{y} \in \mathcal{Y}$  on a given sample  $\xi$ , the gradient value is bounded by  $G$ , i.e.,  $\left\| \left( \begin{array}{c} \eta_{\pmb{x},t} \nabla_{\pmb{x}} \pmb{f}(\pmb{X}^t, \pmb{Y}^t; \pmb{\xi}^t) \\ -\eta_{\pmb{y},t} \nabla_{\pmb{y}} \pmb{f}(\pmb{X}^t, \pmb{Y}^t; \pmb{\xi}^t) \end{array} \right) \right\| \leq \max \{\eta_{\pmb{x},t}, \eta_{\pmb{y},t}\} \sqrt{m} G$ . Following the proof of Lemma 8 in [32], we can get the result, which can be expressed as  $\left[ \sum_{i=1}^{m} \| \pmb{x}_i^t - \pmb{x}^t \|^2 + \| \pmb{y}_i^t - \pmb{y}^t \|^2 \right]^{1/2} \leq 2\sqrt{m} G \sum_{s=0}^{t-1} \eta_s^{max} \lambda^{t-1-s}$ .
Lemma 3. When  $0 < \lambda < 1$
$$
\sum_ {j = 0} ^ {t - 1} \frac {\lambda^ {t - 1 - j}}{(j + 1) ^ {k}} \leq \frac {C _ {\lambda}}{t ^ {k}}, k > 0; C _ {\lambda} \triangleq \frac {(k / e) ^ {k}}{\lambda (\ln \frac {1}{\lambda}) ^ {k}} + \frac {2 e ^ {- 1}}{\lambda \ln \frac {1}{\lambda}} + \frac {2 ^ {k}}{\lambda \ln \frac {1}{\lambda}}
$$
$$
\sum_ {j = 0} ^ {t - 1} \frac {\lambda^ {t - 1 - j}}{(j + 1) ^ {k}} = \lambda^ {t - 1} + \sum_ {j = 1} ^ {t - 1} \frac {\lambda^ {t - 1 - j}}{(j + 1) ^ {k}} \leq \lambda^ {t - 1} + \int_ {1} ^ {t} \frac {\lambda^ {t - 1 - x}}{x ^ {k}} d x = \lambda^ {t - 1} + \lambda^ {t - 1} \int_ {1} ^ {t} \frac {\lambda^ {- x}}{x ^ {k}} d x
$$
whereas for the integral we have:
$$
\begin{array}{l} \int_ {1} ^ {t} \frac {\lambda^ {- x}}{x ^ {k}} d x = \int_ {1} ^ {\frac {t}{2}} \frac {\lambda^ {- x}}{x ^ {k}} d x + \int_ {\frac {t}{2}} ^ {t} \frac {\lambda^ {- x}}{x ^ {k}} d x \leq \lambda^ {- \frac {t}{2}} \int_ {1} ^ {\frac {t}{2}} \frac {1}{x ^ {k}} d x + \frac {1}{(\frac {t}{2}) ^ {k}} \int_ {\frac {t}{2}} ^ {t} \lambda^ {- x} d x \\ \int_ {1} ^ {\frac {t}{2}} \frac {1}{x ^ {k}} d x \leq \left\{ \begin{array}{l l} l n \frac {t}{2} & k = 1 \\ \frac {(\frac {t}{2}) ^ {1 - k}}{1 - k} & 0 <   k <   1 \end{array} \right. \\ \end{array}
$$
$$
\sum_ {j = 0} ^ {t - 1} \frac {\lambda^ {t - 1 - j}}{(j + 1) ^ {k}} \leq \left\{ \begin{array}{l l} \lambda^ {t - 1} + \lambda^ {\frac {t}{2} - 1} l n \frac {t}{2} + \frac {1}{t ^ {k}} \frac {2 ^ {k}}{\lambda l n \frac {1}{\lambda}} & k = 1 \\ \lambda^ {t - 1} + \lambda^ {\frac {t}{2} - 1} t ^ {1 - k} + \frac {1}{t ^ {k}} \frac {2 ^ {k}}{\lambda l n \frac {1}{\lambda}} & 0 <   k <   1 \end{array} \right.
$$
When  $0 < k < 1$ ,  $\sum_{j=0}^{t-1} \frac{\lambda^{t-1-j}}{(j+1)^k} \leq \frac{1}{t^k} \left( \lambda^{t-1} t^k + \lambda^{\frac{t}{2}-1} t + \frac{2^k}{\lambda \ln \frac{1}{\lambda}} \right)$ , where  $\lambda^{t-1} t^k \leq \frac{(k/e)^k}{\lambda (\ln \frac{1}{\lambda})^k}$ ,  $\lambda^{\frac{t}{2}-1} t \leq \frac{2e^{-1}}{\lambda \ln \frac{1}{\lambda}}$ . So we can define  $C_\lambda = \frac{(k/e)^k}{\lambda (\ln \frac{1}{\lambda})^k} + \frac{2e^{-1}}{\lambda \ln \frac{1}{\lambda}} + \frac{2^k}{\lambda \ln \frac{1}{\lambda}}$  that  $\sum_{j=0}^{t-1} \frac{\lambda^{t-1-j}}{(j+1)^k} \leq \frac{C_\lambda}{t^k}$ . And for the case  $k = 1$  we can roughly consider the logarithm as constants so our discussion can be put together into the above category as  $k = 1$ .
Remark 10. Considering function  $h(\lambda) = \frac{1}{\lambda \ln \frac{1}{\lambda}}$ , it monotonically decreases in interval  $(0, \frac{1}{e})$  and monotonically increases in interval  $(\frac{1}{e}, 1)$  with minimal value of  $e$ . Analogously for the function  $g(\lambda) = \frac{1}{\lambda (\ln \frac{1}{\lambda})^k}$ , which monotonically decreases in interval  $(0, \frac{1}{e^k})$  and monotonically increases in interval  $(\frac{1}{e^k}, 1)$  with minimal value of  $(\frac{e}{k})^k$ . Therefore, when  $k = 1$ ,  $C_\lambda$  is bounded by  $\mathcal{O}\left(\frac{1}{\lambda \ln \frac{1}{\lambda}}\right)$ ; otherwise, when  $\lambda \to 1$ ,  $C_\lambda$  is bounded by  $\mathcal{O}\left(\frac{1}{\lambda \ln \frac{1}{\lambda}}\right)$ ; when  $\lambda \to 0$ ,  $C_\lambda$  is bounded by  $\mathcal{O}\left(\frac{1}{\lambda (\ln \frac{1}{\lambda})^k}\right)$ .
D Proof of the Connection
In this section, we will first figure out a fundamental lemma that illustrates the structure of dataset in decentralized setting. Then we will provide proof for the connection between argument stability and primal-dual generalization gap.
Lemma 4. Denoting dataset as  $\mathcal{S} = \{\mathcal{S}_1,\dots ,\mathcal{S}_m\}$ , and denoting each local samples stored in  $\mathcal{S}_i$  as  $\mathcal{S}_i = \{\xi_{i,l_i}\}_{l_i = 1,\dots,n}$ . Then we have the decomposition equation of the empirical function:
$$
F _ {S} (\boldsymbol {x}, \boldsymbol {y}) = \frac {1}{n ^ {m}} \sum_ {l _ {1} = 1} ^ {n} \dots \sum_ {l _ {m} = 1} ^ {n} \frac {1}{m} \sum_ {i = 1} ^ {m} f _ {i} (\boldsymbol {x}, \boldsymbol {y}; \xi_ {i, l _ {i}})
$$
$$
\begin{array}{l} \frac {1}{n ^ {m}} \sum_ {l _ {1} = 1} ^ {n} \dots \sum_ {l _ {m} = 1} ^ {n} \frac {1}{m} \sum_ {i = 1} ^ {m} f _ {i} (\boldsymbol {x}, \boldsymbol {y}; \xi_ {i, l _ {i}}) \\ = \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\frac {1}{n ^ {m - 1}} \sum_ {l _ {1} = 1} ^ {n} \dots \sum_ {l _ {i - 1} = 1} ^ {n} \sum_ {l _ {i + 1} = 1} ^ {n} \dots \sum_ {l _ {m} = 1} ^ {n} \frac {1}{n} \sum_ {l _ {i} = 1} ^ {n} f _ {i} (\boldsymbol {x}, \boldsymbol {y}; \xi_ {i, l _ {i}})\right) \\ = \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\frac {1}{n ^ {m - 1}} \sum_ {l _ {1} = 1} ^ {n} \dots \sum_ {l _ {i - 1} = 1} ^ {n} \sum_ {l _ {i + 1} = 1} ^ {n} \dots \sum_ {l _ {m} = 1} ^ {n} F _ {\mathcal {S} _ {i}} (\boldsymbol {x}, \boldsymbol {y})\right) \\ = \frac {1}{m} \sum_ {i = 1} ^ {m} F _ {\mathcal {S} _ {i}} (\boldsymbol {x}, \boldsymbol {y}) \\ = F _ {S} (\boldsymbol {x}, \boldsymbol {y}) \\ \end{array}
$$
Proof of Theorem 1. We let  $\mathcal{S} = \{\mathcal{S}_1, \dots, \mathcal{S}_m\}$  where  $\mathcal{S}_i = \{\xi_{i,l}\}_{1 \leq l \leq n}$  and  $\mathcal{S}' = \{\mathcal{S}_1', \dots, \mathcal{S}_m'\}$  where  $\mathcal{S}_i' = \{\xi_{i,l}'\}_{1 \leq l \leq n}$  be two different datasets while  $\mathcal{S}_i$  and  $\mathcal{S}_i'$  are drawn from the same distribution  $\mathcal{D}_i$ . Further we define  $S^{(l)} = \{\mathcal{S}_1^{(l_1)}, \mathcal{S}_2^{(l_2)}, \dots, \mathcal{S}_m^{(l_m)}\}$  where  $l = (l_1, l_2, \dots, l_m)$  and  $\mathcal{S}_i^{(l_i)} = \{\xi_{i,1}, \dots, \xi_{i,l_i - 1}, \xi_{i,l_i}', \xi_{i,l_i + 1}, \dots, \xi_{i,n}\}$ . So we can say that there is at most one different sample in each local dataset between  $S$  and  $S^{(l)}$ . For function  $F: \mathcal{X} \times \mathcal{Y} \mapsto \mathbb{R}$ , we denote  $y^*(\boldsymbol{x}) = \arg \max_{\boldsymbol{y} \in \mathcal{Y}} F(\boldsymbol{x}, \boldsymbol{y})$ .
For case a. the weak primal-dual generalization gap  $\epsilon_{gen}^{w}(\mathcal{A}_{\boldsymbol{x}}(\mathcal{S}),\mathcal{A}_{\boldsymbol{y}}(\mathcal{S})) = \Delta^{w}(\mathcal{A}_{\boldsymbol{x}}(\mathcal{S}),\mathcal{A}_{\boldsymbol{y}}(\mathcal{S})) - \Delta_{\mathcal{S}}^{w}(\mathcal{A}_{\boldsymbol{x}}(\mathcal{S}),\mathcal{A}_{\boldsymbol{y}}(\mathcal{S}))$ . We first make some adjustments to the definition: