SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| I I _ {1} + I I _ {1} ^ {\prime} \| \leq \frac {1}{m} \sum_ {i = 1} ^ {m - m _ {0}} \left[ \eta_ {t} ^ {m a x} \left\| \left( \begin{array}{c} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) - \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, j _ {t} (i)}) \\ \nabla_ {\boldsymbol {y}} f _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, j _ {t} (i)}) - \nabla_ {\boldsymbol {y}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) \end{array} \right) \right\| \right] \\ + \frac {1}{m} \sum_ {i = m - m _ {0} + 1} ^ {m} \left[ \eta_ {t} ^ {m a x} \left\| \left( \begin{array}{c} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, n}) - \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, n}) \\ \nabla_ {\boldsymbol {y}} f _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, n}) - \nabla_ {\boldsymbol {y}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, n}) \end{array} \right) \right\| \right] \\ \leq \frac {1}{m} \left[ \sum_ {i = 1} ^ {m} \eta_ {t} ^ {m a x} L \left\| \left( \begin{array}{c} \boldsymbol {x} _ {i} ^ {t} - \boldsymbol {x} ^ {t} \\ \boldsymbol {y} _ {i} ^ {t} - \boldsymbol {y} ^ {t} \end{array} \right) \right\| \right] \\ \leq \frac {\eta_ {t} ^ {m a x} L}{m} \sum_ {i = 1} ^ {m} \left\| \left( \begin{array}{c} \boldsymbol {x} _ {i} ^ {t} - \boldsymbol {x} ^ {t} \\ \boldsymbol {y} _ {i} ^ {t} - \boldsymbol {y} ^ {t} \end{array} \right) \right\| \\ \leq \frac {\eta_ {t} ^ {m a x} L}{\sqrt {m}} \left[ \sum_ {i = 1} ^ {m} \| \boldsymbol {x} _ {i} ^ {t} - \boldsymbol {x} ^ {t} \| ^ {2} + \| \boldsymbol {y} _ {i} ^ {t} - \boldsymbol {y} ^ {t} \| ^ {2} \right] ^ {1 / 2} \\ \leq 2 \eta_ {t} ^ {\max } L G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s} \\ \end{array}
$$
And it is the same as the term  $II_2 + II_2'$  that  $\| II_2 + II_2' \| \leq 2\eta_t^{max} LG \sum_{s=0}^{t-1} \eta_s^{max} \lambda^{t-1-s}$ . Taking expectation on the choice of each local sample, we can get:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{c} \boldsymbol {x} ^ {t + 1} - \dot {\boldsymbol {x}} ^ {t + 1} \\ \boldsymbol {y} ^ {t + 1} - \dot {\boldsymbol {y}} ^ {t + 1} \end{array} \right) \right\| \\ \leq \sum_ {m _ {0} = 0} ^ {m} C _ {m} ^ {m _ {0}} (1 - \frac {1}{n}) ^ {m - m _ {0}} (\frac {1}{n}) ^ {m _ {0}} \left( \right.\left(1 - \eta_ {t} ^ {m i n} \frac {L \mu}{L + \mu}\right) \mathbb {E} _ {\mathcal {A}} \left\|\left(\begin{array}{c}\boldsymbol {x} ^ {t} - \dot {\boldsymbol {x}} ^ {t}\\\boldsymbol {y} ^ {t} - \dot {\boldsymbol {y}} ^ {t}\end{array}\right)\right\| + \frac {m _ {0}}{m} 2 \eta_ {t} ^ {m a x} G \\ \left. + 4 \eta_ {t} ^ {\max } L G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s}\right) \\ \leq \left(1 - \eta_ {t} ^ {m i n} \frac {L \mu}{L + \mu}\right) \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{c} \boldsymbol {x} ^ {t} - \dot {\boldsymbol {x}} ^ {t} \\ \boldsymbol {y} ^ {t} - \dot {\boldsymbol {y}} ^ {t} \end{array} \right) \right\| + 4 \eta_ {t} ^ {m a x} L G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {m a x} \lambda^ {t - 1 - s} \\ + \sum_ {m _ {0} = 0} ^ {m} C _ {m} ^ {m _ {0}} (1 - \frac {1}{n}) ^ {m - m _ {0}} (\frac {1}{n}) ^ {m _ {0}} \frac {m _ {0}}{m} 2 \eta_ {t} ^ {\max } G \\ = \left(1 - \eta_ {t} ^ {\min } \frac {L \mu}{L + \mu}\right) \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{l} \boldsymbol {x} ^ {t} - \dot {\boldsymbol {x}} ^ {t} \\ \boldsymbol {y} ^ {t} - \dot {\boldsymbol {y}} ^ {t} \end{array} \right) \right\| + 4 \eta_ {t} ^ {\max } L G \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {\max } \lambda^ {t - 1 - s} + \frac {2 \eta_ {t} ^ {\max } G}{n} \tag {8} \\ \end{array}
$$
where the last inequality is due to the property of binomial coefficient that  $C_m^{m_0} \cdot \frac{m_0}{m} = C_{m-1}^{m_0 - 1}$ . Recursively applying the above inequality, we can get:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{c} \mathcal {A} _ {\boldsymbol {x}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {x}} (\mathcal {S} ^ {\prime}) \\ \mathcal {A} _ {\boldsymbol {y}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {y}} (\mathcal {S} ^ {\prime}) \end{array} \right) \right\| \\ \leq \frac {2 G}{n} \sum_ {k = 0} ^ {T - 1} \eta_ {k} ^ {\max } \prod_ {s = k + 1} ^ {T - 1} \left(1 - \eta_ {s} ^ {\min } \frac {L \mu}{L + \mu}\right) + 4 G L \sum_ {k = 1} ^ {T - 1} \left(\eta_ {k} ^ {\max } \sum_ {s = 0} ^ {k - 1} \eta_ {s} ^ {\max } \lambda^ {k - 1 - s}\right) \prod_ {j = k + 1} ^ {T - 1} \left(1 - \eta_ {j} ^ {\min } \frac {L \mu}{L + \mu}\right) \tag {9} \\ \end{array}
$$
where the initial difference is zero that  $X^0 = 0$  and  $\pmb{Y}^{0} = 0$ . And we use the last iterate as the output of Algorithm  $\mathcal{A}$ , i.e.,  $(\mathcal{A}_{\pmb{x}}(\mathcal{S}),\mathcal{A}_{\pmb{y}}(\mathcal{S})) = (\pmb{x}^{T},\pmb{y}^{T})$ .
For case a. when  $\eta_{\pmb{x},t}$  and  $\eta_{\pmb{y},t}$  are fixed, we further get:
$$
\begin{array}{l} \epsilon_ {s t a} ^ {\arg} (\mathcal {A}) \leq \frac {2 G}{n} \sum_ {k = 0} ^ {T - 1} \eta^ {\max } \left(1 - \eta^ {\min } \frac {L \mu}{L + \mu}\right) ^ {T - k - 1} + 4 G L \sum_ {k = 1} ^ {T - 1} \left(\eta^ {\max } \sum_ {s = 0} ^ {k - 1} \eta^ {\max } \lambda^ {k - 1 - s}\right) \left(1 - \eta^ {\min } \frac {L \mu}{L + \mu}\right) ^ {T - k - 1} \\ \leq \left(4 \eta^ {m a x} G L \eta^ {m a x} \frac {1}{1 - \lambda} + \frac {2 \eta^ {m a x} G}{n}\right) \sum_ {k = 0} ^ {T - 1} (1 - \eta^ {m i n} \frac {L \mu}{L + \mu}) ^ {k} \\ \leq 2 G \frac {L + \mu}{\eta^ {m i n} L \mu} (\frac {2 (\eta^ {m a x}) ^ {2} L}{1 - \lambda} + \frac {\eta^ {m a x}}{n}) \\ \end{array}
$$
Further when  $\eta^{max} = \eta^{min} \triangleq \eta$ , the argument stability turns out to be
$$
\epsilon_ {s t a} ^ {a r g} (\mathcal {A}) \leq 2 G \frac {L + \mu}{L \mu} (\frac {2 \eta L}{1 - \lambda} + \frac {1}{n})
$$
For case b. when  $\eta_t^{max} = \frac{1}{\mu(t + 1)^c}$  and  $\eta_t^{min} = \frac{1}{\mu(t + 1)}, c \leq 1$ . Since  $1 \pm a \leq \exp \{a\}$ , we have:
$$
\begin{array}{l} \prod_ {j = k + 1} ^ {T - 1} \left(1 - \eta_ {j} ^ {\min } \frac {L \mu}{L + \mu}\right) = \prod_ {j = k + 1} ^ {T - 1} \left(1 - \frac {1}{j + 1} \frac {L}{L + \mu}\right) \\ \leq \prod_ {j = k + 1} ^ {T - 1} \exp \left\{- \frac {1}{j + 1} \frac {L}{L + \mu} \right\} \\ = \exp \left\{\sum_ {j = k + 1} ^ {T - 1} - \frac {1}{j + 1} \frac {L}{L + \mu} \right\} \\ \leq \exp \left\{- \frac {L}{L + \mu} (\ln T - \ln (k + 1)) \right\} \\ = \left(\frac {k + 1}{T}\right) ^ {\frac {L}{L + \mu}} \\ \end{array}
$$
Then back to inequality (9), we can simplify it as:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left\| \left( \begin{array}{c} \mathcal {A} _ {\boldsymbol {x}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {x}} (\mathcal {S} ^ {\prime}) \\ \mathcal {A} _ {\boldsymbol {y}} (\mathcal {S}) - \mathcal {A} _ {\boldsymbol {y}} (\mathcal {S} ^ {\prime}) \end{array} \right) \right\| \\ \leq \frac {2 G}{n} \sum_ {k = 0} ^ {T - 1} \frac {1}{\mu (k + 1) ^ {c}} \left(\frac {k + 1}{T}\right) ^ {\frac {L}{L + \mu}} + 4 G L \sum_ {k = 1} ^ {T - 1} \left(\frac {1}{\mu (k + 1) ^ {c}} \sum_ {s = 0} ^ {k - 1} \frac {1}{\mu (s + 1) ^ {c}} \lambda^ {k - 1 - s}\right) \left(\frac {k + 1}{T}\right) ^ {\frac {L}{L + \mu}} \\ \leq \frac {2 G}{\mu n T ^ {\frac {L}{L + \mu}}} \sum_ {k = 0} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} + \frac {4 G L}{\mu^ {2} T ^ {\frac {L}{L + \mu}}} \sum_ {k = 1} ^ {T - 1} \frac {1}{(k + 1) ^ {c - \frac {L}{L + \mu}}} \frac {C _ {\lambda}}{k ^ {c}} \\ \end{array}
$$
E.2 Proof of the Empirical Risk
Then we are going to present the optimization error of D-SGDA in the SC-SC condition.
Theorem 7 (Optimization error). Under assumption 1,2,3, each local function is  $f_{i}$  is  $\mu_{\mathbf{x}}SC - \mu_{\mathbf{y}}SC$ . We further bound the restriction set by  $\sup_{\mathbf{x}\in \mathcal{X}}\| \mathbf{x}\| \leq C_{\mathbf{x}}$  and  $\sup_{\mathbf{y}\in \mathcal{Y}}\| \mathbf{y}\| \leq C_{\mathbf{y}}$ , then we have the strong primal-dual empirical risk over the dataset  $S$  on the average output in  $T$  iterations as following:
a. When learning rates  $\eta_{\pmb{x},t}$  and  $\eta_{\pmb{y},t}$  are fixed,
$$
\Delta_ {S} ^ {s} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}) \leq \frac {C _ {\boldsymbol {x}} ^ {2} + C _ {\boldsymbol {y}} ^ {2}}{2 \eta^ {m i n} T} + \eta^ {m a x} G ^ {2} + \frac {4 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G L \eta^ {\max}}{1 - \lambda} + \frac {2 (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}}) G}{\sqrt {T}}.
$$
b. When learning rates are varying that  $\eta_{\pmb{x},t} = \frac{1}{\mu_{\pmb{x}}(t + 1)^{c_{\pmb{x}}}}$  and  $\eta_{\pmb{y},t} = \frac{1}{\mu_{\pmb{y}}(t + 1)^{c_{\pmb{y}}}}$
$$
\Delta_ {\mathcal {S}} ^ {s} (\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y} _ {a v e} ^ {T}) \leq \frac {2 G (C _ {\boldsymbol {x}} + C _ {\boldsymbol {y}})}{\sqrt {T}} + T _ {\boldsymbol {x}} + T _ {\boldsymbol {y}} + T _ {m a x}.
$$
where  $\alpha \in \{\pmb {x},\pmb {y}\}$ $k_{min} = \min \{c_x,c_y\}$  and  $\mu = \min \{\mu_{\pmb{x}},\mu_{\pmb{y}}\}$
$$
T _ {\alpha} = \left\{ \begin{array}{c c} \frac {G ^ {2}}{2 \mu_ {\alpha}} \frac {1 + \ln T}{T} & c _ {\alpha} = 1 \\ \frac {G ^ {2}}{2 \mu_ {\alpha} (1 - c _ {\alpha}) T ^ {c _ {\alpha}}} & 0 <   c _ {\alpha} <   1 \end{array} ; \right. T _ {m a x} = \left\{ \begin{array}{c c} \frac {4 G L C _ {\lambda} (C _ {x} + C _ {y}) \ln T}{\mu T} & k _ {m i n} = 1 \\ \frac {4 G L C _ {\lambda} (C _ {x} + C _ {y})}{\mu (1 - k _ {m i n}) T ^ {k _ {m i n}}} & 0 <   k _ {m i n} <   1 \end{array} . \right.
$$
Proof of Theorem 7. First, we should notice that when each  $f_{i}$  owns the property of  $\mu_{\pmb{x}}$ -strong convexity and  $\mu_{\pmb{y}}$ -strong concavity, then the linear summation should inherit the properties, i.e.,  $F_{S}(\pmb{x},\pmb{y}) = \frac{1}{m}\sum_{i = 1}^{m}\frac{1}{n}\sum_{l = 1}^{n}f_{i}(\pmb{x},\pmb{y};\xi_{i,l})$  is  $\mu_{\pmb{x}}$ -strongly convex on  $\pmb{x}$  and  $\mu_{\pmb{y}}$ -strongly concave on  $\pmb{y}$ .
First observe the update rule of  $\pmb{x}$  (see Algorithm 1), for any given  $\pmb{x} \in \mathcal{X}$ :
$$
\begin{array}{l} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} \right\| ^ {2} \\ \leq \| \boldsymbol {x} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \eta_ {\boldsymbol {x}, t} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \boldsymbol {x} \| ^ {2} \\ = \| \boldsymbol {x} ^ {t} - \boldsymbol {x} \| ^ {2} + \| \frac {1}{m} \sum_ {i = 1} ^ {m} \eta_ {\boldsymbol {x}, t} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) \| ^ {2} - 2 \langle \boldsymbol {x} ^ {t} - \boldsymbol {x}, \frac {1}{m} \sum_ {i = 1} ^ {m} \eta_ {\boldsymbol {x}, t} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) \rangle \tag {10} \\ \end{array}
$$
where for the first inequality, we can recall how we transform in the proof for stability (see Appendix E.1) that  $x^{t + 1} = (X^{t + 1})^T\frac{\mathbb{1}_m}{m} = (\boldsymbol {W}\boldsymbol {X}^t -\eta_{\boldsymbol {x},t}\nabla_{\boldsymbol{x}}\boldsymbol {f}(\boldsymbol {X}^t,\boldsymbol {Y}^t;\boldsymbol {\xi}^t))^T\frac{\mathbb{1}_m}{m} = \boldsymbol {x}^t - \eta_{\boldsymbol{x},t}\nabla_{\boldsymbol{x}}\boldsymbol {f}(\boldsymbol {X}^t,\boldsymbol {Y}^t;\boldsymbol {\xi}^t)^T\frac{\mathbb{1}_m}{m}$  regardless of the projection.
For the second term, we have:
$$
\begin{array}{l} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \eta_ {\pmb {x}, t} \nabla_ {\pmb {x}} f _ {i} (\pmb {x} _ {i} ^ {t}, \pmb {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) \right\| ^ {2} \\ = \frac {\eta_ {\pmb {x} , t} ^ {2}}{m ^ {2}} \left[ \sum_ {i = 1} ^ {m} \| \nabla_ {\pmb {x}} f _ {i} (\pmb {x} _ {i} ^ {t}, \pmb {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) \| ^ {2} + \sum_ {i \neq k} \langle \nabla_ {\pmb {x}} f _ {i} (\pmb {x} _ {i} ^ {t}, \pmb {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}), \nabla_ {\pmb {x}} f _ {k} (\pmb {x} _ {k} ^ {t}, \pmb {y} _ {k} ^ {t}; \xi_ {k, j _ {t} (k)}) \rangle \right] \\ \leq \frac {\eta_ {\boldsymbol {x} , t} ^ {2}}{m ^ {2}} \left(m G ^ {2} + (m ^ {2} - m) G ^ {2}\right) \\ = \eta_ {x, t} ^ {2} G ^ {2} \tag {11} \\ \end{array}
$$
We decompose the third term in association with the empirical function that:
$$
\begin{array}{l} 2 \eta_ {\boldsymbol {x}, t} \left\langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) \right\rangle \\ = 2 \eta_ {\boldsymbol {x}, t} \langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) \rangle + 2 \eta_ {\boldsymbol {x}, t} \langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) \rangle \\ \leq 2 \eta_ {\boldsymbol {x}, t} \left(F _ {\mathcal {S}} (\boldsymbol {x}, \boldsymbol {y} ^ {t}) - F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t})\right) - \eta_ {\boldsymbol {x}, t} \mu_ {\boldsymbol {x}} \| \boldsymbol {x} - \boldsymbol {x} ^ {t} \| ^ {2} \\ + 2 \eta_ {\boldsymbol {x}, t} \left\langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} F _ {S} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) \right\rangle \tag {12} \\ \end{array}
$$
where the last inequality is due to the strong convexity of the empirical function  $F_{S}$  on parameter  $x$ . Then we combine the inequalities (10) (11) (12) and we can get:
$$
\begin{array}{l} 2 \eta_ {\pmb {x}, t} (F _ {\mathcal {S}} (\pmb {x} ^ {t}, \pmb {y} ^ {t}) - F _ {\mathcal {S}} (\pmb {x}, \pmb {y} ^ {t})) \leq (1 - \eta_ {\pmb {x}, t} \mu_ {\pmb {x}}) \| \pmb {x} - \pmb {x} ^ {t} \| ^ {2} - \| \pmb {x} ^ {t + 1} - \pmb {x} \| ^ {2} + \eta_ {\pmb {x}, t} ^ {2} G ^ {2} \\ + 2 \eta_ {\boldsymbol {x}, t} \left\langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) \right\rangle \tag {13} \\ \end{array}
$$
For case a. when learning rates  $\eta_{\pmb{x},t}$  and  $\eta_{\pmb{y},t}$  are fixed and we write them as  $\eta_{\pmb{x}}$  and  $\eta_{\pmb{y}}$  respectively. Taking summation over the above inequality from  $t = 0$  to  $t = T$  and using the concavity of the empirical function on  $\pmb{y}$ :
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} 2 \eta_ {\boldsymbol {x}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) - F _ {\mathcal {S}} (\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}) \\ \leq \sum_ {t = 0} ^ {T - 1} 2 \eta_ {\boldsymbol {x}} \left(F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) - F _ {\mathcal {S}} \left(\boldsymbol {x}, \boldsymbol {y} ^ {t}\right)\right) \\ \leq \left(1 - \eta_ {\boldsymbol {x}} \mu_ {\boldsymbol {x}}\right) \| \boldsymbol {x} \| ^ {2} + \eta_ {\boldsymbol {x}} ^ {2} G ^ {2} T + 2 \eta_ {\boldsymbol {x}} \sum_ {t = 0} ^ {T - 1} \langle \boldsymbol {x} - \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) \rangle \\ \leq \left(1 - \eta_ {\boldsymbol {x}} \mu_ {\boldsymbol {x}}\right) \| \boldsymbol {x} \| ^ {2} + \eta_ {\boldsymbol {x}} ^ {2} G ^ {2} T + 2 \eta_ {\boldsymbol {x}} \sum_ {t = 0} ^ {T - 1} \langle \boldsymbol {x} ^ {t}, \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) - \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}) \rangle \\ + 2 \eta_ {\boldsymbol {x}} \| \boldsymbol {x} \| \left\| \sum_ {t = 0} ^ {T - 1} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right)\right) \right\| \\ \end{array}
$$
Then we take expectation on the randomness of the algorithm (notice that we do not take expectation on the randomness of dataset) on both sides of above inequality and choose the infinity of  $\pmb{x}$  on the left side since the above inequality holds for any  $\pmb{x} \in \mathcal{X}$ :
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} 2 \eta_ {\boldsymbol {x}} \mathbb {E} _ {\mathcal {A}} \left[ F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) - \inf  _ {\boldsymbol {x} \in \mathcal {X}} F _ {\mathcal {S}} \left(\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}\right) \right] \\ \leq (1 - \eta_ {\boldsymbol {x}} \mu_ {\boldsymbol {x}}) C _ {\boldsymbol {x}} ^ {2} + \eta_ {\boldsymbol {x}} ^ {2} G ^ {2} T + 2 \eta_ {\boldsymbol {x}} \underbrace {\sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathcal {A}} \langle \boldsymbol {x} ^ {t} , \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} (\boldsymbol {x} ^ {t} , \boldsymbol {y} ^ {t}) - \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} (\boldsymbol {x} _ {i} ^ {t} , \boldsymbol {y} _ {i} ^ {t} ; \xi_ {i , j _ {t} (i)}) \rangle} _ {I} \\ + 2 C _ {\boldsymbol {x}} \eta_ {\boldsymbol {x}} \underbrace {\mathbb {E} _ {\mathcal {A}} \left\| \sum_ {t = 0} ^ {T - 1} \left(\frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t} , \boldsymbol {y} _ {i} ^ {t} ; \xi_ {i , j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t} , \boldsymbol {y} ^ {t}\right)\right) \right\|} _ {I I} \tag {14} \\ \end{array}
$$
For term  $I$ , it is decomposed as:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\boldsymbol {j} _ {t}} \left\langle \boldsymbol {x} ^ {t}, \nabla_ {\boldsymbol {x}} F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) - \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) \right\rangle \\ = \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\boldsymbol {j} _ {t}} \left(\boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \frac {1}{n} \sum_ {l = 1} ^ {n} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, l}\right) - \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, j _ {t} (i)}\right) \right\rangle \\ + \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\boldsymbol {j} _ {t}} \left\langle \boldsymbol {x} ^ {t}, \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \frac {1}{m} \sum_ {i = 1} ^ {m} \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right) \right\rangle \\ \stackrel {(i)} {\leq} 2 C _ {\boldsymbol {x}} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathcal {A}} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}; \xi_ {i, j _ {t} (i)}\right) - \nabla_ {\boldsymbol {x}} f _ {i} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {y} _ {i} ^ {t}; \xi_ {i, j _ {t} (i)}\right)\right) \right\| \\ \leq 2 C _ {\boldsymbol {x}} \sum_ {t = 0} ^ {T - 1} \frac {L}{m} \mathbb {E} _ {\mathcal {A}} \left[ \sum_ {i = 1} ^ {m} \left(\left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} _ {i} ^ {t} \right\| ^ {2} + \left\| \boldsymbol {y} ^ {t} - \boldsymbol {y} _ {i} ^ {t} \right\| ^ {2}\right) ^ {1 / 2} \right] \\ \stackrel {(i i)} {\leq} 4 C _ {\boldsymbol {x}} G L \sum_ {t = 0} ^ {T - 1} \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {m a x} \lambda^ {t - 1 - s} \\ \end{array}
$$
where inequality  $(i)$  is because of the fact that taking expectation on the randomness of  $j_{t}$  is equal to the empirical result on the dataset  $S$  and inequality  $(ii)$  is due to Cauchy-Schwarz inequality and Lemma 2.
Then for the term  $II$ , we decompose its corresponding quadratic one:
where equality  $(a)$  owes to that taking expectation on the randomness of  $j_{t}$  or  $j_{t'}$  equals to the empirical risk and inequality  $(b)$  is due to the Lipschitz smoothness of each  $f_{i}$  and Cauchy-Schwarz inequality.
Combining above inequalities of term  $I$  and  $II$  into the inequality (14) and we can summarize as:
$$
\begin{array}{l} \sum_ {t = 0} ^ {T - 1} 2 \eta_ {\boldsymbol {x}} \mathbb {E} _ {\mathcal {A}} [ F _ {\mathcal {S}} (\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}) - \inf _ {\boldsymbol {x} \in \mathcal {X}} F _ {\mathcal {S}} (\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}) ] \\ \leq (1 - \eta_ {\boldsymbol {x}} \mu_ {\boldsymbol {x}}) C _ {\boldsymbol {x}} ^ {2} + \eta_ {\boldsymbol {x}} ^ {2} G ^ {2} T + 4 \eta_ {\boldsymbol {x}} C _ {\boldsymbol {x}} G L \sum_ {t = 0} ^ {T - 1} \sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {m a x} \lambda^ {t - 1 - s} \\ + 2 C _ {\boldsymbol {x}} \eta_ {\boldsymbol {x}} \sqrt {4 G ^ {2} T + 4 G ^ {2} L ^ {2} \sum_ {t \neq t ^ {\prime}} \left(\sum_ {s = 0} ^ {t - 1} \eta_ {s} ^ {m a x} \lambda^ {t - 1 - s}\right) \left(\sum_ {s = 0} ^ {t ^ {\prime} - 1} \eta_ {s} ^ {m a x} \lambda^ {t ^ {\prime} - 1 - s}\right)} \\ \end{array}
$$
Dividing both sides by  $T$  and we can get:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathcal {A}} \left[ F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) \right] - \mathbb {E} _ {\mathcal {A}} \left[ \inf  _ {\boldsymbol {x} \in \mathcal {X}} F _ {\mathcal {S}} \left(\boldsymbol {x}, \boldsymbol {y} _ {a v e} ^ {T}\right) \right] \\ \leq \frac {(1 - \eta_ {\boldsymbol {x}} \mu_ {\boldsymbol {x}}) C _ {\boldsymbol {x}} ^ {2}}{2 \eta_ {\boldsymbol {x}} T} + \frac {\eta_ {\boldsymbol {x}} G ^ {2}}{2} + \frac {4 C _ {\boldsymbol {x}} G L \eta^ {\mathrm {m a x}}}{1 - \lambda} + \frac {2 C _ {\boldsymbol {x}} G}{\sqrt {T}} \\ \end{array}
$$
And we can get the other-hand result in the same symmetric way:
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {A}} \left[ \sup  _ {\boldsymbol {y} \in \mathcal {Y}} F _ {\mathcal {S}} \left(\boldsymbol {x} _ {a v e} ^ {T}, \boldsymbol {y}\right) \right] - \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\mathcal {A}} \left[ F _ {\mathcal {S}} \left(\boldsymbol {x} ^ {t}, \boldsymbol {y} ^ {t}\right) \right] \\ \leq \frac {(1 - \eta_ {\pmb {y}} \mu_ {\pmb {y}}) C _ {\pmb {y}} ^ {2}}{2 \eta_ {\pmb {y}} T} + \frac {\eta_ {\pmb {y}} G ^ {2}}{2} + \frac {4 C _ {\pmb {y}} G L \eta^ {\mathrm {m a x}}}{1 - \lambda} + \frac {2 C _ {\pmb {y}} G}{\sqrt {T}} \\ \end{array}
$$
Combining above two inequalities we can get the result:
$$
\Delta_ {S} ^ {s} (\pmb {x} _ {a v e} ^ {T}, \pmb {y} _ {a v e} ^ {T}) \leq \frac {C _ {\pmb {x}} ^ {2} + C _ {\pmb {y}} ^ {2}}{2 \eta^ {m i n} T} + \eta^ {m a x} G ^ {2} + \frac {4 (C _ {\pmb {x}} + C _ {\pmb {y}}) G L \eta^ {\max}}{1 - \lambda} + \frac {2 (C _ {\pmb {x}} + C _ {\pmb {y}}) G}{\sqrt {T}}
$$
Then we come to the case b. when  $\eta_{\pmb{x},t} = \frac{1}{\mu_{\pmb{x}}\cdot(t + 1)^{c_{\pmb{x}}}}$  and  $\eta_{\pmb{y},t} = \frac{1}{\mu_{\pmb{y}}\cdot(t + 1)^{c_{\pmb{y}}}}$ . Back to the inequality (13), we simplify it into: