SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma B.10. For any  $x, y \in \mathbb{R}^{n \times d}$ , and  $\Lambda$  the Laplacian of a connected graph, we have:
$$
2 \langle \pi x, y \rangle \leq \frac {1}{4} \| x \| _ {\Lambda} ^ {2} + 4 \chi_ {1} \| y \| _ {F} ^ {2}
$$
Proof. By property of the Laplacian of a connected graph, we have that  $\pi = (\Lambda^{+})^{1 / 2}\Lambda^{1 / 2}$ . Thus:
$$
2 \langle \pi x, y \rangle = 2 \left\langle \Lambda^ {1 / 2} x, \left(\Lambda^ {+}\right) ^ {1 / 2} y \right\rangle \tag {13}
$$
$$
\stackrel {(B. 5)} {\leq} \frac {1}{4} \| \Lambda^ {1 / 2} x \| _ {F} ^ {2} + 4 \| \left(\Lambda^ {+}\right) ^ {1 / 2} y \| _ {F} ^ {2} \tag {14}
$$
$$
\stackrel {(B. 7)} {\leq} \frac {1}{4} \| x \| _ {\Lambda} ^ {2} + 4 \chi_ {1} \| y \| _ {F} ^ {2} \tag {15}
$$
C Proof of the main result of this paper
We remind that we study the following dynamic:
$$
d x _ {t} ^ {i} = \eta (\tilde {x} _ {t} ^ {i} - x _ {t} ^ {i}) d t - \gamma \int_ {\Xi} \nabla F _ {i} (x _ {t} ^ {i}, \xi_ {i}) d N _ {t} ^ {i} (\xi_ {i}) - \alpha \sum_ {j, (i, j) \in \mathcal {E}} (x _ {t} ^ {i} - x _ {t} ^ {j}) d M _ {t} ^ {i j},
$$
$$
d \tilde {x} _ {t} ^ {i} = \eta (x _ {t} ^ {i} - \tilde {x} _ {t} ^ {i}) d t - \gamma \int_ {\Xi} \nabla F _ {i} (x _ {t} ^ {i}, \xi_ {i}) d N _ {t} ^ {i} (\xi_ {i}) - \tilde {\alpha} \sum_ {j, (i, j) \in \mathcal {E}} (x _ {t} ^ {i} - x _ {t} ^ {j}) d M _ {t} ^ {i j},
$$
which simplifies to:
$$
d x _ {t} ^ {i} = - \gamma \int_ {\Xi} \nabla F _ {i} (x _ {t} ^ {i}, \xi_ {i}) d N _ {t} ^ {i} (\xi_ {i}) - \alpha \sum_ {j, (i, j) \in \mathcal {E}} (x _ {t} ^ {i} - x _ {t} ^ {j}) d M _ {t} ^ {i j}
$$
if  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  is not applied. We also recall the main proposition, which we prove next:
Proposition C.1 (Convergence guarantees.). Assume that  $\{x_{t},\tilde{x}_{t}\}$  follow the dynamic Eq. 4 and that Assumption 3.2-3.3 are satisfied. Assume that  $\mathbf{1}\bar{x}_0 = x_0 = \tilde{x}_0$  and let  $T$  the total running time. Then:
- Non-accelerated setting, we pick  $\eta = 0$ ,  $\alpha = \tilde{\alpha} = \frac{1}{2}$  and set  $\chi = \chi_{1}$
- Acceleration  $(A^2 C_i D^2)$ , we set  $\eta = \frac{1}{2\sqrt{\chi_1 \chi_2}}$ ,  $\alpha = \frac{1}{2}$ ,  $\tilde{\alpha} = \frac{1}{2}\sqrt{\frac{\chi_1}{\chi_2}}$ , and  $\chi = \sqrt{\chi_1 \chi_2} \leq \chi_1$ .
- (strong-convexity) the Assumption 3.4 is satisfied, then  $\gamma \leq \frac{1}{16L(1 + \chi)}$  and
$^{1}\tilde{\mathcal{O}}$  -notation hides constants and polylogarithmic factors.
Proof. The core of the proof is to introduce the appropriate Lyapunov potentials  $\phi_k(t,X)$ , where  $X$  could be  $(x,\tilde{x})$  or  $x$  depending on whether we apply  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  or not. If we apply  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , we introduce the momentum matrix  $\mathcal{A} = \left( \begin{array}{cc} -\eta & \eta \\ \eta & -\eta \end{array} \right)$ . Then, by Ito's lemma, given that all the functions are smooth, and remembering that all Point-wise Poisson Processes  $N_t^i$  have unit intensity, that the  $M_t^{ij}$  have intensity  $\lambda^{ij}$  and that they are all independent, we obtain in a similar fashion to [12, 34]:
$$
\begin{array}{l} \phi_ {k} (T, X _ {T}) - \phi_ {k} (0, X _ {0}) = \int_ {0} ^ {T} \partial_ {t} \phi_ {k} (t, X _ {t}) + \underbrace {\langle A X _ {t} , \partial_ {X} \phi_ {k} (t , X _ {t}) \rangle} _ {\text {m o m e n t u m t e r m}} d t \\ + \sum_ {i = 1} ^ {n} \int_ {0} ^ {T} \underbrace {\int_ {\Xi} \phi_ {k} \left(t , X _ {t} - \gamma \left(\frac {\nabla \tilde {F} _ {i} (x _ {t} , \xi)}{\nabla \tilde {F} _ {i} (x _ {t} , \xi)}\right)\right) - \phi_ {k} (t , X _ {t})} _ {\text {v a r i a t i o n d u e t o e a c h i n d e p e n d e n t g r a d i e n t u p d a t e}} d t d \mathcal {P} (\xi) \\ + \sum_ {(i, j) \in \mathcal {E}} \int_ {0} ^ {T} \underbrace {\left[ \phi_ {k} \left(t , X _ {t} - \left( \begin{array}{c} \alpha (e _ {i} - e _ {j}) (e _ {i} - e _ {j}) ^ {\intercal} x _ {t} \\ \tilde {\alpha} (e _ {i} - e _ {j}) (e _ {i} - e _ {j}) ^ {\intercal} x _ {t}\right)\right) - \phi_ {k} (t , X _ {t}) \right] \lambda^ {i j}} _ {\text {v a r i a t i o n d u e t o e a c h i n d e p e n d e n t p 2 p c o m m u n i c a t i o n}} d t \\ + M _ {T}, \\ \end{array}
$$
where  $M_T$  is a martingale. In the case where  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  is not applied, we set  $\mathcal{A} = 0$  to remove the momentum term, and all updates are done only along  $x$  as there is no  $\tilde{x}$ . We remind that:
$$
\int_ {0} ^ {t} e ^ {\alpha u} d u = \frac {1}{\alpha} \left(e ^ {\alpha t} - 1\right) \tag {16}
$$
We now present our choice of potential for each cases:
- For the convex case in the non-accelerated setting, we introduce:
$$
\phi_ {1} (t, x) \triangleq A _ {t} \| \bar {x} - x ^ {*} \| ^ {2} + B _ {t} \| \pi x \| _ {F} ^ {2}.
$$
- For the convex case with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , we introduce:
$$
\phi_ {2} (t, x, \tilde {x}) \triangleq A _ {t} \| \bar {x} - x ^ {*} \| ^ {2} + B _ {t} \| \pi x \| _ {F} ^ {2} + \tilde {B} _ {t} \| \tilde {x} \| _ {\Lambda^ {+}}.
$$
- For the non-convex case in the non-accelerated setting, we introduce:
$$
\phi_ {3} (t, x) \triangleq A _ {t} d _ {f} (\bar {x}, x ^ {*}) + B _ {t} \| \pi x \| _ {F} ^ {2}.
$$
- For the non-convex case with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ , we introduce:
$$
\phi_ {4} (t, x) \triangleq A _ {t} d _ {f} (\bar {x}, x ^ {*}) + B _ {t} \| \pi x \| _ {F} ^ {2} + \tilde {B} _ {t} \| \tilde {x} \| _ {\Lambda^ {+}} ^ {2}.
$$
C.1 Some useful upper-bounds
As the same terms appear in several potentials, we now prepare some intermediary results which will be helpful for the proofs.
Study of the  $\| \bar{x} -x^{*}\|^{2}$  terms:
First, we study the variations in the  $\| \bar{x} - x^{*} \|^2$  term appearing in  $\phi_1$  and  $\phi_2$ . As the updates due to the communication are in the orthogonal of 1, it is only necessary to study the variations induced by the gradient steps. Thus, we define:
$$
\Delta x \triangleq \sum_ {i = 1} ^ {n} \| \overline {{x - \gamma \nabla \tilde {F} _ {i} (x , \xi)}} - x ^ {*} \| ^ {2} - \| \bar {x} - x ^ {*} \| ^ {2}
$$
We note that  $\overline{\nabla\tilde{F}_i(x,\xi)} = \frac{1}{n}\nabla F_i(x_i,\xi_i)$ , which, using  $\sum_{i}\nabla f_{i}(x^{*}) = 0$ , leads to:
$$
\begin{array}{l} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} [ \Delta x ] = \sum_ {i = 1} ^ {n} - \frac {2 \gamma}{n} \left\langle \bar {x} - x ^ {*}, \nabla f _ {i} \left(x _ {i}\right) \right\rangle + \frac {\gamma^ {2}}{n ^ {2}} \mathbb {E} _ {\xi_ {i}} \left[ \| \nabla F _ {i} \left(x _ {i}, \xi_ {i}\right) \| ^ {2} \right] (17) \\ = \sum_ {i = 1} ^ {n} - \frac {2 \gamma}{n} \left\langle \bar {x} - x ^ {*}, \nabla f _ {i} \left(x _ {i}\right) - \nabla f _ {i} \left(x ^ {*}\right) \right\rangle + \frac {\gamma^ {2}}{n ^ {2}} \mathbb {E} _ {\xi_ {i}} \left[ \| \nabla F _ {i} \left(x _ {i}, \xi_ {i}\right) \| ^ {2} \right] (18) \\ \stackrel {(3. 4), (B. 3)} {\leq} \frac {\gamma^ {2}}{n} \sigma^ {2} + \sum_ {i = 1} ^ {n} - \frac {2 \gamma}{n} \left\langle \bar {x} - x ^ {*}, \nabla f _ {i} \left(x _ {i}\right) - \nabla f _ {i} \left(x ^ {*}\right) \right\rangle + \frac {\gamma^ {2}}{n ^ {2}} \| \nabla f _ {i} \left(x _ {i}\right) \| ^ {2} (19) \\ \stackrel {(B. 2)} {=} \frac {\gamma^ {2}}{n} \sigma^ {2} + \sum_ {i = 1} ^ {n} - \frac {2 \gamma}{n} \left(d _ {f _ {i}} (\bar {x}, x ^ {*}) + d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) - d _ {f _ {i}} (\bar {x}, x _ {i})\right) + \frac {\gamma^ {2}}{n ^ {2}} \| \nabla f _ {i} (x _ {i}) \| ^ {2} (20) \\ \stackrel {(B. 4)} {\leq} \frac {\gamma^ {2}}{n} \sigma^ {2} + \sum_ {i = 1} ^ {n} - \frac {2 \gamma}{n} \left(d _ {f _ {i}} (\bar {x}, x ^ {*}) + d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) - d _ {f _ {i}} (\bar {x}, x _ {i})\right) \\ + \frac {2 \gamma^ {2}}{n ^ {2}} \| \nabla f _ {i} \left(x ^ {*}\right) - \nabla f _ {i} \left(x _ {i}\right) \| ^ {2} + \frac {2 \gamma^ {2}}{n ^ {2}} \| \nabla f _ {i} \left(x ^ {*}\right) \| ^ {2} (21) \\ \stackrel {(3. 4), (B. 1)} {\leq} \frac {\gamma^ {2}}{n} \sigma^ {2} + \frac {2 \gamma^ {2}}{n} \zeta^ {2} + \sum_ {i = 1} ^ {n} - \frac {2 \gamma}{n} \left(d _ {f _ {i}} (\bar {x}, x ^ {*}) + d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) - d _ {f _ {i}} (\bar {x}, x _ {i})\right) + \frac {4 L \gamma^ {2}}{n ^ {2}} d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) (22) \\ \stackrel {(B. 1)} {\leq} \frac {\gamma^ {2} \sigma^ {2}}{n} + \frac {2 \gamma^ {2}}{n} \zeta^ {2} - \gamma \mu \| \bar {x} - x ^ {*} \| ^ {2} + \frac {L \gamma}{n} \| \pi x \| _ {F} ^ {2} + \sum_ {i = 1} ^ {n} \left(- \frac {2 \gamma}{n} + \frac {4 L \gamma^ {2}}{n ^ {2}}\right) d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) (23) \\ \end{array}
$$
Study of the  $d_{f}(\bar{x},x^{*})$  terms:
Next, using the same reasoning as for  $\Delta x$ , we also need to only study the gradient updates in the non-convex setting for the first part of  $\phi_3, \phi_4$ . Thus, we set:
$$
\Delta f \triangleq \sum_ {i = 1} ^ {n} d _ {f} (\bar {x} - \gamma \frac {1}{n} \nabla F _ {i} \left(x _ {i}, \xi_ {i}\right), x ^ {*}) - d _ {f} (\bar {x}, x ^ {*}). \tag {24}
$$
First, it is useful to note that under Assumption 3.5, using (B.3), we have:
$$
\mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2} \leq n \sigma^ {2} + (1 + M) \sum_ {i = 1} ^ {n} \| \nabla f _ {i} \left(x _ {i}\right) \| ^ {2}. \tag {25}
$$
Then, using  $\sum_{i}\nabla f_{i}(x^{*}) = 0$  and the  $L$ -smoothness of  $f = \frac{1}{n}\sum_{i}f_{i}$ , we get:
$$
\begin{array}{l} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} [ \Delta f ] \stackrel {(B, 2)} {=} \sum_ {i = 1} ^ {n} \mathbb {E} [ d _ {f} (\bar {x} - \gamma \frac {1}{n} \nabla F _ {i} (x _ {i}, \xi_ {i}), \bar {x}) ] - \frac {\gamma}{n} \langle \nabla f _ {i} (x _ {i}), \nabla f (\bar {x}) \rangle (26) \\ \leq \sum_ {i = 1} ^ {n} \frac {1}{2 n ^ {2}} L \gamma^ {2} \mathbb {E} \left[ \| \nabla F _ {i} \left(x _ {i}, \xi_ {i}\right) \| ^ {2} \right] - \frac {\gamma}{n} \langle \nabla f _ {i} \left(x _ {i}\right), \nabla f (\bar {x}) \rangle (27) \\ \stackrel {(2 5)} {\leq} \frac {L \gamma^ {2}}{2 n} \sigma^ {2} - \gamma \| \nabla f (\bar {x}) \| ^ {2} + \sum_ {i = 1} ^ {n} \frac {M + 1}{2 n ^ {2}} L \gamma^ {2} \| \nabla f _ {i} (x _ {i}) \| ^ {2} - \sum_ {i = 1} ^ {n} \frac {\gamma}{n} \langle \nabla f _ {i} (x _ {i}) - \nabla f _ {i} (\bar {x}), \nabla f (\bar {x}) \rangle (28) \\ \stackrel {(B. 5)} {\leq} \frac {L \gamma^ {2}}{2 n} \sigma^ {2} + \frac {\gamma}{2 n} L ^ {2} \| \pi x \| _ {F} ^ {2} - \frac {\gamma}{2} \| \nabla f (\bar {x}) \| ^ {2} + \sum_ {i = 1} ^ {n} \frac {M + 1}{2 n ^ {2}} L \gamma^ {2} \| \nabla f _ {i} (x _ {i}) \| ^ {2} (29) \\ \end{array}
$$
As we also have:
$$
\sum_ {i = 1} ^ {n} \left\| \nabla f _ {i} \left(x _ {i}\right) \right\| ^ {2} \leq \sum_ {i = 1} ^ {n} 3 \left(\left\| \nabla f _ {i} \left(x _ {i}\right) - \nabla f _ {i} (\bar {x}) \right\| ^ {2} + \left\| \nabla f _ {i} (\bar {x}) - \nabla f (\bar {x}) \right\| ^ {2} + \left\| \nabla f (\bar {x}) \right\| ^ {2}\right) \tag {30}
$$
$$
\stackrel {(3. 5)} {\leq} 3 L ^ {2} \| \pi x \| _ {F} ^ {2} + 3 n \zeta^ {2} + 3 n (1 + P) \| \nabla f (\bar {x}) \| ^ {2} \tag {31}
$$
We get in the end:
$$
\begin{array}{l} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} [ \Delta f ] \leq \frac {L \gamma^ {2}}{2 n} \sigma^ {2} + \frac {3 (M + 1)}{2 n} \gamma^ {2} L \zeta^ {2} + \left(\frac {3 (M + 1)}{2 n ^ {2}} L ^ {3} \gamma^ {2} + \frac {\gamma}{2 n} L ^ {2}\right) \| \pi x \| _ {F} ^ {2} \\ + \left(\frac {3 (M + 1)}{2 n} (1 + P) L \gamma^ {2} - \frac {\gamma}{2}\right) \| \nabla f (\bar {x}) \| ^ {2} \tag {32} \\ \end{array}
$$
Remark C.2. As observed in (5), note that for both the terms  $\| \bar{x} - x^{*}\|^{2}$  and  $d_{f}(\bar{x},x^{*})$ , as we are considering  $\bar{x}$  and  $\frac{1}{n}\mathbf{11}^{\mathrm{T}}\pi = 0$ , Poisson updates from the communication process amount to zero. Moreover, as  $\frac{1}{n}\mathbf{11}^{\mathrm{T}}(x - \tilde{x}) = 0$ , the update from the momentum is also null for these terms.
Study of the  $\| \pi x\| _F^2$  terms:
We get from the Poisson updates for the gradient processes:
$$
\begin{array}{l} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \left[ \sum_ {i = 1} ^ {n} \| \pi (x - \gamma \nabla \tilde {F} _ {i} (x, \xi)) \| _ {F} ^ {2} - \| \pi x \| _ {F} ^ {2} \right] = - 2 \gamma \langle \pi x, \nabla F (x) \rangle + \gamma^ {2} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \pi \nabla \tilde {F} _ {i} (x, \xi) \| _ {F} ^ {2} \\ \leq - 2 \gamma \langle \pi x, \nabla F (x) \rangle + \gamma^ {2} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2}, \\ \end{array}
$$
and, using the definition of the Laplacian  $\Lambda$  (3.1), we get from the communication processes:
$$
\begin{array}{l} \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \left(\| \pi (x - \alpha (e _ {i} - e _ {j}) (e _ {i} - e _ {j}) ^ {\mathsf {T}} x) \| _ {F} ^ {2} - \| \pi x \| _ {F} ^ {2}\right) = - 2 \alpha \langle x, \Lambda x \rangle + \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \alpha^ {2} \| (e _ {i} - e _ {j}) (e _ {i} - e _ {j}) ^ {\mathsf {T}} x \| _ {F} ^ {2} \\ \stackrel {(B. 9)} {=} - 2 \alpha \langle x, \Lambda x \rangle + 2 \alpha^ {2} \| x \| _ {\Lambda} ^ {2} \\ = 2 \alpha (\alpha - 1) \| x \| _ {\Lambda} ^ {2}. \\ \end{array}
$$
Putting together both types of Poisson updates, we define:
$$
\begin{array}{l} \mathbb {E} [ \Delta_ {\pi} ] \triangleq - 2 \gamma \langle \pi x, \nabla F (x) \rangle + \gamma^ {2} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \pi \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2} + 2 \alpha (\alpha - 1) \| x \| _ {\Lambda} ^ {2} (33) \\ \stackrel {(B. 1 0)} {\leq} 4 \chi_ {1} \gamma^ {2} \| \nabla F (x) \| _ {F} ^ {2} + \left(\frac {1}{4} - 2 \alpha (1 - \alpha)\right) \| x \| _ {\Lambda} ^ {2} + \gamma^ {2} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2} (34) \\ \end{array}
$$
For  $\alpha = \frac{1}{2}$ , we get:
$$
\mathbb {E} \left[ \Delta_ {\pi} \right] \leq 4 \chi_ {1} \gamma^ {2} \| \nabla F (x) \| _ {F} ^ {2} - \frac {1}{4 \chi_ {1}} \| \pi x \| _ {F} ^ {2} + \gamma^ {2} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2} \tag {35}
$$
For  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  , we add the momentum term  $\langle \eta (\tilde{x} -x),2\pi x\rangle$  to define:
$$
\begin{array}{l} \Delta_ {\Lambda} ^ {1} \triangleq 2 \eta \langle \tilde {x}, \pi x \rangle - 2 \eta \| \pi x \| _ {F} ^ {2} - 2 \gamma \langle \pi x, \nabla F (x) \rangle + \gamma^ {2} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2} + 2 \alpha (\alpha - 1) \| x \| _ {\Lambda} ^ {2} (36) \\ \leq 2 \eta \langle \tilde {x}, \pi x \rangle - \frac {3}{2} \eta \| \pi x \| _ {F} ^ {2} + \frac {2}{\eta} \gamma^ {2} \| \nabla F (x) \| _ {F} ^ {2} - 2 \alpha (1 - \alpha) \| x \| _ {\Lambda} + \gamma^ {2} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2}. (37) \\ \end{array}
$$
Study of the  $\| \tilde{x}\|_{\Lambda +}^2$  terms:
These terms only appear in the Lyapunov potentials used when applying  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$ . From the Poisson updates and momentum, we get:
$$
\begin{array}{l} \Delta_ {\Lambda} ^ {2} \triangleq 2 \eta \langle x - \tilde {x}, \tilde {x} \rangle_ {\Lambda^ {+}} - 2 \gamma \langle \tilde {x}, \Lambda^ {+} \nabla \tilde {F} (x, \xi) \rangle + \gamma^ {2} \| \nabla \tilde {F} (x, \xi) \| _ {\Lambda^ {+}} ^ {2} - 2 \tilde {\alpha} \langle \pi x, \tilde {x} \rangle \\ + \tilde {\alpha} ^ {2} \sum_ {(i, j) \in \mathcal {E}} \lambda^ {i j} \| \left(e _ {i} - e _ {j}\right) \left(e _ {i} - e _ {j}\right) ^ {\mathsf {T}} x \| _ {\Lambda^ {+}} ^ {2}. \tag {38} \\ \end{array}
$$
Taking the expectation and using (B.9), (B.8), (B.7), (B.5) leads to:
$$
\begin{array}{l} \mathbb {E} \left[ \Delta_ {\Lambda} ^ {2} \right] \leq \chi_ {1} \eta \| \pi x \| _ {F} ^ {2} + \left(\frac {\eta}{2} - \eta\right) \| \tilde {x} \| _ {\Lambda^ {+}} ^ {2} + \frac {2}{\eta} \chi_ {1} \gamma^ {2} \| \pi \nabla F (x) \| _ {F} ^ {2} - 2 \tilde {\alpha} \langle \pi x, \tilde {x} \rangle \\ + 2 \chi_ {2} \tilde {\alpha} ^ {2} \| x \| _ {\Lambda} + \chi_ {1} \gamma^ {2} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} \| \nabla \tilde {F} (x, \xi) \| _ {F} ^ {2} \tag {39} \\ \end{array}
$$
C.2 Resolution: putting everything together
In this part, we combine the terms for each potential. We remind that with Assumption 3.4:
$$
\begin{array}{l} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\xi_ {i}} \left[ \| \nabla F _ {i} \left(x _ {i}, \xi_ {i}\right) \| ^ {2} \right] = n \sigma^ {2} + \sum_ {i = 1} ^ {n} \| \nabla f _ {i} \left(x _ {i}\right) \| ^ {2} (40) \\ \leq n \sigma^ {2} + \sum_ {i = 1} ^ {n} 2 \| \nabla f _ {i} \left(x ^ {*}\right) - \nabla f _ {i} \left(x _ {i}\right) \| ^ {2} + 2 \| \nabla f _ {i} \left(x ^ {*}\right) \| ^ {2} (41) \\ \leq n \sigma^ {2} + 2 n \zeta^ {2} + 4 L \sum_ {i = 1} ^ {n} d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) (42) \\ \end{array}
$$
Convex case, non-accelerated. We remind that
$$
\phi_ {1} (t, x, \tilde {x}) = A _ {t} \| \bar {x} - x ^ {*} \| ^ {2} + B _ {t} \| \pi x \| ^ {2}.
$$
Then, using (23), (35) and defining  $\Psi_{1} \triangleq \partial_{t}\phi_{1}(t,X_{t}) + \mathbb{E}[A_{t}\Delta x + B_{t}\Delta_{\pi}]$ , we have:
$$
\begin{array}{l} \Psi_ {1} \leq \| \bar {x} - x ^ {*} \| ^ {2} \left(A _ {t} ^ {\prime} - \mu \gamma A _ {t}\right) (43) \\ + \| \pi x \| ^ {2} \left(B _ {t} ^ {\prime} + \frac {L \gamma}{n} A _ {t} - \frac {1}{4 \chi_ {1}} B _ {t}\right) (44) \\ + \sum_ {i = 1} ^ {n} d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) \left(- \frac {2 \gamma}{n} A _ {t} + \frac {4 L \gamma^ {2}}{n ^ {2}} A _ {t} + 4 L \left(4 \chi_ {1} \gamma^ {2} + \gamma^ {2}\right) B _ {t}\right) (45) \\ + \left(\frac {\gamma^ {2} \sigma^ {2}}{n} + \frac {2 \gamma^ {2}}{n} \zeta^ {2}\right) A _ {t} + \left(n \gamma^ {2} \sigma^ {2} + 2 n \zeta^ {2} \left(4 \chi_ {1} \gamma^ {2} + \gamma^ {2}\right)\right) B _ {t} (46) \\ \end{array}
$$
We pick  $\alpha = \frac{1}{2}$ ,  $B_{t} = \frac{1}{n} A_{t}$ , with  $A_{t} = e^{-rt}$  (we denote by  $r$  the rate of the exponentials  $A_{t}, B_{t}$ ). Then (43), (44) imply:
$$
r \leq \min  (\mu \gamma , \frac {1}{4 \chi_ {1}} - L \gamma) \tag {47}
$$
As we want (45) to be negative, we have:
$$
- 1 + \gamma \left(\frac {2 L}{n} + 2 L \left(4 \chi_ {1} + 1\right)\right) \leq 0 \tag {48}
$$
$$
\gamma \leq \frac {1}{2 L \left(\frac {1}{n} + 4 \chi_ {1} + 1\right)} \tag {49}
$$
and taking  $\gamma \leq \frac{1}{2}\frac{1}{2L(3 + 4\chi_1 + 1)} = \frac{1}{16L(1 + \chi_1)}$  works. Now, as  $\frac{1}{4\chi_1} - L\gamma \geq \frac{3}{16\chi_1}$  and  $\mu\gamma \leq \frac{1}{16\chi_1}$ , we pick  $r = \mu\gamma$ . As we have:
$$
\mathbb {E} \left[ \phi_ {1} \left(T, x _ {T}\right) - \phi_ {1} \left(0, x _ {0}\right) \right] = \int_ {0} ^ {T} \left(A _ {t} ^ {\prime} \| \bar {x} _ {t} - x ^ {*} \| ^ {2} + B _ {t} ^ {\prime} \| \pi x _ {t} \| ^ {2} + A _ {t} \mathbb {E} _ {\xi_ {1}, \dots , \xi_ {n}} [ \Delta x ] + B _ {t} \mathbb {E} [ \Delta_ {\pi} ]\right) d t \tag {50}
$$
using (46) and (16) leads to:
$$
\mathbb {E} \| \bar {x} _ {t} - x ^ {*} \| ^ {2} \leq e ^ {- \gamma \mu t} \left(\| \bar {x} _ {0} - x ^ {*} \| ^ {2} + \frac {1}{n} \| \pi x _ {0} \| ^ {2}\right) + \frac {\gamma}{\mu} \left(\sigma^ {2} (\frac {1}{n} + 1) + 2 \zeta^ {2} (\frac {1}{n} + 4 \chi_ {1} + 1)\right) \tag {51}
$$
Convex case with  $\mathbf{A}^2\mathbf{C}\mathbf{i}\mathbf{D}^2$  We remind that
$$
\phi_ {2} (t, x, \tilde {x}) = A _ {t} \| \bar {x} - x ^ {*} \| ^ {2} + B _ {t} \| \pi x \| ^ {2} + \tilde {B} _ {t} \| \tilde {x} \| _ {\Lambda^ {+}}.
$$
Then, using (23), (37), (39) and defining  $\Psi_{2} \triangleq \partial_{t}\phi_{2}(t,X_{t}) + \mathbb{E}[A_{t}\Delta x + B_{t}\Delta_{\Lambda}^{1} + \tilde{B}_{t}\Delta_{\Lambda}^{2}]$ , we have:
$$
\begin{array}{l} \Psi_ {2} \leq \| \bar {x} - x ^ {*} \| ^ {2} \left(A _ {t} ^ {\prime} - \mu \gamma A _ {t}\right) (52) \\ + \| \pi x \| ^ {2} \left(B _ {t} ^ {\prime} + \frac {L \gamma}{n} A _ {t} - \frac {3}{2} \eta B _ {t} + \eta \chi_ {1} \tilde {B} _ {t}\right) (53) \\ + \| \tilde {x} \| _ {\Lambda^ {+}} ^ {2} \left(\tilde {B} _ {t} ^ {\prime} - \frac {\eta}{2} \tilde {B} _ {t}\right) (54) \\ + \| x \| _ {\Lambda} ^ {2} \left(2 \tilde {\alpha} ^ {2} \chi_ {2} \tilde {B} _ {t} - 2 \alpha (1 - \alpha) B _ {t}\right) (55) \\ + \left\langle \tilde {x}, \pi x \right\rangle \left(2 \eta B _ {t} - 2 \tilde {\alpha} \tilde {B} _ {t}\right) (56) \\ + \sum_ {i = 1} ^ {n} d _ {f _ {i}} \left(x ^ {*}, x _ {i}\right) \left(- \frac {2 \gamma}{n} A _ {t} + \frac {4 L \gamma^ {2}}{n ^ {2}} A _ {t} + 4 L \left(\frac {2 \gamma^ {2}}{\eta} + \gamma^ {2}\right) \left(B _ {t} + \chi_ {1} \tilde {B} _ {t}\right)\right) (57) \\ + \left(\frac {\gamma^ {2} \sigma^ {2}}{n} + \frac {2 \gamma^ {2}}{n} \zeta^ {2}\right) A _ {t} + \left(n \gamma^ {2} \sigma^ {2} + 2 n \zeta^ {2} \left(\gamma^ {2} + \frac {2 \gamma^ {2}}{\eta}\right)\right) \left(B _ {t} + \chi_ {1} \tilde {B} _ {t}\right) (58) \\ \end{array}
$$
Then, we assume  $\alpha = \frac{1}{2}$ ,  $\tilde{\alpha} = \frac{1}{2}\sqrt{\frac{\chi_1}{\chi_2}}$ ,  $\eta = \frac{1}{2\sqrt{\chi_1\chi_2}}$ ,  $B_{t} = \frac{1}{n} A_{t}$ ,  $\tilde{B}_{t} = \frac{1}{\chi_{1}} B_{t}$ ,  $A_{t} = e^{-rt}$  (we denote by  $r$  the rate of the exponentials  $A_{t}, B_{t}, \tilde{B}_{t}$ ), which satisfies (55) and (56). Then (52), (53), (54) imply:
$$
r \leq \min  (\mu \gamma , \frac {\eta}{2}, \frac {\eta}{2} - L \gamma) = \min  (\mu \gamma , \frac {\eta}{2} - L \gamma) \tag {59}
$$
As we want (57) to be negative, we have:
$$
- 1 + \gamma \left(\frac {2 L}{n} + 4 L \left(\frac {2}{\eta} + 1\right)\right) \leq 0 \tag {60}
$$
$$
\gamma \leq \frac {1}{2 L \left(\frac {1}{n} + \frac {4}{\eta} + 2\right)} \tag {61}
$$
and taking  $\gamma \leq \frac{1}{2L\left(6 + \frac{4}{\eta} + 2\right)} = \frac{1}{16L\left(1 + \sqrt{\chi_1\chi_2}\right)}$  works. Now, we have:
$$
\frac {\eta}{2} - L \gamma \geq \frac {1}{4 \sqrt {\chi_ {1} \chi_ {2}}} \left(1 - \frac {4 \sqrt {\chi_ {1} \chi_ {2}}}{1 6 (1 + \sqrt {\chi_ {1} \chi_ {2}})}\right) \geq \frac {3}{1 6 \sqrt {\chi_ {1} \chi_ {2}}} \tag {62}
$$
As  $\mu \gamma \leq \frac{\mu}{16L\left(1 + \sqrt{\chi_1\chi_2}\right)} \leq \frac{1}{16\sqrt{\chi_1\chi_2}}$ , taking  $r = \mu \gamma$  works. Finally, using (58) and (16) leads to:
$$
\mathbb {E} \| \bar {x} _ {t} - x ^ {*} \| ^ {2} \leq e ^ {- \gamma \mu t} \left(\| \bar {x} _ {0} - x ^ {*} \| ^ {2} + \frac {2}{n} \| \pi x _ {0} \| ^ {2}\right) + \frac {\gamma}{\mu} \left(\sigma^ {2} (\frac {1}{n} + 2) + 2 \zeta^ {2} (\frac {1}{n} + 8 \sqrt {\chi_ {1} \chi_ {2}} + 2)\right) \tag {63}
$$
Non-convex case, non-accelerated. We remind that:
$$
\phi_ {3} (t, x) = A _ {t} d _ {f} \left(\bar {x}, x ^ {*}\right) + B _ {t} \| \pi x \| ^ {2}
$$
Here, we pick  $\alpha = \frac{1}{2}, A_{t} = 1, B_{t} = \frac{L}{n} A_{t}$ . Thus,  $A_{t}' = B_{t}' = 0$ . Then, using (32), (35), (31), (25) we obtain:
$$
\begin{array}{l} \left. A _ {t} \mathbb {E} [ \Delta f ] + B _ {t} \mathbb {E} [ \Delta_ {\pi} ]\right) \leq \| \nabla f (\bar {x}) \| ^ {2} \left(- \frac {\gamma}{2} A _ {t} + \frac {3}{2 n} L \gamma^ {2} (M + 1) (P + 1) A _ {t} + 3 n \gamma^ {2} \left(4 \chi_ {1} + M + 1\right) (P + 1) B _ {t}\right) (64) \\ + \| \pi x \| ^ {2} \left(\frac {L ^ {2} \gamma}{2 n} (1 + \frac {3}{n} (M + 1) L \gamma) A _ {t} + 3 L ^ {2} \gamma^ {2} \left(4 \chi_ {1} + M + 1\right) B _ {t} - \frac {1}{4 \chi_ {1}} L \right. (65) \\ + \gamma^ {2} (\sigma^ {2} + 3 (M + 1) \zeta^ {2}) \left(\frac {L}{2 n} A _ {t} + n B _ {t}\right) + 1 2 n \chi_ {1} \gamma^ {2} \zeta^ {2} B _ {t} (66) \\ \end{array}
$$
Our goal is to use half of the negative term of (64) to cancel the positive ones, so that there remains at least  $-\frac{\gamma}{4} A_t \| \nabla f(\bar{x}) \|^2$  in the end. Thus, we want:
$$
3 L \gamma^ {2} \left(\frac {(M + 1) (P + 1)}{2 n} + (4 \chi_ {1} + M + 1) (P + 1)\right) \leq \frac {\gamma}{4} \tag {67}
$$