SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
U ^ {t + 1} = \left(\mathcal {W} ^ {2} + \frac {\alpha}{m} \mathcal {A} ^ {*} \mathcal {A} \left(Z ^ {\star} - U ^ {t} \left(U ^ {t}\right) ^ {\top}\right)\right) U ^ {t}. \tag {9}
$$
For the convergence of (9) towards low-rank matrices with strong generalization properties, we anticipate certain conditions to be imposed on the operator  $\mathcal{A}$ . The algorithmic mapping structure in (9) provides some insights in this regard. Since  $\mathcal{W}^2\mathcal{J} = \mathcal{J}$  (due to Assumption1(ii)), the linear network operator  $\mathcal{W}^2$  effectively functions as the identity map on matrices  $Z \in \mathbb{R}^{dm \times dm}$  with  $\mathrm{range}(Z) \subset \mathrm{range}(\mathcal{J})$ . This is in particular true for  $(d \times d)$  block) consensual matrices  $Z = \mathcal{J}Z\mathcal{J}$ , including the (augmented) ground-truth  $Z^{\star}$ , as defined in (7). This implies that to accomplish precise reconstructions of  $Z^{\star}$ , the operator  $\mathcal{A}$  ought to exhibit some RIP-like regularity. Postponing to Sec. 3.1 a more rigorous and comprehensive argument, we claim that the following property suffices.
Definition 3 (In-network RIP). The operator  $\mathcal{A}:\mathbb{R}^{md\times md}\to \mathbb{R}^N$  defined in (8) satisfies the in-network  $(\delta ,r)$ -RIP property with tolerance  $\Delta \geq 0$ , if
$$
(1 - \delta) \| \mathcal {J} Z \mathcal {J} \| _ {F} ^ {2} - \Delta \| Z - \mathcal {J} Z \mathcal {J} \| _ {F} ^ {2} \leq \| \mathcal {A} (Z) \| _ {2} ^ {2} \leq (1 + \delta) \| \mathcal {J} \mathcal {Z} \mathcal {J} \| _ {F} ^ {2} + \Delta \| Z - \mathcal {J} Z \mathcal {J} \| _ {F} ^ {2}, \tag {10}
$$
for any matrix  $Z \in \mathbb{R}^{md \times md}$  such that each of its  $d \times d$  blocks  $[Z]_{i,j}$  and its block-average  $\bar{Z} = \frac{1}{m^2} \sum_{i=1}^{m} \sum_{j=1}^{m} [Z]_{i,j}$  are of rank at most  $r$ .
The condition (10) reads as an "exact" RIP property of  $\mathcal{A}$  along (block) consensual directions, allowing for some "perturbation" in the form of consensus errors  $\| Z - \mathcal{J}Z\mathcal{J}\| _F^2$ . The following result shows that the tolerance error can be controlled by the network connectivity  $\rho$  (see Assumption [iv]).
Lemma 1. Suppose  $\bar{\mathcal{A}}$  satisfies the  $(\delta_{2r}, 2r)$ -RIP, and the gossip matrix  $W$  is chosen according to Assumption [7] Then, the augmented operator  $\mathcal{A}$  satisfies the in-network  $(2\delta_{2r}, r)$ -RIP with tolerance
$$
\Delta = \rho^ {2} \cdot \frac {4 m ^ {5} \left(1 + 2 \delta_ {2 r}\right)}{\delta_ {2 r}} \left(1 + \delta_ {2 r}\right). \tag {11}
$$
Clearly, the smaller  $\rho$ , the smaller  $\Delta$ , revealing an unexplored interplay between (potential) generalization properties and network characteristics. Since small  $\rho$  can be enforced also by employing multiple rounds of communications per iteration (see Sec. 3 for details), the communication complexity enters in the tradeoff equation. Our theory in the next section will quantify this interplay, revealing conditions and tuning recommendations to achieve fast convergence and strong generalization properties.
We are ready to state the convergence results of Algorithm [3]. Here we consider the overparametrized case  $r \geq 2r^{\star}$  while the other ranges of  $r$  are discussed in the supplementary material.
Theorem 1. Consider the matrix sensing problem (1), with augmented ground-truth  $Z^{\star}$ , under  $r \geq 2r^{\star}$ , and the measurement operator  $\bar{A}$  satisfying the  $(4(r^{\star} + 1),\delta) - RIP$ , with  $\delta \precsim \kappa^{-4}(r^{\star})^{-1 / 2}$ . Let  $\{U^t\}_t$  be the (augmented) sequence generated by Algorithm (3), under the following tuning: (i) the stepsize  $\alpha \precsim \kappa^{-4}\| \bar{X}\|^{-2}$ ; (ii) the gossip matrix  $W$  is chosen to satisfy Assumption with
$$
\rho \stackrel {\sim} {_ {\sim}} \frac {\delta^ {2}}{m ^ {6} \kappa^ {4} r ^ {\star}}; \tag {12}
$$
and (iii) the initialization  $U^0$  is chosen as  $U^0 = \mu U$ , where  $U \in \mathbb{R}^{md \times r}$  has i.i.d.  $\mathcal{N}(0, \sqrt{m / r})$  distributed entries, and  $\mu$  satisfies
$$
\mu^ {2} \precsim \min  \left\{\frac {\sqrt {r m}}{d \sqrt {d} \kappa^ {9}}, \frac {\sqrt {r}}{d \sqrt {d}} \left(\kappa^ {2} \sqrt {\frac {d}{r}}\right) ^ {- 9 6 \kappa^ {2}} \right\}. \tag {13}
$$
Then, after
$$
\hat {t} \precsim \frac {1}{\alpha \sigma_ {r ^ {\star}} ^ {2} (\bar {X})} \left(\ln \left(\kappa^ {2} \sqrt {\frac {d}{r}}\right) + \ln \left(\frac {\sigma_ {r ^ {\star}} (\bar {X})}{\mu}\right) + \ln \left(\max  \left\{1, \kappa \frac {r ^ {\star}}{r - r ^ {\star}} \right\} \frac {\| \bar {X} \|}{\mu}\right)\right) \tag {14}
$$
iterations, there holds
$$
\frac {\left\| U ^ {\hat {t}} \left(U ^ {\hat {t}}\right) ^ {\top} - Z ^ {\star} \right\| _ {F}}{\| Z ^ {\star} \|} \preceq \left(\left(r - r ^ {\star}\right) ^ {7 / 8} \left(r ^ {\star}\right) ^ {1 / 8} \| \bar {X} \| ^ {- 2 1 / 1 6} \mu^ {2 1 / 1 6} \left(\kappa^ {2} \frac {d}{r}\right) ^ {2 1 / 1 6}\right), \tag {15}
$$
with probability at least  $1 - c_{1}e^{-c_{2}r}$ , where  $c_{1}, c_{2} > 0$  are universal constants.
- Statistical guarantees: (15) demonstrates that, in the setting above, the iterates  $U^t(U^t)^\top$  converge to an estimate of the low-rank solution  $Z^\star$  within a precision that can be made arbitrarily small by reducing the size  $\mu$  of the random initialization. The test error's dependence on  $\mu$  is polynomial, whereas the worst-case convergence time only increases logarithmically with  $\mu$  (see (14)), indicating that significant test error reductions can be achieved with moderate increases in communication and local computations. These guarantees are established under the RIP of the measurement operator  $\bar{A}$ , and thus operate under the same sample complexity as the centralized setting. For instance, for Gaussian measurement matrices,  $N \asymp d(r^\star)^2\kappa^8$ . While the dependence on  $d$  is optimal, the scaling on  $(r^\star)^2$  and  $\kappa^8$  is less favorable compared to convex approaches based on nuclear norm minimization [3]. However, decentralized methods solving such formulations directly (e.g., [23]) would entail a communication cost of  $\mathcal{O}(d^2)$ , which is significantly less favorable than the  $\mathcal{O}(rd)$  of Algorithm 3.
- On the number of iterations: Interestingly, the worst-case iteration complexity aligns with what observed for the GD in the centralized setting, following thus the same interpretation [34]. The first term in (14) represents the duration of the spectral alignment phase: beginning from a small initialization, the iterates  $U^{t}(U^{t})^{\top}$  progressively align with the  $r^{\star}$  leading eigenvectors of the mapping  $\mathcal{W}^2 + \alpha / m\mathcal{A}^*\mathcal{A}(Z^\star)$ . Under the in-network RIP (Lemma 1), which requires the RIP of  $\bar{\mathcal{A}}$  and a sufficiently small  $\rho$ , we establish that this operator approximates the mapping of the power method applied to  $Z^{\star}$  (see the sketch of the proof in Sec. 3.1). The remaining two terms in (14) represent the duration of the subsequent refinement phase. This phase steers the iterates away from certain degenerate saddle points while ensuring convergence towards the low-rank matrix  $Z^{\star}$ .
In line with the findings for centralized GD [34], the test accuracy achieved at time  $\hat{t}$  might not persist for larger iterations. The corollary below refines these results by establishing a nonempty time interval within which the estimation error is guaranteed to stay within the desired accuracy.
Corollary 1. Under the conditions of Theorem [7] it holds
$$
\frac {\left\| U ^ {t} \left(U ^ {t}\right) ^ {\top} - Z ^ {\star} \right\| _ {F}}{\left\| Z ^ {\star} \right\|} \preceq (r - r ^ {\star}) ^ {7 / 8} \left(r ^ {\star}\right) ^ {1 / 8} \mu^ {1 / 8} \| \bar {X} \| ^ {- 2 1 / 1 6} \left(\kappa^ {2} \frac {d}{r}\right) ^ {1 / 8}, \tag {16}
$$
for any  $t\in [\hat{t},T]$  , with
$$
T - \hat {t} \succsim \frac {1 - \frac {d \kappa^ {2} \mu}{r}}{\alpha \left(\kappa^ {2} \sigma^ {2} \mu^ {1 / 8}\right)} \quad a n d \quad \sigma := c _ {3} \left(r ^ {\star}\right) ^ {1 / 8} \left(r - r ^ {\star}\right) ^ {7 / 8} \| \bar {X} \| ^ {1 1 / 1 6}, \tag {17}
$$
where  $c_{3} > 0$  is an universal constant.
The corollary ensures that the test error remains proportional to  $\mu^{1/8}$  throughout the interval  $T - \hat{t}$ . Notably, the duration of this interval increases as  $\mu$  approaches zero. This result is quite desirable, especially in distributed settings where coordinating termination at a specific time may be challenging.
- On the condition (12) on  $\rho$  and network scalability: The stipulation on  $\rho$  signifies the need of a well-connected network—the larger the network size  $m$  or the condition number  $\kappa$  of the ground truth, the smaller  $\rho$ . This is a non-negotiable condition essential for managing consensus errors through the tolerance  $\Delta$ , thus ensuring an adequate in-network RIP for the algorithm operator  $\mathcal{A}$ . When coupled with the RIP of the measurement operator  $\bar{\mathcal{A}}$ , it suffices for a sufficient alignment of the iterates  $U^t(U^t)^\top$  with the signal subspace from the early stages of the algorithm. Our numerical experiments (see Sec. 4) indeed demonstrate that maintaining such a constraint on  $\rho$  is indispensable for securing convergence and favorable estimation errors. When the network graph is predetermined (with given  $W$ ), one can meet the condition (12) (if not a-priori satisfied) by employing at each agent's side multiple rounds of communications per gradient evaluation. This is a common practice [27] that in our case results in a communication overhead that is only logarithmic in  $m$  and  $\kappa$ .
Notice that the generalization error (see (15) and (16)) is independent of  $\rho$  or  $m$ . This demonstrates that the algorithm's performance scales favorably with  $m$ . As  $m$  increases, the generalization error remains unchanged, whereas the communication cost grows only modestly (logarithmically with  $m$ ).
3.1 Sketch of the proof of Theorem 1
This section provides some insights on the proof of the theorem, highlighting the challenges and the differences with existing centralized and decentralized techniques.
The goal is to establish that  $U^t (U^t)^\top \approx Z^\star$  as the algorithm progresses. Following [34], we decompose the iterates as
$$
U ^ {t} = \underbrace {U ^ {t} Q ^ {t} \left(Q ^ {t}\right) ^ {\top}} _ {\triangleq \text {s i g n a l}} + \underbrace {U ^ {t} Q ^ {t , \perp} \left(Q ^ {t , \perp}\right) ^ {\top}} _ {\triangleq \text {n o i s e}}, \tag {18}
$$
where  $Q^t \in \mathbb{R}^{r \times r^*}$  contains the right singular vectors of  $V_{Z^{\star}}^{\top}U^{t}$ , i.e.,  $V_{Z^{\star}}^{\top}U^{t} = V^{t}\Lambda^{t}(Q^{t})^{\top}$ ; and  $Q^{t,\perp} \in \mathbb{R}^{(r \times r - r^{\star})}$  is the orthonormal complement of  $Q^{t}$ . By construction,  $\operatorname{span}(Q^{t}) \cup \operatorname{span}(Q^{t,\perp}) = \mathbb{R}^{r}$ , allowing for the decomposition (18). Further, notice that the noise term is orthogonal to the signal space, i.e.  $V_{Z^{\star}}^{\top}U^{t}Q^{t,\perp} = 0$ , which implies that once  $U^{t}$  is projected onto the signal space, the only relevant term left is  $V_{Z^{\star}}^{\top}U^{t} = V_{Z^{\star}}^{\top}U^{t}Q^{t}(Q^{t})^{\top}$ , hence the name "signal".
Based on (18), and under the assumptions of the theorem, we establish that: (i)  $U^{t}Q^{t}(Q^{t})^{\top}$  is full rank and the signal-term grows as the algorithm progresses. (ii) The noise-term grows slower than the signal and remains sufficiently small. (ii) The error can be bounded by a polynomial proportional to the initialization size. Similar to [34], the analysis is organized in two phases.
Phase I (power-like method): The goal of this phase is to establish that after sufficiently long time  $t_\star$  since the initialization  $(t = 0)$ ,  $\sigma_{\mathrm{min}}(U^{t_\star}Q^{t_\star}) > c\| U^{t_\star}Q^{t_\star,\perp}\|$  and that  $\| (V_{Z^\star}^\perp)^\top V_{U^{t_\star}Q^{t_\star}}\|$  is small, which means that the iterates are better aligned with the signal space than the noise space. Therefore, we are to identify in (9) a mechanism that allows  $V_{U^t Q^t}$  to become aligned with  $V_{Z^\star}$ .
Given the initialization  $U^0 = \mu U$ , at iteration  $t = 1$ , we have
$$
U ^ {1} = \left(\mathcal {W} ^ {2} + \frac {\alpha}{m} \mathcal {A} ^ {*} \mathcal {A} (Z ^ {\star})\right) U ^ {0} + \mu^ {2} \frac {\alpha}{m} \mathcal {A} ^ {*} \mathcal {A} (U U ^ {\top}) U ^ {0}. \tag {19}
$$
Consequently, if  $\mu$  is sufficiently small, for the first few iterations  $t$ , one can write
$$
U ^ {t} \approx \left(\mathcal {W} ^ {2} + \frac {\alpha}{m} \mathcal {A} ^ {*} \mathcal {A} \left(Z ^ {*}\right)\right) ^ {t} U ^ {0} + \mathcal {O} \left(\mu^ {2} \| U ^ {0} \| \| U \| ^ {2}\right). \tag {20}
$$
Under the assumption that  $\bar{A}$  fulfills the  $\delta_{r^{\star}}$  RIP, we can establish using the in-network RIP that if  $\rho \leq \mathcal{O}\left(\frac{\delta_{2(r^{\star} + 1)}}{m^{2}\sqrt{m}}\right)$ ,  $\mathcal{A}^*\mathcal{A}(Z^\star) = Z^\star + \varepsilon$ , with  $\| \varepsilon \| \leq \mathcal{O}(\Delta_{2r^{\star}}\| Z^{\star}\|$ . Further, by construction  $\mathcal{W}^2 Z^\star = Z^\star \mathcal{W}^2$ , implying that  $\mathcal{W}^2$  and  $Z^{\star}$  share the same eigenspace. Also,  $\lambda_i(\mathcal{W}^2) = 1$ , for all  $i = 1,\ldots ,d$ . Consequently  $\lambda_{i}\left(\mathcal{W}^{2} + \frac{\alpha}{m} Z^{\star}\right) = 1 + \frac{\alpha}{m}\lambda_{i}(Z^{\star})$  for  $i = 1,\dots ,d$ . Therefore, we can further approximate (20) as
$$
U ^ {t} \approx \left(\mathcal {W} ^ {2} + \frac {\alpha}{m} Z ^ {\star}\right) ^ {t} U ^ {0} + \mathcal {O} \left(\mu^ {2} \| U ^ {0} \| \| U \| ^ {2}\right), \tag {21}
$$
where we have disregarded  $\varepsilon$ , for simplicity of exposition. Leveraging perturbation theory arguments, in the proof we demonstrate that  $\varepsilon$  can be properly controlled. Using the above arguments, we have
$$
V _ {Z ^ {\star}} V _ {Z ^ {\star}} ^ {\top} U ^ {t} \approx V _ {Z ^ {\star}} \left(\mathcal {I} + \frac {\alpha}{m} \Lambda_ {Z ^ {\star}}\right) ^ {t} V _ {Z ^ {\star}} ^ {\top} U ^ {0} + V _ {Z ^ {\star}} V _ {Z ^ {\star}} ^ {\top} \left(\mathcal {W} ^ {2}\right) ^ {t} V _ {Z ^ {\star}} ^ {\perp} \left(V _ {Z ^ {\star}} ^ {\perp}\right) ^ {\top} U ^ {0} + \mathcal {O} \left(\mu^ {2} \| U ^ {0} \| \| U \| ^ {2}\right) \tag {22}
$$
$$
V _ {Z ^ {\star}} ^ {\perp} \left(V _ {Z ^ {\star}} ^ {\perp}\right) ^ {\top} U ^ {t} \approx V _ {Z ^ {\star}} ^ {\perp} \left(\mathcal {W} ^ {2}\right) ^ {t} \left(V _ {Z ^ {\star}} ^ {\perp}\right) ^ {\top} U ^ {0} + \mathcal {O} \left(\mu^ {2} \| U ^ {0} \| \| U \| ^ {2}\right), \tag {23}
$$
where the first term in the RHS of (22) corresponds to the power method on the matrix  $Z^{\star}$ . Further, we see that the mentioned term grows faster than any other. Consequently, at the time  $t_{\star}$  at which we exit phase I,  $U^{t_{\star}}$  is sufficiently aligned with the signal space as compared to the noise space.
Phase II (refinement): In this phase we establish that, given  $\sigma_{\mathrm{min}}(U^{t_\star}Q^{t_\star})\geq c_0\| U^{t_\star}Q^{t_\star ,\perp}\|$  and  $\| (V_{Z^{\star}}^{\perp})^{\top}V_{U^{t_\star}Q^{t_\star}}\| \leq c_1$  : (i) the alignment  $\sigma_{\mathrm{min}}(V_{Z^{\star}}^{\top}U^{t_\star}Q^{t_\star})$  grows and stabilizes away from zero, (ii) the error  $\| U^{t}Q^{t,\perp}\|$  grows slower than the alignment  $\| U^{t}Q^{t}\|$  and, (iii) the error  $\| V_{Z^{\perp}}^{\top}V_{U^{t}Q^{t}}\|$  remains sufficiently small. A careful study and balance of these quantities yield the final convergence.
Challenges with respect to the centralized case: The main challenge with respect to analyses of the GD (e.g., [34]) comes from the distributed nature of the algorithm generating extra error terms (e.g., consensus errors), which significantly complicate the analysis. Our analysis builds on a newly introduced notion of RIP for the algorithm operator  $\mathcal{A}$ . This is substantially different from the classical RIP or GD mapping [34], which lack of the network gossip matrix  $\mathcal{W}^2$ .
Challenges with respect to existing distributed optimization approaches: The standard approach in the distributed optimization literature typically takes the route of splitting the algorithm dynamics
in its average and consensus error. We deviated from such decomposition, because controlling such errors on  $U^{t}(U^{t})^{\top}$  would result in bounds of the type
$$
\left\| U ^ {t} \left(U ^ {t}\right) ^ {\top} - \mathcal {J} U ^ {t} \left(U ^ {t}\right) ^ {\top} \mathcal {J} \right\| \leq \mathcal {O} (\rho \| U ^ {t} \left(U ^ {t}\right) - Z ^ {\star} \|) \tag {24}
$$
which are insufficient to understand for example the dynamics of  $\| (V_{Z^{\star}}^{\perp})^{\top}V_{U^{t}Q^{t}}\|$ . Furthermore, the split into signal and noise subspaces allows us to invoke the in-network RIP with  $r^{\star}$ , which would not be the case if splitting the iterates along consensus and non-consensus spaces.
Specifically regarding to phase I of the scheme, another mode classical approach would be "centering" the dynamics around the centralized trajectory of the power method, i.e.,
$$
U ^ {1} = \left(\mathcal {J} + \frac {\alpha}{m} \mathcal {J A} ^ {*} \mathcal {A} (Z ^ {\star}) \mathcal {J}\right) U ^ {0} - \mu^ {2} \frac {\alpha}{m} \mathcal {A} ^ {*} \mathcal {A} (U U ^ {\top}) U ^ {0} \tag {25}
$$
$$
+ \left(\mathcal {W} ^ {2} - \mathcal {J}\right) U ^ {0} + \frac {\alpha}{m} \left(\mathcal {A} ^ {*} \mathcal {A} \left(Z ^ {\star}\right) - \mathcal {J} \mathcal {A} ^ {*} \mathcal {A} \left(Z ^ {\star}\right) \mathcal {J}\right) U ^ {0}, \tag {26}
$$
which, using the in-network RIP and the fact that  $\mu^2$  and  $t$  are sufficiently small, would yield
$$
U ^ {t} \approx \left(\mathcal {J} + \frac {\alpha}{m} \mathcal {J A} ^ {*} \mathcal {A} (Z ^ {\star}) \mathcal {J}\right) ^ {t} U ^ {0} + \mu^ {2} \mathcal {O} \left(\| U \| ^ {2} \| U ^ {0} \|\right) + \mathcal {O} \left(\rho^ {2} \| U ^ {0} \|\right). \tag {27}
$$
Here, the degree of freedom to control the error terms (second and third) are  $\rho$  and  $\mu$ . This however would enforce the undesirable condition  $\rho \leq \mathcal{O}(\mu)$ , which couples the network connectivity with the size of the initialization.
We discuss some preliminary experiments validating our theoretical findings. All simulations are performed on a Apple M2 Pro @ 3.5 GHz computer, using 32 GB RAM running macOS Ventura 13.3.1. We generate a random matrix  $\bar{X} \in \mathbb{R}^{d \times r^{\star}}$ , with  $d = 50$  and  $r^{\star} = 2$ , which we use through all experiments. The symmetric measurement matrices are generated as  $A_{i} = (1/2)(S_{i} + S_{i}^{\top})$ , where  $S_{i} \in \mathbb{R}^{d \times d}$  have i.i.d. standard Gaussian elements. The communication networks are generated as Erdős-Rényi graphs, with link activation probability  $p = 0.05$ . and different sizes  $m$  (specified in each experiment below). For any generated graph, we set  $\bar{W}$  according to Metropolis weights [28], and then let  $W = \bar{W}^{K}$ , with the integer  $K$  chosen to meet the condition (12) on  $\rho$ , resulting in  $K$  communication rounds per agent/iteration. Finally, we choose  $\alpha = 1/4$  and  $\mu = d^{-3}$ .
(i) Validating Theorem 1: This experiment shows that under the conditions of Theorem 1, the test error behaves predictably as that of the centralized GD (up to constant factors). Furthermore, the invariance of such an error with the network size  $m$  is also confirmed, as long as  $\rho \leq \mathcal{O}(1 / m^6)$  (as requested in (12)). In the experiments, the total sample size is  $N = 1000$ , split equally among agents  $m = \{5, 100, 500, 1000\}$ . Fig 1a plots the normalized test error; Fig 1b shows  $\| (V_M^\perp)^\top V_{U^t} \|$  which measures the missalignment of  $U^t$  with the power method matrix; and Fig 1c displays the  $\sigma_{r^*}(U^t) / \sigma_{r^*}(X)$  which combined with Fig 1b allows us to claim that the signal  $U^t Q^t$  is well aligned with  $V_{Z^*}$  and full ranked. The curves show that the behavior of the decentralized algorithm is close to that of the centralized GD (blue lines). As predicted, the error decays quickly after the correct subspace has been identified. Furthermore, convergence rate and generalization error are almost invariant to network-size scaling, as long as  $\rho \leq \mathcal{O}(m^{-6})$ .
(ii) Validating condition (12) on  $\rho$ : We showcase the necessity of decreasing  $\rho$  while increasing  $m$ . Given a connected base graph with associated  $\bar{W}$ , for the sequence of graphs generated with increasing  $m = \{10, 50, 100, 500\}$ , we let  $\bar{W} = \bar{W}^T$ , with  $T$  such that  $\rho \approx 0.85$ , for all  $m$ . This eventually violates (12). Fig. [resp. Fig. 2b] plots the normalized generalization error versus the iterations, for  $m = 1$ ,  $m = 10$  and  $m = 50$  (resp.  $m = 100$  and  $m = 500$ , where the two curves are only up to the iterations  $t = 70$  and  $t = 17$ , respectively). The figures demonstrate the necessity of  $\rho$  scaling down with  $m$  increasing. In fact, both the rate and achievable estimation errors degrade (and eventually break down) as the network size increases while keeping  $\rho$  fixed. We claim that this stems from the fact that if the network is not sufficiently well-connected, the in-network RIP does not hold with sufficiently small tolerance, yielding to a failure of the power method early stage and consequently producing an unrecoverable missalignment with the signal subspace.
(c)
Figure 1: Performance of Algorithm 3 for different network size  $m$ , with  $\rho = \mathcal{O}(m^{-6})$ .
(b)
Figure 2: Performance of Algorithm 3, for different network size  $m$ , and fixed  $\rho \approx 0.85$ .
Funding in direct support of this work: ONR Grant # N00014-21-1-267.
[1] S. Bhojanapalli, B. Neyshabur, and N. Srebro. Global optimality of local search for low rank matrix recovery. In Adv. Neural Inf. Process. Syst., pages 3873-3881, 2016.
[2] P. Bianchi and J. Jakubowicz. Convergence of a multi-agent projected stochastic gradient algorithm for non-convex optimization. IEEE Trans. on Automatic Control, 58(2):391-405, Feb. 2013.
[3] Emmanuel J Candès and Yaniv Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on information theory, 57(4):2342-2359., 2011.
[4] Yuejie Chi, Yue M Lu, and Yuxin Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20):5239-5269, 2019.
[5] A. Daneshmand, Ying Sun, G. Scutari, F. Facchinei, and B. Sadler. Decentralized dictionary learning over time-varying digraphs. J. on Machine Learning Research, (139):1-62, 2019.
[6] Amir Daneshmand, Gesualdo Scutari, and Vyacheslav Kungurtsev. Second-order guarantees of distributed gradient algorithms. SIAM Journal on Optimization, 30(4):3029-3068, 2020.
[7] M. A. Davenport and J. Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE Journal of Selected Topics in Signal Processing, 10(4):608-622, 2016.
[8] P. Di Lorenzo and G. Scutari. Next: In-network nonconvex optimization. IEEE Trans. on Signal and Information Processing over Networks, 2(2):120-136, June 2016.