<figure><img src="image_1.png" /></figure>

HAL
open science

NONCONVEX MULTITASK LEARNING OVER
NETWORKS

Stefan Vlaski, Roula Nassif

<figure><img src="image_2.png" /></figure>

To cite this version:

Stefan Vlaski, Roula Nassif. NONCONVEX MULTITASK LEARNING OVER NETWORKS. 2024
32nd European Signal Processing Conference (EUSIPCO), Aug 2024, Lyon, France. hal-04848633

HAL Id: hal-04848633

https://hal.science/hal-04848633v1

Submitted on 19 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
NONCONVEX MULTITASK LEARNING OVER NETWORKS

Stefan Vlaski* and Roula Nassif†

* Department of Electrical and Electronic Engineering, Imperial College London, UK

† Université Côte d'Azur, I3S Laboratory, CNRS, France

ABSTRACT

Distributed learning paradigms, such as federated or decentralized learning, have emerged as effective tools in endowing multi-agent systems with collective intelligence, while preserving privacy, robustness to node or link failure, and communication efficiency. While the performance gains of distributed architectures are well established in homogenous environments where agents observe data following similar distributions and aim to solve similar tasks, the same cannot be said for heterogeneous, multitask, or non-iid settings. These heterogeneous environments emerge naturally across a variety of applications and are known to introduce significant performance degradation to classical strategies for distributed learning. Multitask learning, on the other hand, allows a collection of agents to collaborate on solving different, albeit related tasks, without enforcing full consensus and homogeneity. Performance guarantees for decentralized multitask learning algorithms so far have focused on convex objectives, such as those encountered when solving linear or logistic regression objectives, but do not allow for nonconvexity, such as in the case of deep neural networks. We examine the nonconvex learning dynamics of a recently proposed bias-corrected algorithm for nonconvex subspace constrained optimization and demonstrate the effectiveness of the framework in deep multitask learning.

**Index Terms—** Decentralized learning, federated learning, multitask learning, nonconvex optimization, bias-correction.

1. INTRODUCTION AND RELATED WORKS

We consider a collection of $K$ agents, collected in the set $\mathcal{N}$, where each agent indexed by $k$ is equipped with a local risk function:

$$J_k(w_k) \triangleq \mathbb{E}_{x_k} Q(w_k; x_k) \quad (1)$$

Here, $w_k \in \mathbb{R}^{M_k^w}$ contains parameters chosen by agent $k$, and $x_k \in \mathbb{R}^{M_k^x}$ denotes locally available data. The loss $Q(w_k; x_k)$ quantifies the fit of the parameterization $w_k$ to

the random data $x_k$, and the resulting risk $J_k(w_k)$ is the loss after taking expectations. We additionally associate a graph with the collection of agents, which will determine the flow of information between pairs of agent. In particular, we associate with each agent $k$ its neighborhood $\mathcal{N}_k \subseteq \mathcal{N}$, where $\mathcal{N}_k$ denotes the subset of agents with which agent $k$ is able to communicate directly.

If we collect the local parameter vectors $w_k$ into the network-level quantity $w \triangleq \text{col}\{w_k\} \in \mathbb{R}^M$ with $M = \sum_{k=1}^K M_k^w$, we can formulate a coupled global optimization problem [1–3]:

$$w^o \triangleq \arg \min_{w \in \mathbb{R}^M} \sum_{k=1}^K J_k(w_k) \quad (2a)$$

$$\text{subject to } w \in \mathcal{R}(\mathcal{U}) \quad (2b)$$

Here, $\mathcal{R}(\mathcal{U})$ denotes the range space of the matrix $\mathcal{U} \in \mathbb{R}^{M \times P}$, which ensures that $w = \mathcal{U}v$ for some $v \in \mathbb{R}^P$, and couples the local objectives $J_k(w_k)$. Problem (2a)–(2b) is referred to as a *multitask* learning problem [4, 5], since agents seek distinct, albeit related, parameter vectors $w_k$ subject to a coupling constraint defined by $\mathcal{U}$. When $M_k^w = M^w$, we can set $\mathcal{U} = \mathbb{1}_K \otimes I_{M^w}$ and recover:

$$w^o \triangleq \arg \min_{w \in \mathbb{R}^M} \sum_{k=1}^K J_k(w_k) \quad (3)$$

$$\text{subject to } w_1 = w_2 = \dots = w_K \quad (4)$$

which is equivalent to the consensus optimization problem:

$$w^o \triangleq \arg \min_{w \in \mathbb{R}^{M_w}} \frac{1}{K} \sum_{k=1}^{K} J_k(w) \quad (5)$$

The formulation (5) forces agents to pursue a common model $w^o$, which fits the average of local objectives best. Problems of the form (5) have been the focus of a plethora of works on distributed optimization and learning, ranging from consensus [6–8] and diffusion algorithms [9, 10] to more recent works involving bias-correction using gradient tracking [11–13], primal-dual arguments [14–16] — see [17] for a recent survey. Finding a single common model to fit the average risk across the network is effective whenever local data distributions of $x_k$ are sufficiently similar, resulting in similar local

<footnote>Emails: s.vlaski@imperial.ac.uk, roula.nassif@unice.fr. The work of R. Nassif was supported by ANR JCJC grant ANR-22-CE23-0015-01 (CEDRO project).</footnote>
objectives $J_k(\cdot)$, or when homogeneity of local models is desired for application specific reasons. Nevertheless, and particularly for highly heterogeneous data or tasks, insistence on single model which performs best on average will necessarily deteriorate performance on local data sets.

In contrast, the more general coupling through the range constraint (2b) allows for some task relationship, which can be exploited through communication, while not enforcing full consensus among local models. These relationships arise naturally in a number of signal processing and learning applications, ranging from signal recovery and denoising [1], beamforming [2], low-pass filtering in the graph domain [3], to the training of multi-head [4] and graph convolutional neural networks [18]. Since the subspace constrained problem (2a)–(2b) is a direct generalization of the single-task problem (5), most algorithms for subspace constrained multitask learning available in the literature can be viewed as a generalization of their single-task counterpart. For instance, the diffusion based algorithm from [2,3] for subspace constrained problems of the form (2a)–(2b) takes the form:

$$w_{k,i} = \sum_{\ell \in \mathcal{N}_k} A_{\ell k} \left( w_{\ell,i-1} - \mu \widehat{\nabla J}_{\ell}(w_{\ell,i-1}) \right) \quad (6)$$

where $\widehat{\nabla J}_{\ell}(w_{\ell,i-1})$ denotes a stochastic approximation of the gradient $\nabla J(w_{\ell,i-1})$. A common construction is to employ the instantaneous approximation $\widehat{\nabla J}_{\ell}(w_{\ell,i-1}) \triangleq \nabla Q(w_{\ell,i-1}; x_{\ell,i})$ based on the sample $x_{\ell,i}$ available to agent $\ell$ at time $i$, though alternative constructions based for example on mini-batches are possible as well. Furthermore, $\{A_{\ell k}\} \in \mathbb{R}^{M_k^w \times M_k^w}$ denote matrix-valued combination weights, which define the linear combination transformation and satisfy:

$$A_{\ell k} \triangleq [\mathcal{A}]_{\ell k} = 0, \quad \text{if } \ell \notin \mathcal{N}_k \quad (7a)$$

$$\lim_{i \to \infty} \mathcal{A}^i = \mathcal{P}_U \quad (7b)$$

Here, condition (7a) ensures that (6) can be implemented in a distributed manner by relying only on communication among neighboring nodes, while $\mathcal{P}_U \triangleq \mathcal{U}(\mathcal{U}^\mathrm{T}\mathcal{U})^{-1}\mathcal{U}^\mathrm{T}$ denotes the projection matrix onto $\mathcal{R}(\mathcal{U})$ and hence (7b) ensures that the combination steps approximate asymptotically the projection onto $\mathcal{R}(\mathcal{U})$. In contrast, the diffusion algorithm for the single-task problem (5) takes the form:

$$w_{k,i} = \sum_{\ell \in \mathcal{N}_k} a_{\ell k} (w_{\ell,i-1} - \mu \nabla J_{\ell}(w_{\ell,i-1})) \quad (8)$$

where $a_{\ell k} \in \mathbb{R}$ now denote scalar combination weights. Analogously, the DiSPO algorithm of [1] corresponds to a generalization of the distributed gradient algorithms of [6–8]. More recently, the work [19] derived subspace constrained generalizations of a large number of bias-corrected algorithms [11–16], while [20] derived steady-state performance guarantees for the exact subspace diffusion algorithm. Of the

existing convergence guarantees for distributed subspace constrained learning, the works [1–3, 19, 20] focus on convex risk function. While [1, 18] provide first-order stationarity guarantees in nonconvex environments, these apply only to implementations which employ exact gradients $\nabla J_k(w_{k,i-1})$, rather than stochastic gradient approximations. In this work, we study the dynamics of a stochastic algorithm for distributed subspace constrained learning, and quantify the impact of the resulting gradient noise on learning performance. Furthermore, we consider a local variant of (6) where agents are allowed to take multiple local updates inbetween communication steps, and quantify said impact on performance. In particular, we will consider the following local subspace constrained learning algorithm:

**Algorithm 1: Local Subspace Diffusion**

Initialize $w_{k,0}$ arbitrarily. For $i \ge 1$, set
$$\psi_{k,i,0} = w_{k,i-1} \text{ and perform } E \text{ local updates for} \\ e = 1, \dots, E:$$

$$\psi_{k,i,e} = \psi_{k,i,e-1} - \frac{\mu}{E} \widehat{\nabla J}_k(\psi_{k,i,e-1}) \quad (9)$$

Exchange and transform:

$$w_{k,i} = \sum_{\ell \in \mathcal{N}_k} A_{\ell k} \psi_{\ell,i,E} \quad (10)$$

Multiple local updates as in (9) are commonly employed in federated learning structures [21] and the associated performance trade-offs have received considerable attention in the single-task setting [22, 23]. Their impact in distributed multitask learning remains underexplored, despite numerical evidence of their potential benefit [20].

## 2. CONVERGENCE ANALYSIS

### 2.1. Modeling Conditions

We adopt the following typical conditions on the risk functions and gradient approximation [5, 10, 20, 24].

**Assumption 1 (Conditions on risk functions).** Each local objective $J_k(\cdot)$ has $\delta_k$-Lipschitz gradients, meaning that for every $x, y \in \mathbb{R}^{M_k^w}$, it holds that:

$$\|\nabla J_k(x) - \nabla J_k(y)\| \le \delta_k \|x - y\| \quad (11)$$

for some $\delta_k > 0$. Additionally, the aggregate problem (2a)–(2b) is bounded from below, meaning:

$$\mathcal{J}(w) \triangleq \sum_{k=1}^{K} J_k(w_k) \ge \mathcal{J}^o \quad (12)$$

for all $w \in \mathcal{R}(\mathcal{U})$. $\square$
**Assumption 2 (Conditions on the gradient approximation).** Define the local gradient noise process:

$$ \mathbf{s}_{k,i}(\mathbf{w}_{k,i-1}) \triangleq \widehat{\nabla J}_k(\mathbf{w}_{k,i-1}) - \nabla J_k(\mathbf{w}_{k,i-1}) \quad (13) $$

The gradient noise is zero-mean after conditioning on the current iterate:

$$ \mathbb{E}\{\mathbf{s}_{k,i}(\mathbf{w}_{k,i-1}) | \mathbf{w}_{k,i-1}\} = 0 \quad (14) $$

Furthermore, its variance is bounded according to:

$$ \mathbb{E}\{\|\mathbf{s}_{k,i}(\mathbf{w}_{k,i-1})\|^2 | \mathbf{w}_{k,i-1}\} \leq \sigma_k^2 \quad (15) $$

If we additionally define $\mathbf{s}_i(\mathbf{w}_{i-1}) \triangleq \text{col}\{\mathbf{s}_{k,i}(\mathbf{w}_{k,i-1})\}$, we can bound the projected gradient noise as:

$$ \mathbb{E}\{\|\mathcal{P}_U \mathbf{s}_i(\mathbf{w}_{i-1})\|^2 | \mathbf{w}_{i-1}\} \leq \sigma_U^2 \quad (16) $$

We will find $\sigma_U^2$ to be one key quantity in determining the performance of Algorithm 1. □

**Assumption 3 (Gradient bounds).** The local gradients are bounded and satisfy for all $\mathbf{w}_k \in \mathbb{R}^{M_k^w}$:

$$ \|\nabla J_k(\mathbf{w}_k)\|^2 \leq G_k^2 \quad (17) $$

Additionally, we can bound:

$$ \|(I - \mathcal{P}_U)\nabla J(\mathbf{w})\|^2 \leq G_U^2 \quad (18) $$

We will find $G_U^2$ to be a key quantity in the performance of Algorithm 1. □

## 2.2. Network Recursions

If we collect $\mathbf{w}_i \triangleq \text{col}\{\mathbf{w}_{k,i}\}, \psi_{i,e} \triangleq \text{col}\{\psi_{k,i,e}\}$ and define $\widehat{\nabla J}(\psi_{i,e-1}) \triangleq \text{col}\{\widehat{\nabla J}_k(\psi_{k,i,e-1})\}$, we can write Algorithm 1 compactly as:

$$ \psi_{i,e} = \psi_{i,e-1} - \frac{\mu}{E} \widehat{\nabla J}(\psi_{i,e-1}), \quad e = 1, \dots, E \quad (19a) $$

$$ \mathbf{w}_i = \mathcal{A}\psi_{i,E} \quad (19b) $$

We follow an argument analogous to [25] in the context of federated learning, iterate (19a) and note that $\psi_{i,0} = \mathbf{w}_{i-1}$, yielding:

$$ \psi_{i,E} = \mathbf{w}_{i-1} - \frac{\mu}{E} \sum_{e=1}^{E} \widehat{\nabla J}(\psi_{i,e-1}) \quad (20) $$

Inspection of this relation reveals that (20) and hence (19a) are reminiscent of a mini-batch update using $E$ distinct gradient approximations $\widehat{\nabla J}(\psi_{i,e-1})$, with the detail that the gradient approximations are evaluated at intermediate estimates $\psi_{i,e-1}$ rather than $\mathbf{w}_{i-1}$. Nevertheless, we can write:

$$ \psi_{i,E} = \mathbf{w}_{i-1} - \frac{\mu}{E} \sum_{e=1}^{E} \nabla J(\mathbf{w}_{i-1}) - \frac{\mu}{E} \sum_{e=1}^{E} \mathbf{s}_{i,e} - \frac{\mu}{E} \sum_{e=1}^{E} \mathbf{q}_{i,e} \quad (21) $$

where we defined the perturbation terms:

$$ \mathbf{s}_{i,e} \triangleq \widehat{\nabla J}(\psi_{i,e-1}) - \nabla J(\psi_{i,e-1}) \quad (22) $$

$$ \mathbf{q}_{i,e} \triangleq \nabla J(\psi_{i,e-1}) - \nabla J(\mathbf{w}_{i-1}) \quad (23) $$

Returning to (19a)–(19b), we can write:

$$ \begin{aligned} \mathbf{w}_i = \mathcal{A} \mathbf{w}_{i-1} & - \frac{\mu}{E} \sum_{e=1}^{E} \mathcal{A} \nabla J(\mathbf{w}_{i-1}) \\ & - \frac{\mu}{E} \sum_{e=1}^{E} \mathcal{A} \mathbf{s}_{i,e} - \frac{\mu}{E} \sum_{e=1}^{E} \mathcal{A} \mathbf{q}_{i,e} \end{aligned} \quad (24) $$

Next, note that we can decompose $\mathbf{w}_i$ into:

$$ \mathbf{w}_i = \mathcal{P}_U \mathbf{w}_i + (I - \mathcal{P}_U) \mathbf{w}_i = \mathbf{w}_i^\mathcal{U} + \mathbf{w}_i^{\perp\mathcal{U}} \quad (25) $$

where $\mathbf{w}_i^\mathcal{U} \triangleq \mathcal{P}_U \mathbf{w}_i$ denotes the component of $\mathbf{w}_i$ which is aligned with the range space of $\mathcal{U}$, while $\mathbf{w}_i^{\perp\mathcal{U}} \triangleq (I - \mathcal{P}_U) \mathbf{w}_i$ denotes the component perpendicular to $\mathcal{R}(\mathcal{U})$. Decompositions of this type have proven indespensible when studying both distributed single and multitask learning leaning algorithms [1,2,10,20,24]. Analogously to [20], and using the fact that (7b) implies $\mathcal{P}_U \mathcal{A} = \mathcal{P}_U$ [1,2], we can transform (24) as:

$$ \begin{aligned} \mathbf{w}_i^\mathcal{U} = \mathbf{w}_{i-1}^\mathcal{U} & - \frac{\mu}{E} \sum_{e=1}^{E} \mathcal{P}_U \nabla J(\mathbf{w}_{i-1}^\mathcal{U}) - \frac{\mu}{E} \sum_{e=1}^{E} \mathcal{P}_U \mathbf{d}_i \\ & - \frac{\mu}{E} \sum_{e=1}^{E} \mathcal{P}_U \mathbf{s}_{i,e} - \frac{\mu}{E} \sum_{e=1}^{E} \mathcal{P}_U \mathbf{q}_{i,e} \end{aligned} \quad (26) $$

where we additionally introduced $\mathbf{d}_i \triangleq \nabla J(\mathbf{w}_{i-1}) - \nabla J(\mathbf{w}_{i-1}^\mathcal{U})$. For the perpendicular component we find:

$$ \begin{aligned} \mathbf{w}_i^{\perp\mathcal{U}} = (\mathcal{P}_U - \mathcal{A}) \mathbf{w}_{i-1}^{\perp\mathcal{U}} & - \frac{\mu}{E} \sum_{e=1}^{E} (\mathcal{P}_U - \mathcal{A}) \nabla J(\mathbf{w}_{i-1}) \\ & - \frac{\mu}{E} \sum_{e=1}^{E} (\mathcal{P}_U - \mathcal{A}) \mathbf{s}_{i,e} - \frac{\mu}{E} \sum_{e=1}^{E} (\mathcal{P}_U - \mathcal{A}) \mathbf{q}_{i,e} \end{aligned} \quad (27) $$

We conclude that (26) describes a projected gradient recursion for the subspace constrained problem (2a)–(2b) with perturbation terms $\mathbf{d}_i, \mathbf{s}_{i,e}$ and $\mathbf{q}_{i,e}$ arising from the decentralized approximation of the projection, stochastic gradient approximations and local updates respectively. Under assumptions 1 through 3 we can bound each of these terms as shown in the following lemma.

**Lemma 1 (Perturbation bounds).** Under Assumptions 1–3, and after sufficient iterations $i \geq i_o$, the perpendicular component $\mathbf{w}_i^{\perp\mathcal{U}}$ scales quadratically with the step-size for small step-sizes, i.e.:

$$ \mathbb{E}\{\|\mathbf{w}_i^{\perp\mathcal{U}}\|^2 | \mathbf{w}_{i-1}\} \leq \frac{\mu^2 \lambda_{\mathcal{A}}^2 (G_{\mathcal{U}}^2 + \max_k \sigma_k^2)}{(1 - \lambda_{\mathcal{A}})^2} \quad (28) $$
Here, $\lambda_A \triangleq \rho(\mathcal{P}_U - \mathcal{A}) < 1$ quantifies how well the decentralized projection approximates $\mathcal{A}$. We conclude that the perpendicular component will be small whenever the step-size is small ($\mu^2 \to 0$), the decentralized projection is accurate, which can be achieved for well-connected graphs [1, 2] ($\lambda_A \to 0$), or when the local gradients are consistent in the space spanned by $\mathcal{U}$ ($G_U^2 \to 0$) and gradient approximations are accurate ($\sigma_k^2 \to 0$). Further, the perturbation terms in (26)–(27) satisfy the following bounds:

$$ E \left\{ \left\| \frac{1}{E} \sum_{i=1}^{E} \mathcal{P}_U s_{i,e} \right\|^2 \middle| \mathbf{w}_{i-1} \right\} \le \frac{\sigma_U^2}{E} \quad (29) $$

$$ E \left\{ \|\mathbf{d}_i\|^2 \middle| \mathbf{w}_{i-1} \right\} \le \frac{\mu^2 \lambda_A^2 (G_U^2 + \max_k \sigma_k^2)}{(1 - \lambda_A)^2} \quad (30) $$

$$ E \left\{ \| \mathbf{q}_{i,e} \|^2 \middle| \mathbf{w}_{i-1} \right\} \le O(\mu^2 E \max_k G_k^2) \quad (31) $$

*Proof.* The proof follows by combining Jensen's inequality with assumptions 1 through 3 and mirrors the arguments in [25, 26]. □

## 2.3. Stationarity Guarantee

The bounds in lemma 1 demonstrate that the perturbation terms in (26) can be controlled by appropriately tuning the step-size parameter, ensuring that (26) approximates a projected gradient algorithm well. We can leverage this fact to establish the following stationarity guarantee.

**Theorem 1 (Stationarity guarantee).** Under Assumptions 1–3, the iterates generated by Algorithm 1 satisfy:

$$ \frac{1}{i^o} \sum_{i=1}^{i^o} E \| \mathcal{P}_U \nabla \mathcal{J}(\mathbf{w}_i) \|^2 \\ \le \frac{\mathcal{J}(\mathbf{w}_0) - \mathcal{J}^o}{\mu i^o} + O\left(\frac{\mu \sigma_U^2}{E}\right) + O\left(\frac{\mu^2 \lambda_A^2 (G_U^2 + \max_k \sigma_k^2)}{(1 - \lambda_A)^2}\right) \\ + O(\mu^2 E \max_k G_k^2) + O(\mu^3) \quad (32) $$

*Proof.* The proof follows by establishing a descent relation on $\mathcal{J}(\cdot)$ based on the Lipschitz gradient conditions in assumption 1, and subsequently bounding the negative effect of perturbations from lemma 1. Details are omitted due to space limitations. □

## 3. NUMERICAL RESULTS

We illustrate the applicability of the subspace constrained framework for the decentralized training of multitask neural networks. To this end, we construct a scenario from the MNIST data set. Recall that the data set contains images of

<figure><img src="image_4.png" /><figcaption>**Fig. 1:** Comparison in the single- and multitask implementations, both with $E = 5$ local updates.</figcaption></figure>

handwritten digits from 0 to 9. Our objective is to learn a neural network classifier which determines whether an observed image is even or odd. However, we provide agents with highly imbalanced data sets, with individual agents being provided only with examples from one of the pairs 0 and 1, 2 and 3, 4 and 5, and so on. Since individual agents are only provided with one even and one odd digit, rather than learn to distinguish all even from all odd digits, they could in principle simply learn to distinguish the two digits which are available to them. At the same time, there may be some transferable knowledge available in individual local data sets, despite the fact that agents observe different digits locally. We implement a convolutional neural network with nested structure $f(g(\mathbf{h}; W_1); W_2)$, where $\mathbf{h}$ denotes the incoming vectorized image, and $W_1$ parameterizes three early convolutional layers yielding the mapping $g(\cdot; W_1)$. Next, $W_2$ parameterizes the final two linear layers yielding the mapping $f(\cdot; W_2)$. We can then envision three potential architectures for collaboration: (1) In a non-cooperative mode, agents may train independent classifiers based only on their locally available data, with no exchange of information and independently trained weights $W_1$ and $W_2$. (2) In a single-task mode of operation, agents may learn a single global model as in (5), which corresponds to setting $\mathcal{U} = 1_K \otimes I_{M^w}$ and enforcing consensus in the weights $W_1$ and $W_2$. (3) Finally, agents may operate in a multitask mode, and learn subspace constrained models enforcing consensus in $W_1$ but not in $W_2$. This allows agents to learn high-level features from one another, while still allowing them to specialize to their local tasks, and is a popular technique in multitask learning [4]. This construction fits into the subspace constrained framework (2a)–(2b) and as such the algorithms and analysis developed here apply. We compare the learning performance of the three constructions in Fig. 1 and observe a marked benefit in multitask learning. The data set is significantly subsampled to create data scarcity and each agent is provided with 200 samples.
4. REFERENCES

[1] P. Di Lorenzo, S. Barbarossa, and S. Sardellitti, "Dis- tributed signal processing and optimization based on in- network subspace projections," *IEEE Transactions on Signal Processing*, vol. 68, pp. 2061–2076, 2020.

[2] R. Nassif, S. Vlaski, and A. H. Sayed, "Adaptation and learning over networks under subspace constraints—Part I: Stability analysis," *IEEE Transactions on Signal Processing*, vol. 68, pp. 1346–1360, 2020.

[3] R. Nassif, S. Vlaski, and A. H. Sayed, "Adaptation and learning over networks under subspace constraints—Part II: Performance analysis," *IEEE Transactions on Signal Processing*, vol. 68, pp. 2948–2962, 2020.

[4] R. Caruana, "Multitask learning," *Machine Learning*, vol. 28, no. 1, pp. 41–75, 1997.

[5] R. Nassif, S. Vlaski, C. Richard, J. Chen, and A. H. Sayed, "Multitask learning over graphs: An approach for distributed, streaming machine learning," *IEEE Signal Processing Magazine*, vol. 37, no. 3, pp. 14–25, 2020.

[6] J. Tsitsiklis, D. Bertsekas, and M. Athans, "Distributed asynchronous deterministic and stochastic gradient optimization algorithms," *IEEE Trans. Automatic Control*, vol. 31, no. 9, pp. 803–812, 1986.

[7] A. Nedić and A. Ozdaglar, "Distributed subgradient methods for multi-agent optimization," *IEEE Trans. Automatic Control*, vol. 54, no. 1, pp. 48–61, Jan 2009.

[8] S. Kar and J. M. F. Moura, "Convergence rate analysis of distributed gossip (linear parameter) estimation: Fundamental limits and tradeoffs," *IEEE Journal of Selected Topics in Signal Processing*, vol. 5, no. 4, pp. 674–690, 2011.

[9] C. G. Lopes and A. H. Sayed, "Diffusion least-mean squares over adaptive networks: Formulation and performance analysis," *IEEE Trans. Signal Processing*, vol. 56, no. 7, pp. 3122–3136, 2008.

[10] J. Chen and A. H. Sayed, "On the learning behavior of adaptive networks - Part I: Transient analysis," *IEEE Trans. Information Theory*, vol. 61, no. 6, pp. 3487–3517, June 2015.

[11] P. Di Lorenzo and G. Scutari, "NEXT: In-network non-convex optimization," *IEEE Trans. Signal and Information Processing over Networks*, vol. 2, no. 2, pp. 120–136, 2016.

[12] A. Nedić, A. Olshevsky, and W. Shi, "Achieving geometric convergence for distributed optimization over time-varying graphs," *SIAM Journal on Optimization*, vol. 27, no. 4, pp. 2597–2633, 2017.

[13] R. Xin, A. K. Sahu, U. A. Khan, and S. Kar, "Distributed stochastic optimization with gradient tracking

over strongly-connected networks," in *Proc. IEEE CDC*, 2019, pp. 8353–8358.

[14] W. Shi, Q. Ling, G. Wu, and W. Yin, "A proximal gradient algorithm for decentralized composite optimization," *IEEE Transactions on Signal Processing*, vol. 63, no. 22, pp. 6013–6023, 2015.

[15] K. Yuan, B. Ying, X. Zhao, and A. H. Sayed, "Exact diffusion for distributed optimization and learning – Part II: Convergence analysis," *IEEE Trans. Signal Processing*, vol. 67, no. 3, pp. 724–739, Feb 2019.

[16] H. Tang, X. Lian, M. Yan, C. Zhang, and J. Liu, "$D^2$: Decentralized training over decentralized data," in *Proceedings of International Conference on Machine Learning*, Jul 2018, vol. 80, pp. 4848–4856.

[17] S. Vlaski, S. Kar, A. H. Sayed, and J. M. F. Moura, "Networked signal and information processing: Learning by multiagent systems," *IEEE Signal Processing Magazine*, vol. 40, no. 5, pp. 92–105, 2023.

[18] S. Scardapane, I. Spinelli, and P. Di Lorenzo, "Distributed training of graph convolutional networks," *IEEE Transactions on Signal and Information Processing over Networks*, vol. 7, pp. 87–100, 2021.

[19] D. J. Marquis, D. Abou Jaoude, M. Farhood, and C. A. Woolsey, "Robustness and convergence analysis of first-order distributed optimization algorithms over subspace constraints," in *2023 American Control Conference (ACC)*, 2023, pp. 4753–4758.

[20] S. Wadehra, R. Nassif, and S. Vlaski, "Exact subspace diffusion for decentralized multitask learning," to appear in *Proc. IEEE CDC*, available as arXiv:2304.07358, April 2023.

[21] P. Kairouz, H. B. McMahan, et al., "Advances and open problems in federated learning," *Foundations and Trends in Machine Learning*, vol. 14, pp. 1–210, 2021.

[22] A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. Stich, "A unified theory of decentralized SGD with changing topology and local updates," in *Proceedings of the 37th International Conference on Machine Learning*, Jul 2020, vol. 119, pp. 5381–5393.

[23] S. A. Alghunaim, "Local exact-diffusion for decentralized optimization and learning," available as arXiv:2302.00620, Feb 2023.

[24] A. H. Sayed, "Adaptation, learning, and optimization over networks," *Foundations and Trends in Machine Learning*, vol. 7, no. 4-5, pp. 311–801, July 2014.

[25] E. Rizk, S. Vlaski, and A. H. Sayed, "Federated learning under importance sampling," *IEEE Transactions on Signal Processing*, vol. 70, pp. 5381–5396, 2022.

[26] S. Vlaski and A. H. Sayed, "Distributed learning in non-convex environments — Part II: Polynomial escape from saddle-points," *IEEE Trans. Signal Processing*, vol. 69, pp. 1257–1270, 2021.