<figure><img src="image_1.png" /></figure>

HAL
open science

# Sparsity-based regularization for learning over multitask graphs

Yara Zgheib, Marc Antonini, Roula Nassif

► To cite this version:

Yara Zgheib, Marc Antonini, Roula Nassif. Sparsity-based regularization for learning over multitask graphs. 23rd IEEE Statistical Signal Processing Workshop (SSP 2025), Jun 2025, Edimbourg, United Kingdom. 10.1109/SSP64130.2025.11073358. hal-05227630v2

HAL Id: hal-05227630

https://hal.science/hal-05227630v2

Submitted on 19 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Sparsity-based regularization for learning over multitask graphs

Yara Zgheib

Marc Antonini

Roula Nassif

13S Laboratory, CNRS

13S Laboratory, CNRS

13S Laboratory, CNRS

Université Côte d'Azur, France

Université Côte d'Azur, France

Université Côte d'Azur, France

**Abstract**—In this work, we consider learning over multitask graphs, where each agent seeks to estimate its own parameter vector (task) while benefiting from collaboration with its neighboring agents with related tasks. A key method for promoting relationships between tasks is regularization. Unlike previous studies that enforce graph smoothness via graph-based smooth regularization, we investigate non-smooth regularization to promote sparsity and piecewise-constant transitions between tasks at neighboring agents. We formulate a globally regularized optimization problem and, motivated by the forward-backward splitting approach, we propose a decentralized approach for solving the non-smooth optimization problem. Then, under some general convexity assumptions on the individual costs and regularization, we study the convergence w.r.t. the optimal solution of the globally regularized cost, in the mean-square-error sense. Finally, we illustrate the theoretical findings and the effectiveness of the proposed approach through simulations.

**Index Terms**—Distributed optimization, multitask graph, personalized learning, non-smooth regularization, proximal operator, gradient noise, mean-square-error stability.

# I. INTRODUCTION

The rapid proliferation of smart devices, including mobile phones, wearable technology, and autonomous vehicles, has led to a significant increase in the amount of private data generated continuously from distributed sources. While this data presents valuable opportunities for applications in health-care and smart cities, its sensitivity and sheer volume pose significant privacy challenges for machine learning methods. Federated and decentralized learning address these issues by enabling collaborative model training while keeping data local. In federated learning, multiple devices (or agents) collaboratively learn a shared model under the orchestration of a central server, which coordinates and aggregates the model updates from each client [1]–[3]. In contrast, decentralized learning enables collaborative learning without relying on a central server [4]–[8], and typically assumes a network topology where each agent communicates directly with its neighbors.

In this work, we consider decentralized learning, and assume that the data generated by different devices is non-iid (i.e., non independent and identically distributed), making it sub-optimal to learn a single global model for all devices. To address the statistical heterogeneity, we consider *personalized learning*, where each device aims to learn a personalized

model based on its own data, while still benefiting from the data of others. Several personalization techniques have been investigated in the literature [6], [9]–[11], including local fine tuning, contextualization, and multitask learning. Multitask learning, for instance, allows for the simultaneous learning of multiple related tasks by exploiting their commonalities and differences [6], [9], [12].

In this work, we focus on networked applications where the tasks at neighboring devices are expected to share a large number of (feature) weights, while only a relatively small subset may vary locally. We propose a decentralized multitask learning strategy for jointly learning the multiple tasks from streaming data. Since the weight variability pattern across agents is unknown beforehand, formulating a constrained optimization problem as in [13]–[18] to enforce consensus on the shared components of the parameter vectors at neighboring agents is not possible. Instead, our approach employs regularization to capture and promote relationships between tasks, a technique previously explored in decentralized collaborative multitask learning. While most studies employ smooth graph-based regularizers to enforce task smoothness [19]–[22], only a few investigate non-smooth regularization. This tool enables, for example, task clustering by employing $l_2$-norm co-regularizers to perform automatic clustering over the graph [23], or piecewise constant transitions by employing $l_1$-norm co-regularizers to enforce sparsity in neighboring task differences [24]. The latter is particularly useful in our networked application, as it does not rely on prior knowledge of which weights vary across neighboring agents.

In the remainder of the paper, we begin by formulating a global multitask optimization problem consisting of the sum of individual cost functions, regularized by non-smooth terms that promote sparsity in parameter differences among neighbors. To handle non-smooth regularization, we employ proximal-based techniques, which have demonstrated greater stability and faster convergence than sub-gradient methods in deterministic settings [25]–[27]. We present our decentralized proximal-based approach and, under general conditions, show that for sufficiently small step-size $\mu$, the proposed approach converges in the mean-square-error sense within $O(\mu)$ of the solution to the global regularized multitask optimization problem. Simulations are finally conducted to illustrate the theoretical findings and the effectiveness of the approach.

**Notation.** Vectors are column vectors, with random quanti-

<footnote>The work of Y. Zgheib and R. Nassif was supported by the ANR JCJC grant ANR-22-CE23-0015-01 (CEDRO project). Emails: {yara.zgheib, m.antonini, roula.nassif}@unice.fr.</footnote>
ties in boldface. Matrices are uppercase, while vectors and scalars are lowercase. The operator $\text{col}\{\cdot\}$ stacks column vectors, $\text{diag}\{\cdot\}$ forms a block-diagonal matrix, and $\preceq$ denotes element-wise inequality.

## II. PROBLEM FORMULATION AND DECENTRALIZED PROXIMAL-BASED APPROACH

### A. Problem formulation

We consider a network of $K$ agents, where each agent $k$ is associated with an $M \times 1$ parameter vector $w_k \in \mathbb{R}^M$ and a strongly convex risk $J_k(w_k)$, expressed as the expectation of some loss function $Q_k(w_k; x_k)$ (where $x_k$ denote the random data at agent $k$ and the expectation is w.r.t. the data distribution). The unique minimizer of $J_k(w_k)$ is denoted by $w_k^o$. If nodes $k$ and $\ell$ are neighbors, then their objectives or models, $w_k^o$ and $w_\ell^o$, are expected to differ only sparsely. Therefore, we propose to develop a decentralized strategy for collaborative learning that aims to solve the following global regularized cost:

$$W_{\eta}^{o} \triangleq \arg \min_{w} \left\{ J^{\text{glob}}(w) \triangleq \sum_{k=1}^{K} J_{k}(w_{k}) + \frac{\eta}{2} R(w) \right\}, \quad (1)$$

where $W \triangleq \text{col}\{w_1, \dots, w_K\}$ and $R(w)$ is a convex but non-differentiable regularization term given by:

$$R(w) \triangleq \sum_{k=1}^{K} \sum_{\ell \in \mathcal{N}_k} \rho_{k\ell} f_{k\ell}(w_k, w_\ell). \quad (2)$$

Here, $\mathcal{N}_k$ denotes the set of neighbors of agent $k$, excluding $k$ itself, $\rho_{k\ell} \ge 0$ controls the regularization strength between agents $k$ and $\ell$, and $f_{k\ell}(w_k, w_\ell)$ is a symmetric regularization function enforcing sparsity on the difference $w_k - w_\ell$. The regularization strength $\eta > 0$ controls the level of cooperation between agents. If $\eta = 0$, then each agent seeks to minimize its own cost without cooperating with neighbors. If $\eta \to \infty$, then all agents reach a global consensus by seeking a common parameter vector. Since the regularization is symmetric, i.e., $f_{k\ell}(w_k, w_\ell) = f_{\ell k}(w_\ell, w_k)$, we can rewrite $R(w)$ in (2) as:

$$R(w) = \sum_{k=1}^{K} \sum_{\ell \in \mathcal{N}_k} p_{k\ell} f_{k\ell}(w_k, w_\ell), \quad (3)$$

where the factors $\{p_{k\ell}\}$ are symmetric, and are defined as:

$$p_{k\ell} \triangleq \frac{\rho_{k\ell} + \rho_{\ell k}}{2}. \quad (4)$$

### B. Decentralized stochastic proximal gradient approach

In decentralized implementations, agent $k$ seeks to estimate its sub-vector, $w_{k,\eta}^o$, of $W_\eta^o = \text{col}\{w_{1,\eta}^o, \dots, w_{K,\eta}^o\}$ in (1) through local computations and exchanges with neighbors. In stochastic settings, the data distribution $x_k$ in $J_k(w_k) = \mathbb{E}Q_k(w_k; x_k)$ is generally unknown, making the true risk and its gradient unavailable. Instead, an approximate gradient vector is used based on available data [4]:

$$\widehat{\nabla}_{w_k} \widehat{J}_k(w_k) = \nabla_{w_k} Q_k(w_k; x_{k,i}), \quad (5)$$

where $x_{k,i}$ represents the data observed by agent $k$ at iteration $i$. The difference between the true gradient and its approximation, known as the *gradient noise*, can be expressed as [4]:

$$s_{k,i}(w) \triangleq \nabla_{w_k} J_k(w) - \widehat{\nabla}_{w_k} J_k(w). \quad (6)$$

In general, it is crucial to ensure that the randomness introduced by this noise does not destabilize the learning process.

In the following, we propose a stochastic-based decentralized approach to solve (1) using forward-backward splitting [28], [29], which is a powerful iterative technique for solving optimization problems of the form:

$$\min_{x \in \mathbb{R}^M} f(x) + g(x), \quad (7)$$

where $f(\cdot)$ is a real-valued, differentiable convex function with a gradient that is $\beta$-Lipschitz continuous, and $g(\cdot)$ is a real-valued, non-differentiable convex function. The forward-backward splitting algorithm for solving (7) is given by:

$$x_i = \text{prox}_{\gamma g} (x_{i-1} - \gamma \nabla_x f(x_{i-1})), \quad (8)$$

where the proximal operator of $\gamma g(x)$ at a given point $v \in \mathbb{R}^M$ is defined as:

$$\text{prox}_{\gamma g}(v) = \arg \min_x g(x) + \frac{1}{2\gamma} \|x - v\|^2, \quad (9)$$

with $\gamma$ a constant step-size chosen such that $\gamma \in (0, 2\beta^{-1})$.

Returning to (1), since the aggregate sum of individual costs is differentiable, the forward step takes the following form:

$$\psi_i = \text{col} \left\{ w_{k,i-1} - \mu \widehat{\nabla}_{w_k} J_k(w_{k,i-1}) \right\}_{k=1}^{K}, \quad (10)$$

where $w_{k,i-1}$ is the current weight vector at agent $k$, $\mu$ is a small positive step-size, and $\widehat{\nabla}_{w_k} \widehat{J}_k(w_{k,i-1})$ is the instantaneous gradient approximation at agent $k$, iteration $i$. Since the non-smooth function in (1) is $\eta R(w)$, we obtain the following forward-backward splitting algorithm for solving (1):

$$\mathcal{W}_i = \text{prox}_{\mu\eta R} \left( \text{col} \left\{ \mathcal{W}_{k,i-1} - \mu \widehat{\nabla}_{\mathcal{W}_k} \widehat{J}_k(\mathcal{W}_{k,i-1}) \right\}_{k=1}^{K} \right), \quad (11)$$

where $\mathcal{W}_i = \text{col}\{w_{1,i}, \dots, w_{K,i}\}$ is the estimate of $W_\eta^o$ at iteration $i$. While the gradient step in (11) can be implemented in a fully decentralized manner, the proximal step requires the computation of the proximal operator of $\mu\eta R(w)$, which cannot be performed in a decentralized manner. We instead propose a decentralized approach where each agent handles its own regularization locally. In particular, we investigate the following decentralized stochastic proximal gradient approach for solving (1):

$$\left\{ \begin{array}{l} \psi_{k,i} = \mathbf{w}_{k,i-1} - \mu \widehat{\nabla}_{\mathbf{w}_k} \widehat{J}_k(\mathbf{w}_{k,i-1}) \\ \mathbf{w}_{k,i} = \text{prox}_{\mu\eta g_{k,i}}(\psi_{k,i}) \end{array} \right. \quad (12a)$$

where $g_{k,i}: \mathbb{R}^M \to \mathbb{R}$ is a non-smooth regularization function given by:

$$g_{k,i}(w_k) = \sum_{\ell \in \mathcal{N}_k} p_{k\ell} f_{k\ell}(w_k, \psi_{\ell,i}). \quad (13)$$
The first step (12a), known as the self-learning step [6], is a stochastic gradient update on $J_k(w_k)$, yielding an intermediate estimate $\psi_{k,i}$. The second step (12b), known as the social-learning step [6], aggregates estimates from neighbors $\ell \in \mathcal{N}_k$ and applies the proximal operator of $\mu\eta g_{k,i}(w_k)$, enabling collaborative learning.

Efficient computation of the proximal operator in (12b) is critical, motivating the derivation of closed-form expressions for commonly used sparsity-based norms. It is sufficient to consider in this work $\ell_1$-norm co-regularization, which was previously investigated in the context of mean-square-error networks [24]. Let $\delta_{k,\ell} = w_k - w_\ell$ denote the parameter difference. Since the $\ell_1$-norm, which is defined as:

$$ \|\delta_{k,\ell}\|_1 = \sum_{m=1}^{M} |[\delta_{k,\ell}]_m|. \quad (14) $$

uniformly shrinks all the components of a vector, and does not distinguish between zero and non-zero entries, it is common to employ reweighted $\ell_1$-norm [30] defined as:

$$ \|\delta_{k,\ell}\|_{1,\text{rew}} = \sum_{m=1}^{M} \alpha_{k\ell}^{m} |[\delta_{k,\ell}]_m|. \quad (15) $$

where $\{\alpha_{k\ell}^m\}$ are positive weights chosen, at iteration $i$, according to $\alpha_{k\ell}^m(i) = \frac{1}{\epsilon + |[\psi_{k,i} - \psi_{\ell,i}]_m|}$, with $\epsilon$ a small constant preventing the denominator from vanishing [24], [30]. When we employ the $\ell_1$ or reweighted $\ell_1$-norm as co-regularizer, the function $g_{k,i}(w_k)$ in (13) takes the following form:

$$ \begin{aligned} g_{k,i}(w_k) &= \sum_{\ell \in \mathcal{N}_k} p_{k\ell} \left( \sum_{m=1}^{M} \alpha_{k\ell}^{m}(i) |[w_k]_m - [\psi_{\ell,i}]_m| \right) \\ &= \sum_{m=1}^{M} \phi_{k,i,m} ([w_k]_m), \end{aligned} \quad (16) $$

where

$$ \phi_{k,i,m}([w_k]_m) = \sum_{\ell \in \mathcal{N}_k} \boldsymbol{p}_{k\ell}^m(i) |[w_k]_m - [\psi_{\ell,i}]_m|, \quad (17) $$

with $\boldsymbol{p}_{k\ell}^m(i) = p_{k\ell}$ in the $\ell_1$-norm case and $\boldsymbol{p}_{k\ell}^m(i) = p_{k\ell}\alpha_{k\ell}^m(i)$ in the reweighted case. When implementing (12), we use the closed-form expression for the proximal operator of a weighted sum of $\ell_1$-norms, as derived in [24].

### III. NETWORK MEAN-SQUARE-ERROR STABILITY

Let $w_{k,\eta}^o$ denote the $k$-th sub-vector of the optimal solution $w_\eta^o$ in (1). In this section, we analyze the mean-square-error, $\mathbb{E}\|w_{k,\eta}^o - \boldsymbol{w}_{k,i}\|^2$, to assess how well the multitask strategy (12) approximates the optimal solution $w_\eta^o$, when $\ell_1$-norm co-regularization is employed, and under the following assumptions on the individual costs and gradient noise processes.

**Assumption 1.** (Conditions on the individual costs). Each cost $J_k(w_k)$ is twice differentiable and strongly convex, with a Hessian matrix function $H_k(w_k) = \nabla^2_{w_k} J_k(w_k)$ satisfying:

$$ 0 < \nu_k I_M \le H_k(w_k) \le \delta_k I_M, \quad (18) $$

for some $0 < \nu_k \le \delta_k$.

□

**Assumption 2.** (Conditions on the gradient noise). The gradient noise process defined in (6) is assumed to satisfy:

$$ \mathbb{E}[\boldsymbol{s}_{k,i}(\boldsymbol{w}) | \mathcal{F}_{i-1}] = 0, \quad (19) $$

$$ \mathbb{E}[\|\boldsymbol{s}_{k,i}(\boldsymbol{w})\|^2 | \mathcal{F}_{i-1}] \le \beta_{s,k}^2 \|\boldsymbol{w}\|^2 + \sigma_{s,k}^2, \quad (20) $$

for any $\boldsymbol{w} \in \mathcal{F}_{i-1}$, for some $\beta_{s,k}^2 \ge 0$ and $\sigma_{s,k}^2 \ge 0$, and where $\mathcal{F}_{i-1}$ denotes the filtration generated by the random processes $\{\boldsymbol{w}_{\ell,j}\}$ for all $\ell = 1, \dots, K$ and $j \le i - 1$. □

Regarding the $\ell_1$-norm based co-regularization, it is worth mentioning that the subdifferential set of $r_k(w_k, \{w_\ell\}_{\ell \in \mathcal{N}_k}) \triangleq \sum_{\ell \in \mathcal{N}_k} p_{k\ell} \|w_k - w_\ell\|_1$ is uniformly bounded, namely,

$$ \|v\| \le e_k, \quad \forall v \in \partial_{w_k} r_k(w_k, \{w_\ell\}_{\ell \in \mathcal{N}_k}), w_k, w_\ell \in \mathbb{R}^M \quad (21) $$

where $e_k$ is some non-negative constant. Before stating the theorem, we first define the mean-square perturbation (MSP) vector at time $i$ relative to $w_\eta^o$:

$$ \text{MSP}_i \triangleq \text{col} \{\mathbb{E}\|\boldsymbol{w}_{k,i} - \boldsymbol{w}_{k,\eta}^o\|^2\}_{k=1}^K. \quad (22) $$

**Theorem 1.** (Network mean-square-error stability). Under Assumptions 1 and 2, and when the $\ell_1$-norm based co-regularization is employed with a regularization strength $\eta = \kappa\mu^\alpha$, where $\alpha \ge \frac{1}{2}$ and $\kappa$ is a constant, and for $\mu$ satisfying:

$$ 0 < \mu < \min_{1 \le k \le K} \left\{ \frac{\nu_k}{\delta_k^2} \right\}, \quad (23) $$

the mean-square perturbation can be recursively bounded as:

$$ \text{MSP}_i \le A \text{ MSP}_{i-1} + \mu(c + \mu d), \quad (24) $$

where:

$$ A \triangleq \text{diag} \left\{ 1 - \frac{\mu \nu_k}{2} + \frac{2 \mu^2 \beta_{s,k}^2}{1 - \frac{\mu \nu_k}{2}} \right\}_{k=1}^K, \quad (25) $$

$$ c \triangleq \eta^2 \text{col} \left\{ \frac{8e_k^2}{\nu_k} \right\}_{k=1}^K, \quad (26) $$

$$ d \triangleq \text{col} \left\{ \frac{2\beta_{s,k}^2}{1 - \frac{\mu\nu_k}{2}} \|w_{k,\eta}^o\|^2 + \frac{\sigma_{s,k}^2}{1 - \frac{\mu\nu_k}{2}} \right\}_{k=1}^K. \quad (27) $$

A sufficient condition for ensuring the stability of recursion (24) is:

$$ 0 < \mu < \min_{1 \le k \le K} \left\{ \frac{\nu_k}{4\beta_{s,k}^2 + \frac{\nu_k^2}{2}} \right\}. \quad (28) $$

It then follows that:

$$ \left\| \limsup_{i \to \infty} \text{MSP}_i \right\|_{\infty} = O(\mu), \quad (29) $$

where $\|x\|_\infty = \max_m |x_m|$ denotes the $\ell_\infty$-norm.

Proof. Proof omitted due to space limitations. □
<figure><img src="image_3.png" /><figcaption>Fig. 1: Experimental setup. (Left) Network topology. (Right) Regression and noise variances.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2: (Left) Performance of algorithm (12) ($\mu_0$ = 0.0025). (Right) Benefit of collaborative learning.</figcaption></figure>

## IV. SIMULATIONS

We consider a MSE network with $K = 30$ agents, where each agent $k$ is observing a streaming sequence $\{d_k(i), u_{k,i}\}$ satisfying the linear regression model:

$$d_k(i) = u_{k,i}^{\top} w_k^o + v_k(i), \quad (30)$$

where $w_k^o$ is some $M \times 1$ vector to be estimated, and $v_k(i)$ denotes a noise term. The processes $\{u_{k,i}, v_k(i)\}$ are assumed to be zero-mean jointly wide-sense stationary with: i) $Eu_{k,i}u_{\ell,i}^{\top} = R_{u,k} > 0$ if $k = \ell$ and 0 otherwise; ii) $Ev_k(i)v_{\ell}(i) = \sigma_{v,k}^2$ if $k = \ell$ and 0 otherwise; and iii) $u_{k,i}$ and $v_k(i)$ are independent of each other. To estimate $w_k^o$, each agent can minimize the MSE cost defined as [5]:

$$J_k(w_k) = \frac{1}{2} \mathbb{E} \left( d_k(i) - \mathbf{u}_{k,i}^{\top} w_k \right)^2, \quad (31)$$

which is minimized at $w_k^o$. Alternatively, the agents can employ the cooperative strategy (12), which takes the following form when approximation (5) is employed:

$$\left\{ \begin{array}{l} \psi_{k,i} = w_{k,i-1} + \mu u_{k,i} (d_{k,i} - \mathbf{u}_{k,i}^{\top} w_{k,i-1}) \\ w_{k,i} = \text{prox}_{\mu\eta g_{k,i}} (\psi_{k,i}) \end{array} \right. \quad (32)$$

where the function $g_{k,i}(w_k)$ is given by (16). The network topology is shown in Fig. 1 (left). The variances $\sigma_{u,k}^2$ and $\sigma_{v,k}^2$, shown in Fig. 1 (right), are generated from the uniform distributions $U(1, 1.5)$ and $U(0.15, 0.25)$, respectively. The regularization weights are set to $\rho_{k\ell} = \frac{1}{\text{card}\{N_k\}}$ for $\ell \in N_k$, where $\text{card}\{N_k\}$ represents the cardinality of the set $N_k$. The models $w_k^o$ in (30) are generated according to:

$$w_k^o = w_c + \delta_k, \quad k = 1, \dots, 30, \quad (33)$$

where $w_c$ is a $15 \times 1$ vector randomly generated from the Gaussian distribution $N(0, I_{15})$. The vectors $\{\delta_k\}_{k=1}^{15}$ are $15 \times 1$ with all entries equal to 0 except for the $k$-th entry, which is equal to 1. Similarly, the vectors $\{\delta_k\}_{k=1}^{30}$ are $15 \times 1$ with all entries equal to 0 except for the $(k-15)$-th entry, which is equal to $-1$. Consequently, the individual models differ only sparsely across the nodes.

1) **Illustration of Theorem 1:** In order to illustrate the theoretical findings in Theorem 1, we set $\eta = \kappa\mu^\alpha$ with $\kappa = 50$ and $\alpha = 1$. In Fig. 2 (left), we report the network mean-square-deviation (MSD) learning curves:

$$\text{MSD}(i) = \frac{1}{K} \sum_{k=1}^{K} \mathbb{E} \| w_{k,\eta}^{o} - w_{k,i} \|^2, \quad (34)$$

of algorithm (32), for 3 different values of the step-size $\mu = \{\mu_0/2, \mu_0, 2\mu_0\}$. The results are averaged over 50 Monte-Carlo runs. For a given $\eta$, we compute the reference vector $w_\eta^o$ in (1) using the CVX solver [31] in deterministic settings where the data distribution is known. As it can be observed from Fig. 2 (left), in steady-state, the network MSD increases by approximately 3 dB when $\mu$ goes from $\mu_0$ to $2\mu_0$, confirming the $O(\mu)$ performance result from Theorem 1.

2) **Benefit of collaborative learning:** To illustrate the benefit of collaborative learning, we assess the algorithm's performance w.r.t. the local minimizers $w_k^o$, which correspond to the local models in (30). Specifically, rather than analyzing the MSD w.r.t. $w_\eta^o$ as in (34), we focus on the mean-square-error relative to $\{w_k^o\}$:

$$\text{MSD}^{\text{loc}}(i) = \frac{1}{K} \sum_{k=1}^{K} \mathbb{E} \| w_k^o - w_{k,i} \|^2, \quad (35)$$

where the superscript “loc” is used to emphasize that the error is evaluated w.r.t. the local models. We fix the step-size $\mu = 0.005$, and report the steady-state value of MSD$^{loc}(i)$ for different values of the regularization strength $\eta$. We compare two co-regularization norms: the $\ell_1$-norm and the reweighted $\ell_1$-norm ($\epsilon = 0.1$), as described in Sec. II-B. The individual models $\{w_k^o\}$ are generated as previously explained in (33). The global regularization strength $\eta$ is uniformly sampled from the interval $[0, 0.1]$. For each value of $\eta$, the resulting reported MSD$^{loc}$ is obtained by averaging the instantaneous mean-square-deviation MSD$^{loc}(i)$ in (35) over 200 samples after convergence of the algorithm (the expectation in (35) is estimated using 30 Monte-Carlo runs for all norms). As it can be observed from Fig. 2 (right), the $\ell_1$-norm and reweighted $\ell_1$-norm co-regularizers enhance the network performance compared to the non-cooperative solution ($\eta = 0$), with reweighted $\ell_1$-norm achieving a 6.5 dB improvement.

## V. CONCLUSION

We considered multitask learning over networks, where each agent seeks to estimate its own parameter vector, and where the parameter vectors at neighboring agents are assumed to differ sparsely. Using sparsity-based co-regularization and proximal-based techniques, we show that, under some general conditions on the risks and gradient noise, the network converges within $O(\mu)$ to the solution of the global regularized cost. Simulations illustrate the effectiveness of the approach.
REFERENCES

[1] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtarik, A. T. Suresh, and D. Bacon, "Federated Learning: Strategies for improving communication efficiency," in *NIPS Workshop on Private Multi-Party Machine Learning*, Barcelona, Spain, Dec. 2016.

[2] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, "Federated learning: Challenges, methods, and future directions," *IEEE Signal Processing Magazine*, vol. 37, no. 3, pp. 50–60, 2020.

[3] P. Kairouz and et al., "Advances and open problems in federated learning," *Foundations and Trends in Machine learning*, vol. 14, no. 1-2, pp. 1–210, 2021.

[4] A. H. Sayed, "Adaptation, learning, and optimization over networks," *Foundations and Trends in Machine Learning*, vol. 7, no. 4-5, pp. 311–801, 2014.

[5] A. H. Sayed, S.-Y. Tu, J. Chen, X. Zhao, and Z. J. Towfic, "Diffusion strategies for adaptation and learning over networks: an examination of distributed strategies and network behavior," *IEEE Signal Processing Magazine*, vol. 30, no. 3, pp. 155–171, 2013.

[6] R. Nassif, S. Vlaski, C. Richard, J. Chen, and A. H. Sayed, "Multitask learning over graphs: An approach for distributed, streaming machine learning," *IEEE Signal Processing Magazine*, vol. 37, no. 3, pp. 14–25, 2020.

[7] A. Nedic and A. Ozdaglar, "Distributed subgradient methods for multi-agent optimization," *IEEE Transactions on Automatic Control*, vol. 54, no. 1, pp. 48–61, 2009.

[8] A. Koloskova, S. Stich, and M. Jaggi, "Decentralized stochastic optimization and gossip algorithms with compressed communication," in *Proc. International Conference on Machine Learning*, vol. 97, Long Beach, California, 2019, pp. 3478–3487.

[9] J. Plata-Chaves, A. Bertrand, M. Moonen, S. Theodoridis, and A. M. Zoubir, "Heterogeneous and multitask wireless sensor networks—algorithms, applications, and challenges," *IEEE Journal of Selected Topics in Signal Processing*, vol. 11, no. 3, pp. 450–465, 2017.

[10] A. Z. Tan, H. Yu, L. Cui, and Q. Yang, "Towards personalized federated learning," *IEEE Transactions on Neural Networks and Learning Systems*, vol. 34, no. 12, pp. 9587–9603, 2023.

[11] V. Kulkarni, M. Kulkarni, and A. Pant, "Survey of personalization techniques for federated learning," in *Proc. Fourth World Conference on Smart Trends in Systems, Security and Sustainability*, London, UK, 2020, pp. 794–797.

[12] V. Smith, C. Chiang, M. Sanjabi, and A. Talwalkar, "Federated multitask learning," in *Proc. International Conference on Neural Information Processing Systems*, vol. 30, Long Beach, CA, USA, 2017, pp. 4427–4437.

[13] J. Plata-Chaves, N. Bogdanović, and K. Berberidis, "Distributed diffusion-based LMS for node-specific adaptive parameter estimation," *IEEE Transactions on Signal Processing*, vol. 63, no. 13, pp. 3448–3460, 2015.

[14] J. F. C. Mota, J. M. F. Xavier, P. M. Q. Aguiar, and M. Püschel, "Distributed optimization with local domains: Applications in MPC and

network flows," *IEEE Transactions on Automatic Control*, vol. 60, no. 7, pp. 2004–2009, 2014.

[15] G. N. Korres, "A distributed multiarea state estimation," *IEEE Transactions on Power Systems*, vol. 26, no. 1, pp. 73–84, 2010.

[16] V. Kekatos and G. B. Giannakis, "Distributed robust power system state estimation," *IEEE Transactions on Power Systems*, vol. 28, no. 2, pp. 1617–1626, 2012.

[17] R. Nassif, S. Vlaski, and A. H. Sayed, "Adaptation and learning over networks under subspace constraints – Part I: Stability analysis," *IEEE Transactions on Signal Processing*, vol. 68, pp. 1346–1360, 2020.

[18] P. Di Lorenzo, S. Barbarossa, and S. Sardellitti, "Distributed signal processing and optimization based on in-network subspace projections," *IEEE Transactions on Signal Processing*, vol. 68, pp. 2061–2076, 2020.

[19] R. Nassif, S. Vlaski, C. Richard, and A. H. Sayed, "Learning over multitask graphs – Part I: Stability analysis," *IEEE Open Journal of Signal Processing*, vol. 1, pp. 28–45, 2020.

[20] J. Chen, C. Richard, and A. H. Sayed, "Multitask diffusion adaptation over networks," *IEEE Transactions on Signal Processing*, vol. 62, no. 16, pp. 4129–4144, 2014.

[21] A. Koppel, B. M. Sadler, and A. Ribeiro, "Proximity without consensus in online multiagent optimization," *IEEE Transactions on Signal Processing*, vol. 65, no. 12, pp. 3062–3077, 2017.

[22] X. Cao and K. J. R. Liu, "Distributed efficient optimization for general network cost minimization problems," in *IEEE International Conference on Communications*, Kansas City, MO, USA, 2018, pp. 1–6.

[23] D. Hallac, J. Leskovec, and S. Boyd, "Network Lasso: Clustering and optimization in large graphs," in *Proc. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*, Sydney, Australia, 2015, pp. 387–396.

[24] R. Nassif, C. Richard, A. Ferrari, and A. H. Sayed, "Proximal multitask learning over networks with sparsity-inducing coregularization," *IEEE Transactions on Signal Processing*, vol. 64, no. 23, pp. 6329–6344, 2016.

[25] P. L. Combettes and V. R. Wajs, "Signal recovery by proximal forward-backward splitting," *Multiscale Modeling & Simulation*, vol. 4, no. 4, pp. 1168–1200, 2005.

[26] I. Wee, W. M. and Yamada, "A proximal splitting approach to regularized distributed adaptive estimation in diffusion networks," in *Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, Canada*, 2013, pp. 5420–5424.

[27] D. P. Bertsekas, "Incremental proximal methods for large scale convex optimization," *Mathematical programming*, vol. 129, no. 2, pp. 163–195, 2011.

[28] P. L. Combettes, D. Dũng, and B. C. Vũ, "Proximity for sums of composite functions," *Journal of Mathematical Analysis and applications*, vol. 380, no. 2, pp. 680–688, 2011.

[29] N. Parikh and S. Boyd, "Proximal algorithms," *Foundations and Trends in Optimization*, vol. 1, no. 3, pp. 127–239, 2014.

[30] E. J. Candes, M. B. Wakin, and S. P. Boyd, "Enhancing sparsity by reweighted $ℓ_1$ minimization," *Journal of Fourier Analysis and Applications*, vol. 14, pp. 877–905, 2008.

[31] M. Grant and S. Boyd, "CVX: Matlab software for disciplined convex programming, version 2.1," https://cvxr.com/cvx, Mar. 2014.