Tackling Feature and Sample Heterogeneity in Decentralized Multi-Task Learning: A Sheaf-Theoretic Approach
Authors: Chaouki Ben Issaid
Praneeth Vepakomma
Mehdi Bennis
Venue: Accepted by TMLR ()

Abstract: Federated multi-task learning (FMTL) aims to simultaneously learn multiple related tasks across clients without sharing sensitive raw data. However, in the decentralized setting, existing FMTL frameworks are limited in their ability to capture complex task relationships
and handle feature and sample heterogeneity across clients. To address these challenges, we introduce a novel sheaf-theoretic-based approach for FMTL. By representing client relationships using cellular sheaves, our framework can flexibly model interactions between
heterogeneous client models. We formulate the sheaf-based FMTL optimization problem using sheaf Laplacian regularization and propose the Sheaf-FMTL algorithm to solve it. We show that the proposed framework provides a unified view encompassing many existing federated learning (FL) and FMTL approaches. Furthermore, we prove that our proposed algorithm, Sheaf-FMTL, achieves a sublinear convergence rate in line with state-of-the-art decentralized FMTL algorithms. Extensive experiments show that although Sheaf-FMTL introduces computational and storage overhead due to the management of interaction maps, it achieves substantial communication savings in terms of transmitted bits when compared to decentralized FMTL baselines. This trade-off makes Sheaf-FMTL especially suitable for cross-silo FL scenarios, where managing model heterogeneity and ensuring communication efficiency are essential, and where clients have adequate computational resources.

Published in Transactions on Machine Learning Research (06/2025)
Tackling Feature and Sample Heterogeneity in Decentralized Multi-Task Learning: A Theaf-Theoretic Approach
Chaouki Ben Issaid
chaouki Benissaid@oouh.
Centre for Wireless Communications University of Oulu, Finland
Pranceth Vepakomma
vepakom &ww.t edu
MBZUAI and Massachusetls Institute of Technology
Mehdi Bennis
mehdi. Bewnis
Centre for Wireless Communications
University of Oulu, Finland
Reviewed on OpenReview: https: // openreview. net/ forum? id= JtP
Abstract
Federated multi-task learning (FMTL) aims to simultaneously learn multiple related tasks across clients without sharing sensitive raw data. However, in the decentralized setting, existing FMTL frameworks are limited in their ability to capture complex task relationships and handle feature and sample heterogeneily across clients. To address these challenges, we introduce novel theaff-heoretic. based approach for FMTL. By representing client re- lationships using cellular sheaves, our framework can flexibly model interactions between heterogeneous client models. We formulate the sheaf-based FMTL optimization problem using sheaf Laplacian regularization and propose the Sheaf-FWTL algorithm to solve it. We show that the proposed framework provides unified view encompassing many existing federated lcarning (FL) and FMTL approaches. Furthermore, we prove that our proposed algorithm, Sheaf-FMTL, achieves a sublinear convergence rate in line with state-of-the-arl decentralized FMTL algorithms. Extensive experiments show that although Sheaf-FWTL in- troduces computational and storage overhead due to the management interaction maps, achieves substantial communication savings terms of transmitted bits when compared to racened FMTL baselines. This trade-off makes -hea especially suitable for cross-silo FL scenarios, where managing model heterogeneity and ensuring communication efficiency are essential, and where clients have adequate computational resources.
1 Introduction
The growing demand for privacy-preserving distributed learning algorithms has steered the research com- munity towards federated learning (FL) (McMahan et al. 2017), a learning paradigm that allows several clients, such as mobile devices or organizations, to cooperatively train a model without revealing their raw data. By aggregating locally computed updates rather than raw data, FL aims to learn global model that benefits from the different data. distributions inherently present across the participating clients. Despite its promise, conventional FL faces significant hurdles when dealing with client data heterogeneity. In fact, while the global model may perform well on average, the statistically heterogencous clients' data have been shown to affect the model's existence and convergence (Sattler et. al.) 2020; Li el. al.) 2020b). These challenges are exacerbated in a decentralized environment where coordination is limited and direct control over the client models not feasible. Recently, there have been several attempts to bring personalization into FL learn distinct local models (angg al.) [2019] Fallah et al. 20200] Hanzely & Richtárik] 2020) since learning personalized model per client is more suitable than single global model to tackle data heterogeneity.

---

Published in Transactions on Machine Learning Research (06/2025)
These models are specifically learned to fit the heterogencous local data distribution via techniques such federated multi-task learning (FMTL. (Smith et al. 2017 Dinh et al. 2022 that model the interactions between the different personalized local models.
FMTL generalizes the FL framework by allowing the simultancous learning of multiple related but distinct tasks across several clients. Unlike traditional FT., which focuses on training a single global model, FMTT, acknowledges that different clients may be interested in solving distinct tasks that are related but not iden- tical. By leveraging task relatedness, FMTL aims to improve the performance of individual task models through shared knowledge while maintaining task-specific uniqueness. This approach not only enhances the generalization performance of the models on individual tasks by leveraging shared information but also con- tributes to tackling the non-independent and identically distributed (non-IID) nature of data across clients. FMTL considers the different objectives and data distributions across clients, customizing models to per- form optimally on each task while still benefiting from the federated structure of the problem as well as the similarity between these tasks. However, existing FMTL frameworks are not without limitations. A major concern is the oversimplified view of task interdependenciess where relationships between tasks are often modeled using simple fixed scalar weights. This approach captures only the basic notion of task re- latedness but fails to represent more intricate and higher-order dependencies that may exist among tasks. As result, these models may overlook subtle interconnections and dynamic patterns of interdependence, leading to suboptimal knowledge sharing and reduced performance in heterogeneous and decentralized en- vironments. For a comprehensive understanding of scenarios where task similarities are naturally defined in vector spaces, kindly refer to Appendix C Furthermore, a critical issue with the current FMTL frame- works is the assumption that the models have the same size, which limits the applicability of FMTL in the case of different model sizes. These challenges are particularly pronounced in cross-silo FL scenarios, where collaborating organizations may possess large, complex datasets with significant heterogeneity and may use different model architectures, necessitating a flexible and robust learning framework. Last but not least, to the best of our knowledge, apart from MOCHA (Smith et al. 2017]. which requires the presence of a server, the interactions between the tasks are assumed to be known and not learned during training. Therefore, to address these challenges, we seek to answer the following question:
"How we effectively model and learn the complex interactions between various tasks/models in an FMTL decentralized setting, even in the presence of different model sizes?"
To answer this question, the concept of sheaves provides a novel lens through which the interactions between clients in a decentralized FMTL setting can be modeled. The mathematical notion of a sheaf initially invented and developed in algebraic topology, is a framework that systematically organizes local observations in a way that allows one to make conclusions about the global consistency of such observations (Robinson) 2014: 2013 Riess & Ghrist, 2022). As such requirements are a central part of FL problems, it is natural to ask how one could utilize sheaf-based framework to find an effective solution to the above question. Given underlying topology of the client relationships, we employ the notion of a cellular sheaf that captures the underlying geometry, enabling a richer and more manced multi-task learning environment. Sheaves enable the representation of local models as sections over the underlying space, offering a structured way to capture the relationships between fasks/modee in FMTL settings. As an inherent feature of the sheaf-based framework, our approach can support heterogeneity over local models. More specifically, our framework naturally facilitates learning models with different model dimensions. Morcover, sheaves are inherently distributed in nature and hence facilitate ralieed training. A sheal data structure consists of vector spaces and linear mappings between them. In this work, we model the underlying space as graph, and vector spaces are defined over vertices and edges, capturing pairwise interactions. Crucially, we are required to learn these maps that constitute the sheaf structure, as a decision variable of our problem. Learning these maps is instrumental in comparing heterogeneous models by projecting them onto a common space.
Contributions. This paper introduces a novel unified approach that fundamentally rethinks FMTL and gives it new interpretation by incorporating principles from sheaf theory. Our work primarily targets cross-silo FL environments. Unlike cross-device FL involving millions of resourco-constrained edge devices, cross-silo settings typically feature fewer but more computationally capable participants with substantial local computing resources. This context is particularly well-suited for exploring richer collaborative learn-
2

---

Published in Transactions on Machine Learning Research (06/2025)
ing frameworks that can effectively handle heterogencity in exchange for some additional computational overhead. In what follows, we summarize our main contributions
Our proposed framework demonstrates a high degree of fiexibility as it addresses the challenges arising from both feature and sample heterogeneity in the context of FMTL exploiting sheaf theory. It may be regarded as comprehensive and unified framework for FMTL, as it encompasses mul- litude of existing frameworks, including personalized FT. (Hanzely & Richtárik 2020), convent FMTL (Dinh et al.) 2022), hybrid FL (Zhang et al. 2024), and conventional FL (McMahan et al., 2017
To the best of our knowledge, this is the first work that proposes to solve the FMTL in a decentralized setting while modelling higher-order relationships among heterogeneous clients. Furthermore, unlike existing decentralized FMTL, frameworks, we learn the interactions between the clients as part of our optimization framework.
Our algorithm, coined Sheaf-FNTL, exhibits high communication efficiency, as the size of shared vectors among clients is significantly smaller in practice compared to the original models. This ad- vantage is accompanied by computational and storage overhead due to the management of restriction maps, making the approach especially well-suited for cross-silo FL environments where clients have sufficient computational resources.
A detailed convergence analysis of our proposed algorithm shows that the average squared norm of the objective function gradient decreases at rate of (1/K)), where K is the number of iterations,
recovering the convergence rate of state-of-the-art FMTL. (Smith et al. 2017 Dinh et al. 2022
Extensive simulation results demonstrate the performance of our proposed algorithms on several benchmark datasets compared to state-of-the-ase approaches.
2 Related Work
Federated Learning (FL). FL is designed to train models on decentralized user data without sharing raw data. While numerous FL algorithms (McMahan al. 2017: Karimireddy et al. 2020 Li et al.) [020a]] Lin et al., 2020 Elgabli et al. 2022) have been proposed, most of them typically follow similar iterative procedure where server sends global model to clients for updates. Then, each client trains local model using its data and sends it back to the server for aggregation to update the global model. However, due to significant variability in locally collected data across clients, data heterogeneity poses a serious challenge. A prevalent assumption within FL literature is that the model size is the same across clients. However, recent works (hhang el al., 2024 Liu et al.) 2022b) have highlighted the significance of incorporating heterogeneous model sizes in FT. frameworks in the presence of parameter server.
Personalized FL (PFL). To address the challenges arising from data heterogeneity, PFL aims to learn individual client models through collaborative training, using different techniques such local fine-tuning
(Wang et al. 2019 Yu et al.] 2020),000.. meta-learning (Fallah et al. 2020 Chen et al.
2018 Jiang et al.
2019/, layer personalization
[Arivazhagan et al. 2019 Liang et al. 2020 Collins et al.)
2021), model mixing
(Hanzely & Richtárik 2020
Deng et al.] 2020a and model-parameter regularization T Dinh et al. 2020
Li et al. 2021a] Huang et al. 2021 Liu et al.
2022a One way to personalize FT. is to learn a global
model and then fine-tune its parameters at each client using a few stochastic gradient descent steps, as
(Yu et al. 2020). Per-FedAAg (Fallah et al. 2020) combines mela-learning with FedAvg to produce better initial model for each client. Algorithms such as FedPer (Arivazhagan et al., 2019), LG-FedAvg {Liang et al, 2020), and FedRep (Collins et al. 2021) | involve layer-based personalization, where clients share certain layers while training personalized layers locally. A model mixing framework for PFL, where clients learn a mixture of the global model and local models was proposed in (Hanzely & Richtárik 2020). In T Dinh et al, 2020 PFedMe uses an Le regularization to restrict the difference between the local and global parameters.
Federated Musk Learning (FMTL). FMTL aims to train separate but related models simultanc- ously across multiple clients, each potentially focusing on different but related tasks. It can be viewed as
3

---

Published in Transactions on Machine Learning Research (06/2025)
form of PFL by considering the process of learning one local model as single task. Multi-task learning was first introduced into FI, in (Smith et al.) 2017). The authors proposed MOCHA, an FMTL, algorithm that jointly learns the local models as well as a task relationship matrix, which captures the relations between tasks. In the context of FMTL, task similarity can be represented through graphs, matrices, or clustering. In (Sattler et al. 2020), clustered FI., an FI. framework that groups participating clients based on their local data distribution was proposod. The proposed method tackles the issue of heterogeneity the local datasets by clustering the clients with similar data distributions and training personalized model for each cluster. FedU, an FMTL algorithm that encourages model parameter proximity for similar tasks via Laplacian reg- ularization, was introduced in (Dinh et al. 2022). In (SarchehmmehPon Pour et al.] 2023). the authors leverage a generalized total variation minimization approach to cluster the local datasets and train the local models for ralized collections of datasets with an emphasis on clustered FL.
Data Heterogeneity in FL. fundamental challenge in FL. is the inherent heterogeneity of data across clients, commonly referred as non-IID data (Kirrouz et al.) 2021 Li al., 2020a). Following taxonomics Li el. al.
proposed in the literature (Zhao et. al.] 2018 Hsieh et. al.; 2020
2022p, data heterogeneily can be categorized into several distinct types. Feature distribution skew (covariate shift) occurs when clients have different marginal feature distributions P(X) while conditional label distributions P(Y[X) remain
consistent across clients (Quionero-Candenn Candela et al. 2009 Koh et al. 2021). This is common in scenarios where data collection environments differ, such as medical imaging equipment varying across hospitals (Antunes et al. 2022). Label distribution skew arises when label distributions P(Y) vary across clients but feature distributions conditioned on labels P(X remain similar (Wang et al. 2020a) Li et al.) 2021b). For instance, some clients may have more instances of certain classes than others, reflecting demographic or geographical differences (Hsu et al., 2019). Concept shift occurs when the relationship between features and labels P(Y[X) differs across clients due to local environmental factors or preferences (Karimireddy et al. Luo et al.) 2021). This manifests as either different feature manifestations for the same labels (Tan 2022) or entirely different decision boundarics across clients for conceptually similar tasks (Collins Quantity skew refers to a significant imbalance in the amount of data possessed by each client
2020
ct a..
et al.
2021j.
(Wang et al.
2020b Duan et al. 2019), creating a disparity in their contributions to the global model.
Sheaves. A major limitation in FMTL over a graph, e.g.; FMTL with graph Laplacian regularization in (Dinh et al. 2022) and FMTL with generalized total variance minimization in (SarcheshmchPour et al., 2023), that we wish to address in this work, is their inability to deal with feature heterogeneity wetw clients. In contrast, sheaves, well-established notion in algebraic topology, can inherently model higher- order relationships among heterogencous clients. Despite the limited appearance of sheaves in the engincering domain, their importance in organizing informat ion /aata distributed over multiple
has been emphasized the recent literature (Robinson) 2014: 2013 Riess & Ghrist 2022). In fact, sheaves can be considered as the canonical data structure to systematically organize local information so that useful global information can be extracted (Robinson) 2017). The above-mentioned graph models with node features lying in some fixed space can be considered as the simplest examples of sheaves, where such a graph is equivalent to a constant sheaf structure that dircetly follows from the graph. Motivated by these idcas, our work focuses on using the generality of sheaves to propose a generic framework for FMTL with both data and feature heterogeneity over nodes, generalizing the works of (Dinh et al.] 2022 Sarcheshmehhouu et al., 2023). The analogous generalization of the graph Laplacian in the sheaf context is the sheaf Laplacian. In the context of distributed optimization, (Hansen & Ghrist 2019) consider sheaf Laplacian regularization with sheaf constraints, i.e., Homological Constrannts, and the resulting saddle-point. dynamics.
3
Sheaf-based Federated Multi-Tase Learning (Sheaf-FMTL)
3.1 FMTL Problem Setting
We consider a connected network of N clients modeled by a graph 9 = (V,8), where -= (1,...,N) is the set of clients, and E S V x V represents the set of edges, i.e., the set of pairs of clients that can communicate with each other. Each client E V has a local loss function fi Rd: - R and only has access to its own local data distribution D.. Client i can only comminicate with the set of its neighbors defined No = (j (i, E) whose cardinality is Wil = 8, In this work, we aim to fit. different models, i.e.,
4

---

Published in Transactions on Machine Learning Research (06/2025)
Client 0, E F(k) = R%%
Projection
Projection and compariton
Fisley on Pir
Fedtes Pii
PP P, Os
[Pa0, ....
Fica) RR
Edge 8, (1.K)
Edge 02 = (1,4)
Fisay = P
Fign Pas
0, F()) R4
0, F(U R4
Client
Client
Figure 1: Schematic illustration of the sheaf- based modeling of FMTL.
0, E R&, Vi E [N], to the local data of clients, while accounting for the interactions between these models.
Finally, let 0 10 ..... OZ Rd be the stack of the local decision variables {0.) where d = E d,
3.2 A Sheaf Theoretic Approach of the FMTL Problem
A "cellular sheaf" F of R-vector spaces simple graph g (V,E), i.., without loops and multiple edges, consists of the following assignments
For each i E V, vector space F(i) = Rd of dimension d,.
for each edge (i, E, a vector space F(e) IRd of dimension daj, and
for each edge e E & and a vertex i E V that is incident to the edge e, linear transformation Fige: : F(i) + F(e).
We shall refer F(i) and F(e) as stalks over and e, respectively, and the map Fige as the restriction map from to e. Also, given edge = (i,j) &, we denote the matrix representation of Fise, with respect to chosen basis such as the standard basis, by P.j. With an abuse of notation, use F,ge and Pij interchangeably, required, in the remainder of the paper.
Naturally associated with such sheaf structure are the dual maps, Fi F(e) + F(i), of the restriction maps. Note that we are using the identification of the dual of a finite-dimensional vector space with itself. It a standard fact that the matrix representation of the dual map Fise given by the transpose of P.j. Similarly, with an abuse of notation, we shall also use Fise and pf interchangeably.
For each i E V, F(t) is the space in which the local model of client is parameterized, i.c., O, E F(). A choice (0. of local models lies the total space (F) == F(i). Note that, we do not assume de and dj be the same for i # j. In particular, different clients can have different model sizes that could arise from feature heterogeneity and/or different learning tasks. Also note that an element of ()) is not fully observable by single client, assumed in the FL setting.
Therefore, any two models can be compared via the restriction maps, provided they share an edge. More specifically, as can be seen from Figure [] for two clients and j such that = (i.j) E E, F(e) can be considered as the "disclose space" in which models 0, and 0, are compared via the projections Fise (0.) = P. and Fi (0,) = Pj-- Here, the local models 0 are assigned to the vertices (the clients), while the restriction maps Ps project the local models onto the edge space capturing the shared features or relationships between the clients.
5

---

Published in Transactions on Machine Learning Research (06/2025)
We refer the reader to Appendix B for an interpretation of this viewpoint in the context of linear models. Given a choice of local models, the overall comparison of these models is done in the total space (F) == Dee F(e) of the disclose spaces, The total discrepancy of such a choice of local models, as measured by comparing their projections onto the disclose spaces, can be formalized via the Laplacian quadratic form associated with the so-called "sheaf Laplacian",""" the analogous to the graph Laplacian. To define the Laplacian in the sheaf setting, we first need to orient the edges and define the "co-boundary map" 8 (F) - CF)...
From now onwards, shall fix orientation for each edge and write (i.j) for an oriented edge. For such an oriented edge (i.j), write et j and e i. Our discussion is not subjective to the choice of orientation; however, one can choose canonical orientation associated with an ordering of the vertices, c.g. when vertices are indexed by numbers, by choosing e+ max(i,j} and e = min(i, j} for an unoriented edge e e= = {i. Given such an orientation, the co- co- bouudnd map 6 is given as follows. For 0 (0,)ev, the co-boundary of 0, 5 (0) (6 (0)))eEe E (F), is defined by
(0) Fe se (0.)) ( (0
(1)
As in the case of restriction maps, one also has the dual 8 (F) + (F) of the co- co- boundary map 8, The sheaf Laplacian LF (F) + (F) associated with the cellular sheaf F is then given by LF 8 ob. For given 0, the sheaf Laplacian is defined (0) LL given by
(Lx(0); (0);) JEV
where LF (0); Eiev L,j, (0.) is
F o Fise i -Fjse o Figes if e- E E otherwise.
if
(2)
0.
In particular, based on the ordering of V that is used to stack the local models, and the chosen bases for F(i)'s and F(e)'s, Ly is a block matrix structure indexed by V, whose (j,i) block can be directly obtained from (2). More specifically, with an abuse of notation, writing Lis in terms of its matrix representation, we have that
Σ Pf of
Pay.
if
JEN,
T,
PP.
if ef (i.)) E
(3)
0,
otherwise.
In fact, one write the matrix representation of the co-boundary map, P, as follows. It has block structure whose rows are indexed by edges and columns are indexed by vertices. Then, the submatrix Pa.; that corresponds to row e E & and column i E V is given by
Pop. if e = (i,j)
Pej
Poo
e (j,
(4)
0, otherwise.
From the definition Ly 8" o 6, one get the matrix form of Ly Ly PTP. Assuming the matrix representation of LF, often write L5 (0) L 0 p PO. Note that this block structure aligus well with the distributed optimization goal of the FMIL setting.
The significance of the sheaf Laplacian Ly is characterized by the consonsus property given by
ker(Lx) (0))) (F) Fige (0.) Fjge (0z) for e (i.j) E).
(5)
In other words, ker(L consists of the choices of local models that are in global consensus that any two comparable local models 0 and 0 agree when projected onto the disclose space F(e), where (i.j) E E. Accordingly, the global consensus constraint on 0 E CO is given by L 0 -
Similar to that of a graph Laplacian, the sheaf Laplacian quadratic form (= 0- = 0 L, 0 quantifies by how much a given 0 deviates from the constraint I 0 O. The following lemma shows that the sheaf Laplacian quadratic form measures the total discrepancy between the projections of the local models onto the edge spaces, summed over all edges in the graph.

---

Published in Transactions on Machine Learning Research (06/2025)
Lemma 1.
Σ
Fege (0.) - Fzze (0))1
Σ
PP - P
(6)
c=(b))E
e=(+))
Proof. The details of the proof are deferred to Appendix D
Next, show that 0 being global section, i.c., E ker(Ly), is equivalent to 0 minimizing the sheaf Laplacian quadratic form.
Lemma 2.
(F) arg min Q=(0) = arg min 0 LF (EC%(F) (EC*(F)
(7)
Proof. The proof is provided in Appendix D
In the context of FMTL, cellular sheaf is structured way to assign information to each client (node) and their connections (odges) in network as follows
Nodes (Clients): Each client i has its own local model 0; E R44.
Edges (Interaction space): Each connection between clients and j has a shared space F(e) E Rds,
Restriction Maps:
The mappings P.j R4 + Rdy project the local model of client into the shared space with client j. This projection facilitates meaningful comparisons and collaborations between clients, ensuring that heterogeneous models can still interact effectively within the network.
This setup allows us to systematically ensure that the models of connected clients align within their shared interaction spaces, promoting consistency and cooperation across the network. To formulate the FMTL opti- mization problem, we aim to minimize the combined objectives of individual client losses and a regularization term that enforces consistency across client models. Specifically, each client seeks to minimize its own loss function based its local data, while the regularization term penalizes discrepancies between connected clients in their shared interaction spaces. Building upon this, we can express the regularization term more succinctly using the sheaf Laplacian matrix (P(P). This matrix encapsulates the structural relationships and shared interaction spaces between clients, allowing us to reformulate the optimization problem a compact and matheratticalyy elegant manner as follows
\ TT T.
min (0, P) P)= 5(0) P.P
where S(0) -i and (P(P) is the sheal Laplacian for the choices of the restriction maps. The sheaf Laplacian regularization term L0"L=(()e enforces consistency between the projections of the local models onto the edge space, promoting collaboration among the clients. more in-depth discussion on the rationale behind using Sheaf theory to model clients' interaction in the context FMTL can be found in Appendix In the next subsection, we show that our proposed framework very general and covers many special cases previously introduced in the literature.
3.3 Special Cases
In this section, we show how some previous works can be considered as special cases of our framework. Most the works that we refer to, except (Zhang et al. (024)), consider the same model size for all the clients. Thus, unless otherwisc stated, we assume the rest of this subsection that all the local models are assumed have the same size p N, ie., Vi E V.O, E Rº. In particular, chose Vi V and Ve E E, F(i) F(e) = RRP.

---

Published in Transactions on Machine Learning Research (06/2025)
Connection with conventional FMTL (Dinh et al., 2022) In conventional FMTL, aim to solve the problem
N
min he
f((0.) + 2 JEN;
ai 10: - 0
(9)
where the weights {aiy] are assumed be known in advance. This problem special case associated
with the sheaf F arising by choosing Fige P.j Vaijpp where I is the p p identity map/matrix. The dimension of the projection space is dıj = p for all edges (i,j). For this particular choice of the sheal F, the associated Laplacian quadratic form is precisely
QF 0" L=
Σ i,j4e
ai 100 0,1 (E,)EE
Σ
ai 10, -,,1²
(10)
Replacing this into 17 we get the conventional FMTL problem
(
Connection with conventional FL (Ye et al., 2020). Setting the restriction maps to be P. Ip and de - P. v(i,j) E [M] x [N]. Then, the sheal Laplacian regularization is given by
QN(0)= Σ 100 - 0
(11)
(iE)E
Hence, 17
reduces to
X
min (0
f((0.)
Σ
10: 0
(12)
(L,))E
Taking + as pointed out in Remark
one recovers the conventional FL problem
N
min hev 1=1
((OO.)
s.t 0 oj. ((i,j) E E.
(13)
Connection with personalized FL (Hanzely et al. 2020). Introducing the client 0, e.g., a server, where fo _ 0 and 0g = Furthermore, let Po: Ps I,. Vi E [N], and P.j Op. Vi, E [M] x [N], where Op the p x p zero map/matrix. Hence, the set neighbours client is No = [N], and the set of each client e [N] MG {0). We observe that this amounts to choosing the constant sheaf over the graph that connects each client the server. Then, the associated Laplacian quadratic form can be written
Qr (0)=
Σ Po:00 Pio0,12 Σ 0 10-0,11- iENo
0,1
(14)
Therefore, 17 reduces to the personalized FL objective
(Hanzely et al. 2020 Eq. (2))
N f((0.) Σ 00 0|
N
min hev
+
(15)
2
Connection with hybrid FL (Zhang et al. 2024 In this case, communication framework is estab- lished between a server (with index 0) and set of clients (i E [M]), with each client connected solcly to the server. The server has access to all features, while clients are constrained by their local features. Hence, for each client i, d, S do, where 0, E g4 and Og E R& are the models of client i and the server, respectively. Let II, denote binary matrices that prune the server model to align with the client local model, referred as the selection matrices. Given the above description, we have F(i) d and F(e) Rdd for every i [N] and edge of the form (i, 0). The associated restriction maps are Po: II, and Po Id,; for E [N] and
8

---

Published in Transactions on Machine Learning Research (06/2025)
P,y= Op. v(1,)) E [N] [N]. With these choices, the Laplacian quadratic form of this sheaf is equal to the
regularizer term
QF III III
Σ 0. -1,00
(16)
Replacing this into 17
we get the hybrid FL objective Zhang et al. 2024 Eq. (6) given 141 0)-
3.4 Proposed Algorithm & Convergence Analysis
Note that problem
8
arising from the sheaf formulation can be re-written as follows
N Σ f((0,) + 2 i=1 JEN.
min hee (P.
Σ
Poj0: - P3:0,11?
(17)
where || |- the Euclidean norm, and (ii E, Pij is a matrix with size (dij. de) such that di = dzi- In 117 we propose to jointly learn the models H0. and the matrices (P.jhcs))e-- The matrix P.j can be seen as an encoding or compression matrix since it maps the higher-dimensional vector 0 to lower- dimensional space with dimension dij, effectively retaining only the most important features or information shared between the two clients i and j. Hence, the term (P.j0, Pj(O,) captures the dissimilarity discrepancy between the two vectors 0, and 0, in this shared subspace.
Remark 1. In (17), the hyperparameter X controls the impact of the models of neighhoring clients on each local model. When x > O, the minimization of the regularization term promotes the prozimity among the models of neighboring clients. On the other hand, if X == 0, (17) reduces to an individual learning problem, wherein each client independently learns its local model 0, solely from its local data, without engaging in any collaborative efforts with the other clients. Finally, as X - oc. (17) boils down to the classical FL problem
where the aim is to learn a global model (Hunzely 8 Richlárik 2020
Next, we propose a cummunication-efficiee algorithm to solve 17 by adopting an iterative optimization approach. Since the objective function is assumed to be differentiable, we can use gradient-basod optimization methods. More specifically, we will use alternating gradient descent (AGD) updates for [0theeN] and (Py) respectively. At iteration (k + 1), client sends Of and receives (P)0)ee from its neighbours, update model 0,, using one gradient descent stop
0 -
V/HO))=> Σ (Ph) (P (P) (P - JENS
(18)
Then, client sends Ph g and receives (P Jje, from its neighbours, to be able to update its matrices (Pylen,, using one gradient descent step, according to
P - -p. - ph P
ph+l=
(19)
where and are two learning rates. Note that, in our proposed algorithm, neighbouring clients only share vectors and no matrix exchange needed in both updates (88 and 19 Our proposed method is summarized in Algorithm [
Remark 2. Note that each neighbour of the node is required to send the vector PjO0, in order to update 0, and P,j as per [18) and (1). The dimension of P, is dij, which in practice could be much smaller than d;, the size of 0, For example, a reasomable choice of dij is di. = min(d,,d,). Hence, our proposed algorithm more
cotarauiiiiaiio-ffffiieee. than sending the models [0.)iev.
Remark 3. It is crucial to initialize the restriction maps Pd with non-zero values (e.g., randomly, as done in our experiments). If inétialized as zero matrices, the gradient update for P.ş in (19) would always be zero, preventing the learning of interactions and causing the algorithm to reduce to independent local training.
9

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm Sheaf-based Federated Multi-Task Learning (Sheaf -FMLL
Parameters: Number of clients N, number of iterations K, learning rales (ax, 11), regularization param-
eter X.
Initialization: Initial models (0 Jimi: initial matricos {P.) ((S)EE-
3: for k = o,.... K do
4:
for i= 1..... N in parallel do
5:
> Sends Of and receives P from neighbors j E N₁
b Updates its model:
0 = 0¹
VS((0!)) + Σ (PP) (P P;:0.)
JEN
> Sends Of and receives Of from neighbors j No
7:
8:
b Updates its matrix:
pk+1 P - -nA( +)
end for 10: end for
Next, we turn to analyzing the convergence of Sheaf-FNTL. To this end, we start by making the following standard assumptions.
Assumption (Smoothness). Vi E [N] the function f, is assumed to be L-smooth, i.c., there exists I. > 0 such that, Vi E [N], v0,, 02, [[V] - (.,(ll))ll - L||02- - 01 ||-
Assumption 2 (Bounded domain). There exists Dg such that ||]]| S Dg.
Assumptions 1-2 are key disumpplinss that are often used in the context of distributed optimization KKrrim- freddy et al. 2020 Li et al. 2020a Deng et al. 2020b 2023 In particular, Assumption 2 is commonly used in the convex-concave minimax literature (Deng et al. 2020b 2023). The following theorem establishes the convergence rate the SheLf-FMTL algorithm.
Theorem 1. Let Assumptions 1 and 2 hold, and the learning rates and satisfy the conditions a < NL and 1) < x respectively. Then, the averaged gradient norm is upper bounded as follows
2
2
1 (V(00, po) TT*).
(20)
K
pk
where p= min fa (1 _NL) .7 (1- (1-mpp))
TADi
and x is the optimal value of V.
Proof.
The proof can be found in Appendix E
Theorem shows that the Sheaf-FMTL algorithm converges to stationary point of the objective function al rale of O(1KK). The proof involves two main steps. First,, we study the descent. step in the variable gk+1 and then, we study the descent step in the variable pk+1
4 Experiments
4.1 Experimental Setup
To validate our thcoretical foundations, we numerically evaluate the performance of our proposed algorithm
Sheaf-PWTL using two experiments: (i) the clients have the same model size in Section 4.2 and (ii) the
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Regularization parameters used during training.
Datasct
HAR
Vehicle
GLEAM
School
R-MNIST
H-CTFAR-10
Regularization parameter (A)
0.05
0.001
0.001
0.01
0.001
0.001
clients have different model sizes in Section 4.3
Datascts. In the first experiment, examine two datascts: rotated MNIST and heterogeneous CIFAR-10. A detailed description of the datasets can be found in Appendix F In appendix G we report additional results using four more datasets.
Model ureh For image datasets, we employ CNN architectures: rotated MNIST uses a CNN with two convolutional layers (32 and 64 filters) followed by a fully connected layer outputting 10 classes, while heterogeneous CIFAR-10 uses a similar architecture adapted for RGB inputs. For tabular datasets, we use multinomial logistic regression for classification tasks and lincar models for regression tasks. In Sections 1.2 and [.4] where clients have the same model size, the entire network architecture, including the final classification layer, is identical across all clients. In Section 4.3 investigate FMTL in heterogeneous setting where clients have different model architectures. To simulate devices with varying computational capabilities, we employ different CNN architectures summarized in Table 2 These models are distributed among clients in almost equal proportions, creating a heterogeneous federation.
Evaluation. In the first experiment, we use a train/test split ratio of 75%/25% for all datasets, as done in (mmthh et al., 2017), where the test sot for each client maintains the same data distribution characteristics (e.g., feature skew, label distribution) as their respective training set. Similar to (Dinh el. al., 2022), we reduce the data size by 80% for half of the clients to mimic the real-world FL setting where some clients have small datasets and can benefit from collaboration. For the regression tasks, we consider the loss function to be the regularized L2 loss, while we use the Le-regularized cross-entropp loss for the classification tasks. For each experiment, we report both the average test accuracy MSFF and its corresponding one standard error shaded area based on five runs.
Parameters and hyperparameters, In Sections 4.2 and 4.4 where all clients have the same model dimen- sion d, we set dı =[d]] for all edges (i.j) E E, where y (0. controls the communication/ ccmpputttinnooottttttiiinnnn trade-off. For experiments with heterogencous model dimensions d: * dj (Section 4.3/, we ensure com- patibility and symmetry (di dzi)) by setting dij = 17 min(d, d-)]- In all our experiments, unless it is otherwise stated, the graph topology is based on the Erdös-Rényi model, where we randomly generate a network consisting of N clients with a connectivity ratio = 0.15 for rotated MNIST and heterogeneous CIFAR-10 datascts and p = 0.2 for the rest of the datascts. Both learning rates are chosen to be small to satisly the conditions in Theorem 1 Furthermore, Sheaf-FKTL uses the same global learning rate (a) as dFedU update the models for a fair comparison. The linear maps are initialized randomly from a normal distribution with mcan of 0 and a variance of 1. The values of the regularization parameter (A) used for every dataset are listed in Table D Finally, learning rates and other relevant hyperparameters were tuned for all baseline methods using grid search on validation set to ensure fair comparison.
Baselines. In the first experiment, we first compare our algorithm to the dFedU algorithm Dinh et al. 2022 as well as D-PSGD (Lian et al., 2017), and local training. Then, we provide more results by comparing Sheaf-FWTL to state-of-the-art FL algorithms (McMahan et al. 2017 Li et al., 2021aa] Fallah et al.] 2020 Dinh et al. 2020). We implement our proposed algorithm using a mini-batch stochastic gradient in 18) to make the comparison fair. For dFedU, the weights {a:j), defined in @ are taken to be a.j = 1, ((i E E. In the second one, we compare to stand-alone baseline where each client trains on each local dataset without communicating with the rest of the clients. To the best of our knowledge, Sheaf-FMTL is the only algorithm solving the FMTL problem over decentralized lopology with the clients having different model sizes, hence the comparison to a stand-alone baseline in the second experiment.
Hardware and code. Our experiments were carried out on a system equipped with an Intel(R) Xcon(R) CPU operating at 2.20GHz with 2 cores and 12 GB of RAM. The algorithms are implemented in Python using Torch (Paszke et al. 2019). and NetworkX (Hagberg et al. 2008).
11

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Summary of heterogeneous CNN architectures employed in Section 4.3 for rotated MNIST and
heterogencous CIFAR-10 datasets, simulating clients with varying computational capabilities. The total number of clients is N.
Dataset
Characteristic
Small
Medium
Large
Conv Layers
1
2
2
Conv Filters/Layer
(16) 1
(24, 48)
(32, 64)
FC Layers Hidden FC Units
2
R-MNIST
None
None
120
Approx. Params"
23 x 103
121 x 103
212 x 10
Conv Layers
1
2 (24, 48)
2
Conv Filters/Tayer
(16) 1 None
(32, 64)
FC Layers
2
H-CIFAR-10
Hidden FC Units
None
120
Approx. Params"
28 x 103
154 x 10
297 x 103
Client Allocation per Architecture
N/3
N/3
N/3
AAproximmaee parameters (weights and biases) calculated based standard inputs: R-MNIST (28 x 28 x 1), H-CIFAR-10 (32 32 3). Values rounded.
Test accumay
0.6
0.6
dFedU
Sheaf-FMTL, 0,03)
dFedU
0.4
Sheaf-FMTLL 0.01)
0.4
Sheaf (y= =
D.PSGD
Sheaf-FMTL (y 0.001
0.2
D4SGG
02
50
100
200
10
Communication Rounds
Transmited (KB)
(a)
(b)
0.7
0.6
0.6
Test accuracy rac
0.5
0.5
0.4
Sheaf-FMTL (y= 0.03)
0.4
dFedU
Sheaf-FMTL, (y- 0.01)
Meeaf-FMTLLL (y= 0.03)
D.PSGD
Sheaf-FMTL (y- 0.01)
Local
0.3
D-PSGD
25
50
75
100
125
150
10'
Communication Rounds
Transmilted Bits (KB)
(a)
(b)
Figure 2: Test accuracy as a function of the number of communication rounds and the number of transmitted bits for the rotated MNIST dataset (top), and the heterogencous CIFAR-10 datasct (bottom).
4.2 Experiment 1: same model size
Figure
2 illustrates the performance of the proposed Sheaf-FKTL algorithm and dFedU on the rotated MNIST and heterogeneous CIFAR-10 datasets, respectively, showcasing the test accuracy as a function of the number of communication rounds and the total number of transmitted bits. We consider two values for ? (0.01, 0.03) such that the projection space dimension is yd. In Figure 4a). we can see that Sheaf -PHTL achieves similar test accuracies as dFedU. For both datasets, Sheaf-FMTL (7 0.01) and dFedU
12

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Comparative analysis of storage, computational overheads, and communication costs.
Method
Storage per client
Compute per iteration per client
Communication per
iteration per client
Sheaf-FNTL
di + Σ JEN,
dig x di
O(d, x diy)
Σ dij JEN,
dFedU
di
(d:))
Σ dj JEN,
0.7
0.9
0.8
Test Accuracy
0.7
0.6
0.5
0.5
0.4
Local
0.4
Local
Sheaf-FMTL
Sheaf-FMTL
0.3
50
100
150
200
0
50
100
150
200
Communication Rounds
Commurication Rounds
(a) R-MNIST dataset
(b) H-CIFAR-10 dataset
Figure 3: Test accuracy as a function of the communication rounds for (a) R-MNIST dalaset and (b) H- CIFAR-10 dataset.
converge significantly faster than D-PSGD and Local training. On rotated MNIST, Sheaf-FMTL achieves 85% accuracy within 15 rounds, while D-PSGD requires approximately twice the number of rounds to reach comparable performance. On rotated MNIST, Sheaf-FMTL and dFedU both achieve final test accuracies above 94%, outperforming D-PSGD (~ 89%) and Local training (~ 88%). The performance gap is even more pronounced Heterogeneous CIFAR-10, where Sheaf and dFedU reach ~ 66 accuracy while D- PSGD struggles to exceed 55%. On the other hand, Figure ( shows that Sheaf Cheaf-FNTL achieves higher test accuracy with fewer transmilted bits compared to the baselines, demonstrating its ability to learn effectively while minimizing communication overhead. For the rotated MNIST dataset, using = 0.01 leads to almost similar test accuracy as dFedU, while it requires 100x loss in terms of the number of transmitted bits to achieve this accuracy. For the Helerogeneous CIFAR-10, Sheaf-FMTL requires slightly more communication rounds than dFFdU,, but the benefit in terms communication overhead evident as requires exchanging significantly fewer bits.
As illustrated in Table 3 Sheaf-FWTL incurs additional storage and computational costs due to the main- tenance and training of restriction maps. Specifically, each restriction map requires storing a matrix of size dij x di, leading to a cumulative storage requirement of O(EE) x dij x d4) across the network. Computation- ally, updating these maps involves matrix multiplications and gradient calculations, adding a complexity of Odgy x d,) per edge per iteration. However, these costs are significantly offset by the substantial commu- nication savings achieved, particularly when di chosen to be small fraction of d, making Sheaf-FMTL viable option in resource-rich FL environments such as cross-silo FL settings. Furthermore, our analysis in Table 4 shows that the actual memory footprint, while potentially higher than the simplest baselines, remains within reasonable limits compared to the basclines.
In addition to FedAvg and D-PSGD, we compare the performance of our approach to three state-of-the- art PFL algorithms: DITTO, pFedMe, and Per-FedAvg. These PFL algorithms are well-regarded in the literature for addressing heterogeneity in FL environments. To evaluate the performance of our method against these baselines, we conducted additional experiments on two datasets: Vehicle Sensor and HAR. Table measures the performance of various federated learning algorithms across four critical dimensions:
13

---

Published in Transactions on Machine Learning Research (06/2025)
0
10000
B000
12
Frobenius Norm
Frobenius Norm
8000
10
C
8000
8000
4000
4000
20
2000
30
2000
36
0
0
0
6
12
18
24
30
36
5
10
15
20
25
Client|
Client]
(a) R-MNIST dataset
H-CIFRRD- dataset
Figure 4: Heat maps of the Frobenius norms of the final restriction maps for (a) R-MNISTT dataset and (b) H-CIFAR-10 dataset.
test accuracy (model effoctiveness), transmitted bits in megabytes (commuiication efficiency), floating-poinn operations in trillion (computational complexity), and memory usage in megabytes (resource requirements) on both the Vehicle Sensor and Human Activity Recognition datasets. The test accuracy averaged over runs, reporting both mean and standard deviation, FLOPS represents the total number of floating-point operations (in trillion) required for each algorithm to converge, and the memory column shows the peak memory consumption in megabytes during the training process. The measurement captures the maximum memory footprint. For the rotated MNIST dataset, Sheaf-FMTL achieves the highest test accuracy (94.5%) regardless of the parameter 7, surpassing both decentralized algorithms like D-PSGD (92.2%) and central- ized approaches like pFedme (90.05%). Notably, Sheaf-FNTL maintains this superior performance while requiring only 38MB of communication (at 7 = 0.1), which about 100x lower than dFedU (3230.9KB) and comparable to FedAvg. On the heterogeneous CIFAR-10 dataset, Sheaf-FMTL shows competitive ac- curacy (65.8% al 7 0.3) compared to dFedU (65.7%) while requiring about 100% less communication overhead. As expected, the PFL baselines (DITTO, pFedMe, Per-FedAvg) and FedAvg achieve lower munication overheads duc to their star topology, leveraging central parameter server. The computational efficiency of Sheaf-FWTL is particularly evident when compared to personalizatiin-foousss algorithms such as pFedme, which demands approximately 24x more Moating-point operations on the heterogeneous CIFAR- 10 dataset. This substantial improvement in the conputation-commmicttooo communication trade-off can be attributed to our algorithm's ability to learn task relationships through flexible sheaf structures rather than enforcing rigid constraints across all clients. The parameter y effectively controls this trade-off, with higher values yielding marginal accuracy improvements at the cost of increased communication overhead. These results confirm that Sheaf-FMTL successfully balances personalization, communication efficiency, and computational complexity in heterogeneous federated environments.
4.3 Experiment 2: different model sizes
Figure 3 compares the performance of the proposed Sheaf-FMTL algorithm with the local training, i.e., training each model independently without communication with other clients, on the rotated MNIST and heterogencous CIFAR-10 datasets by plotting the test accuracy function of the number of communication rounds. The horizontal dashed line (Local baseline) in each subplot represents the converged performance achievable when clients train solely on their own data without any communication, serving as reference for the elficacy of collaboration. For both datasets, clients train different CNN architectures as per Table 2 Subplot (a) presents the results on the rotated MNIST dataset, where heterogeneity arises from concept shift (different rotations applied client data). Sheaf demonstrates rapid convergence, surpassing the local baseline accuracy of approximately 90% within the initial 40 communication rounds. The algorithm continues to improve, eventually plateauing significantly higher test accuracy of around 93%, showcasing
14

---

Published in Transactions on Machine Learning Research (06/2025)
Table 4: Performance Sheaf-FMTL compared to baselines for the rotated MNIST and heterogeneous
CIFAR-10 datasets.
Test Accuracy
Transmitted Bits (MB)
FLOPS
Memory
Dataset
Algorithm
(x10
(MB)
Rotated MNIST
Sheaf-FWTL (1 = 0.01)
94.3 + 0.12
38.2
240.7
7563.5
Sheaf-FWTL (7 = 0.03)
94.5 + 0.08
165.7
249.3
24022.2
dFedU
94.4 + 0.06
3230.9
1184.4
1582.9
D-PSGD
92.22 + 0.06
3230.9
236.8
1438.6
FedAvg DITTO
81.4 + 0.16
27.9
1184.1
1531
92.2 + 0.11
27.9
473.8
1714.6
pFedme
90.05 = 0.14
27.9
5923.6
1550.7
Per-FedAvg
88.45 + 0.11
27.9
202
1421
Hetcrogencous CIFAR-10
Sheaf-FWTL (r 0.01)
65.5 + 1.2
41.4
282.2
9582.3
Sheaf-FWTL (> = 0.03)
65.8 + 1.3
108.8
294.5
27452.2
dFedU
65.7 + 1.8
3937.4
1376.5
9197.6
D-PSGD
64 + 0.5
3937.4
207.5
8402.4
FedAvg
60.88 - 1.4
25.5
1040
8431.6
DITTO
63.4 1.1
25.5
413.4
8650.9
pFeedme 62.5 + 1.3
25.5
6886.2
8496.6
Per-FedAvg
61.25 = 1.4
25.5
216
3480
the substantial advantage gained through collaborative learning facilitated by the sheaf mechanism in this setting. Subplot (b) displays the performance on the heterogeneous CIFAR-10 dataset, which exhibits label distribution skew and quantity skew. Here, the task is more challenging due to the increased hetero- gencity. While convergence is slower compared to rotated MNIST, Sheaf-FKTL consistently improves over communication rounds. Tt surpasses the local baseline accuracy by approximately 62.5% around 60 rounds and reaches a final accuracy of 68%
To gain qualitative insights into the interaction structures learned by Sheaf in the presence of archi- tectural heterogeneity, Figure visualizes the Frobenius norms (i)lll) of the learned restriction maps for the rotated MNIST (left) and heterogeneous CIFAR-10 (right) datasets. These plots correspond to the per- formance results shown F where clients employed diverse CNN architectures as detailen Table 2 Each pixel (i.j) in the heatmaps represents the magnitude of the learned interaction map P.j projecting the model parameters of client i into the shared space with client Brighter colors indicate larger norm, sig- nifying stronger learned coupling or influence between the projected representations of the clients' models. The heatmaps clearly show that the algorithm learns non-trivial interaction maps (non-zero norms) between connected clients. Crucially, the norms vary significantly across different client pairs. This demonstrates that Sheaf-FWTL docs not impose uniform interaction strengths but rather learns them. While many potential interactions exist (off-diagonal entries), the number of pairs exhibiting very high norms (bright yellow spots) is relatively sparse in both datasets. This suggests that the algorithm identifies and emphasizes connections between specific client pairs deemed most beneficial for the joint optimization objective, while maintaining weaker couplings between others.
4.4
Ablation Study
4.4.1 Effect of the Regularization Strength
To investigale the impact of both network topology and regularization strength on our proposed method, we conducted experiments on the Vehicle dataset using three distinct network structures as illustrated in Figure small-world, scale-free, and complete graphs. For the small-world topology, we used the Watts-Strogatz
model with each node connected l nearest neighbors and rewiring probability p = 0.1, creating a network with high clustering and relatively short path lengths. The scale-free network generated using
15

---

Published in Transactions on Machine Learning Research (06/2025)
(a) Small world
(b) Scale free
(c) Complete
Figure 5: Network topologies used in the regularization experiments: (a) Small-world topology generated
using Walts-Strogalz model with k = 4 neighbors and rewiring probability 0.1, (b) Scale-free topology
generated using Barabási-Aie Albert model with m = 2 attachments per new node, and (c) Complete graph where every client is connected to all others.
Table 5: Performance comparison of -heaf-FMTL and dFedU across different network topologics and regu- larization strengths on the Vehicle dataset.
Sheaf-FMTL 0.1)
Regularization Strength
dFedU
Topology
Average Test Accuracy
Transmitted Bits (kB)
Average Test
Transmitted Bits (kB)
Accuracy
X = 10
92.38 +: 0.38
91.79 + 0.51
10-3
92.31 + 0.38
91.57 0.36
Small-world
(Fig. 5a
À = 10-2
92.19 + 0.40
162
91.89 + 0.44
808
À 10-1
91.63 + 0.47
90.65 = 0.58
À = 1
89.80 + 0.73
84.64 1.72
A 10
84.15 + 1.20
80.88 + 0.91
10-4
92.42 + 0.40
91.58 0.43
X = 10-3
91.63 + 0.48
91.28 + 0.72
Scale-free
(Fig. 5b
À 10-2
92.45 +: 0.24
485
91.12 0.20
2424
= 10-1
90.19 + 0.55
90.54 0.50
À 1
89.08 + 0.75
85.71 + 0.30
10
80.55 + 1.34
80.41 0.35
X = 10-4
92.06 +: 0.67
88.98 + 0.46
10-3
91.06 + 0.26
89.60 0.61
Complete
(Fig. 5c
À = 10-2
89.62 + 0.52
711
89.40 + 0.63
3555
À 10-1
89.15 + 0.56
89.07 0.34
À 1
84.5 = 0.74
83.67 0.63
À 10
78.2 + 0.45
80.40 + 0.35
the Barabási-Alleet preferential attachment model with m 2 new odges per node, resulting in a power-law degree distribution where some nodes as hubs with many connections. The regularization parameter X varied across six orders of magnitude from 10-4 to 10 to observe its effect model performance.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Table 6: Final average test accuracy of Sheaf-FMTL for different initialization strategies the restriction
maps P.j under heterogencous CNN architectures (Table 2
Initialization Method
Paramcter
R-MNIST
H-CIFAR-10
0.01
0.9280
0.6767
Gaussian o2)
o 0.1
0.9279
0.6722
0 = 1.0
0.9277
0.6815
a 0.01
0.9288
0.6842
Uniform u((-a,a))
a 0.1
0.9268
0.6759
1
0.9277
0.6733
Orthogonal
N/A
0.9273
0.6746
Identity + Noise
noise 0.01
0.9268
0.6624
Table presents comprehensive comparison between Sheaf-FNTL and dFedU across these different con- figurations. Several key obscrvations emerge from these results. First, Sheaf-FMTL consistently achieves superior communication efficiency, requiring approximately 5x fewer transmitted bits than dFedU across all topologies. Second, regarding accuracy performance, Sheaf-FKTL generally outperforms dFedU across most regularization values, with the performance gap becoming more pronounced at higher regularization strengths (A 2 1). For instance, with À in the small-world topology, Sheaf-FWTL achieves 89. 89.80% accuracy compared dFedU's 84.64%. Among the three topologies, the small-world network provides the best balance between communication efficiency and model performance, requiring the least communication while maintaining comparable better accuracy than more densely connected topologies.
While our experiments on the Vehicle dataset indicated optimal performance for À in the range [10-1, 10-2) across different topologics, this should not be interpreted as a universal recommendation. The optimal choice of is highly dependent on factors such as the degree and lype of data heterogeneily, nelwork topology, and the data and model size. In practice, À should be treated as hyperparameter tuned using a validation set (typically a held-out portion of each client's 's local data). We recommend scarching logarithmic scale (e.g., [10-5, 10]), noting that excessively large values () Z 1) can overly constrain personalization and harm. performance.
4.4.2 Effect of the Restriction Maps Initialization
We conduct an ablation study to assess the sensitivity of the Sheaf-FMTL algorithm to the initialization strategy of the restriction maps P.j, particularly within the challenging context of heterogeneous model architectures across clients. Using the CNN architectures specified in Table 2 for rotated MNIST and heterogeneous CIFAR-10 datasels, we evaluated several initialization methods
Gaussian Initialization: Each element of Pij drawn independently from normal distribution o3),,²), with variances [0.01.0.11 0.
Uniform Initialization: Each element was drawn independently from uniform distribution u([-a,a), with ranges corresponding to E 0.1,
Orthogonal Initialization: Ps was initialized as a matrix with orthonormal rows, ensuring P.j -ay
Identity + Noise Initialization: Pú was initialized based on the identity matrix (selecting the first dy rows of the d, x d, identity, padded with zeros needed), with small Gaussian noise 0.01)))))) added break perfect symmetry and allow adaptation.
The results, presented in Table measure the final average test accuracy achieved after convergence.
17

---

Published in Transactions on Machine Learning Research (06/2025)
On the rotated MNIST dataset, the performance of Sheaf-FMTL exhibits remarkable robustness to the choice of initialization. Across all tested strategies, the final accuracy remains tightly clustered between approximately 92.68% and 92.88%. The best performance was marginally achieved with narrow Uniform initialization (0.01, (.000))) Still, the minimal variation suggests that the algorithm effectively converges to high-quality solution regardless of the starting point for P.g in this setting.
For the heterogeneous CIFAR-10 dataset, which presents greater data and architectural heterogeneity, we observe slightly more variation, though the overall stability is still noteworthy. The final accuracy ranges from 66. 24% to 68.42%. Similar to rotated MNIST, the narrow Uniform initialization (4([-0.01, 0.01))) yielded the highest accuracy. Notably, the (Identity + Noise) initialization resulted in the lowest accuracy (66.24%), suggesting that starting with projections aligned to canonical parameter subsets might be less effective than random or orthogonal initializations when dealing with complex models and diverse data distributions inherent the heterogeneous CIFAR-10.
5 Conclusion
In this work, we introduced novel sheaf-based framework for FMTL that effectively tackles challenges arising from data and sample heterogeneity across clients. By leveraging cellular sheaves, our approach can flexibly model complex interactions between client models, even with varying feature spaces and model sizes. While Sheaf-FWTL introduces additional memory and computational overhead for managing the sheaf structure, this often offset by significant. communication savings, the ability handle heterogeneous model architectures and providing unified view subsuming various existing FL methods, making it a compelling approach, particularly in cross-silo FL settings. Theoretically, Sheaf-FWTL, establishing a sublinear convergence rale in line with state-of-the-arl ralized FMTL, algorithms. Empirically, extensive experiments on benchmark datasets demonstrated the communication savings of Sheaf-FNTL compared to the state-of-the-arl baselines.
analyzed the convergence propertics of
Acknowledgments
This work was supported by the ERA-NET CHIST-ERA Project MUSF-COM
References
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge Luis Reyes-Ortize et al. A public domain dataset for human activity recognition using smartphones. In Esann, volume 3, pp. 3, 2013.
Rodolfo Stoffel Antunes, Cristiano André da Costa, Arne Küderle, Imrana Abdullahi Yari, and Björn Es- kofier. Federated learning for healthcare: Systematic review and architecture proposal. ACM Transactions on Intelligent Systems and Technology (TIST), 13(4):1-23, 2022.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. Federated learning with personalization layers. arXiv preprint crXto:1912.0083 00818, 2019.
Fei Chen, Mi Luo, Zhenhua Dong, Zhenguo Li, and Xiuqiang He. Federated meta- learning with fast conver- gence and efficient communication. arXiv preprint
2018.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized lederated learning. In International conference on machine learning, pp. 2089 2099. PMLR, 2021.
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning. arXiv preprint ar xrxx:20033. XxXx:2003.13461, 2020a.
Yuyang Deng, Mohammad Mahdi Kamani, and Mchrdad Mahdavi. Distributionally robust federated aver- aging. Advances in neural information processing systems, 33:15111-15122, 2020b.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Yuyang Deng, Mohammad Mahdi Kamani, Pouria Mahdavinia, and Mehrdad Mahdavi. Distributed person- alized empirical risk minimization. Aduances in Newral Information Processing Systems, 36, 2023.
Canh T Dinh, Tung T Vu, Nguyen H Tran, Minh N Dao, and Hongyu Zhang. A new look and convergence rate of federated multitask learning with laplacian regularization. IEEE Transactions on Newral Networks and Learning Systems, 2022.
Moming Duan, Duo Läu, Xianzhang Chen, Yujuan Tan, Jinting Ren, Tei Qiao, and Liang Liang. Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications. In 2019 IEEE 37th international conference on computer design (ICCD), pp. 246 254. IEEE, 2019.
Marco F Duarte and Yu Hen Hu. Vehicle classification distributed sensor networks. Journal of Parallel and Distributed Computing, 64(7):826-88) 888,, 2004.
Anis Elgabli, Chaouki Ben Issaid, Amrit Singh Bedi, Ketan Rajawat, Mchdi Bennis, and Vancet Aggarwal. FedNew: A communication--ffiiene and privacy-preserving Newton-type method for federated learning, Tn Proceedings of the 39th International Conference Machine Learning, volume 162, pp- 5861-5877, 2022.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Sys- tems, 33:3557-3556 2020.
Harvey Goldstein. Multilevel modelling of survey data. Journal of the Royal Statistical Society. Series D (The Slatisliciam), 40(2):235 244, 1991.
Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL). Los Alamos, NM (United States), 2008,
Jakob Hanson and Robert Ghrist. Distributed optimization with sheaf homological constraints. In 2019 57th Annual Allerton Conference on Communiication, Control, and Compuling (Allerton), pp. 565 571. IEEE, 2019.
Filip Hanzely and Peter Richtárik. Fedcrated learning of a mixture of global and local models. arXiv preprint wrXivv2002.05516 2020.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. Advances in Newral Information Processing Systems, 33:2304-29 2315, 2020.
Kevin Hsich, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of de- centralized machine learning. In International Conference on Machine Learning, pp. 4387-4398. PMLR, 2020.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Mcasuring the effects of non-identical data distribution for federated visual classification. Xrxiv preprinl arxiv: 1909. 00335, 2019.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangehuan Liu, Jian Pei, and Yong Zhang. Per- sonalized cross-silo federated learning on non-iid data. In Proceedings of the AAAT conference on artificial intelligence, volume 35, pp. 7865 7873, 2021.
Yihan Jiang, Jakub Konečný, Keith Rush, and Sreeram Kannan. Improving federated learning personaliza- tion via model agnostic meta learning. arXiv preprint arXiv: 1909.
1909.194888.. 12488, 2019.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mchdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trendsl in machine learning, 14(1-2):1-210 2021.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Sai Pranceth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp. 5132 5143. PMLR, 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, el al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEEE 38th international conference on data engineering (ICDE), pp. 965- 978. IEFE, 2022.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet. Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429 450, 2020a.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357 6368. PMLR, 2021a.
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020b.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. FedBN: Federated learning on
non-lID features via local batch normalization. In International Conference on Learning Representations, 2021b. URL https://openreview..tt
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy Auerbach, David Brent, Ruslan Salakhut- dinov, and Louis-Philipoe Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXin:2001.0 01 529, 2020.
Tao Lin, Sebastian U. Stich, Kumar Kshit Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. In International Conference on Learning Representations, 2020.
Ken Liu, Shengyuan Hu, Steven Wu, and Virginia Smith. On privacy and personalization in cross-silo lederated learning. Advances in neural information processing syslems, 35:5925 5940, 2022a.
Ruixuan Liu, Fangzhao Wu, Chuhan Wa, Yanlin Wang, Lingjuan Lyu, Hong Chen, and Xing Xie. No left behind: Inclusive federated learning over heterogencous devices. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3398 3406, 2022b.
Mi Luo, Fei Chen, Dapeng HU, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972 5984, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commaiication-cffeient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273 11822 PMLR, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
J. Cuiddero-Canddla, M. Sugiyama, A. Schwaighofer, and N. D Lawrence. Dataset shift in machine learning. The MIT Press, 2009.
20

---

Published in Transactions on Machine Learning Research (06/2025)
Shah Atiqur Rahman, Christopher Merck, Yuxiao Huang, and Samantha Kleinberg. Unintrusive eating recognition using google glass. In 2015 9th International Conference on Peruasive Compuling Technologies for Healthcare (PerrasiiveHealth)) pp. 108 111. IEEE, 2015.
Hans Riess and Robert Ghrist. Diffusion of information networked lattices by gossip. Tn 2022 IEEE 61st Conference Decision and Control (CDC), pp. 5946-5952. IEEE, 2022.
Michael Robinson. Understanding networks and their behaviors using sheaf theory. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 911-914. IEEE, 2013.
Michael Robinson. Topological signal processing. 81, 2014.
Michael Robinson. Sheaves are the canonical data structure for sensor integration. Information Fusion, 36: 208-224, 2017.
Yasmin SarcheshmehPour, Yu Tian, Linli Zhang, and Alexander Jung. Clustered federated learning via generalized lotal variation minimization. IEEE Transactions on Signal Processing, 71:4240 144200-4256, 2023.
Felix Sattler, Klaus-Rober Müller, and Wojeiech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural nebworks and learning systems, 32(8))3710 3722, 2020.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. Advances in Newral Information Processing Systems, 33:21394-21405, 2020.
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chongqi Zhang. Fedproto: Federated protolype learning across heterogeneous clients. In Proceedingss of the AAAI conference on artificial intelligence, volume 36, pp. 8432 8440, 2022.
Hao Wang, Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning non-iid data with reinforcement learning. In JEEE INFOCOM 2020-IEEE Conference on computer communications, pp. 1698 1707. IEEE, 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 7611-7623 2020b.
Kangkang Wang, Rajiv Mathews, Chloé Kiddon, Hubert Eichner, Françoise Beaufays, and Daniel Ramage. Federated evaluation of on-device personalization. arXio preprint arXix: 10252, 2019.
Haishan Ye, Ziang Zhou, Luo Luo, and Tong Zhang. Decentralized accelerated proximal gradient descent.. Advances in Neural Information Processing Systems, 33:18308 18317, 2020.
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. Salvaging federated learning by local adaptation.
arXiv preprint arXiv:2002.0473 04000, 2020.
Xinwei Zhang, Wotao Yin, Mingyi Hong, and Tianyi Chen. Hybrid federated learning for feature & sample heterogencity: Algorithms and implementation. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL htttps://ppereviiiee wwtt://orumm/iddeeccllllle
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXio preprint orX:o:1806. 00582, 2018.
Jiayu Zhou, Jianhui Chen, and Jieping Ye. Malsar: Multi-task learning via structural regularization. Arizona State University, 21:1 50, 2011.
21

---

Published in Transactions on Machine Learning Research (06/2025)
A Sheaf-Theorett Approach in FMTL
A.1 Rationale for Adopting Sheaf Theory in FMTL
Sheaf theory provides a powerful mathematical framework for modelling and analyzing complex relationships in FMTL. The adoption of this approach in our context is motivated by several key advantages
Heterogeneity modeling. FMTL often involves clients with different data distributions, model architectures, task objectives. Sheal theory allows us to capture these heterogeneous relationships in structured and mathematically rigorous manner.
2. Local-Global consistency. Sheaves provide natural way to ensure consistency between local (client-specific) and global (network-wide) information. This is crucial in FMTL scenarios where we aim to leverage network information to improve local performance.
Flexible representation. The sheaf structure allows for representing varying degrees of similarity or difference between clients. This nuanced representation is more sophisticated than traditional approaches that often assunne uniform relationships across the network.
A.2 The Interaction Space and Client Relationships
The interaction space, denoted F(c), plays contral role in sheaf-theoretic approach to FMTL. It serves shared space where local models 0, and 0 are projected using restriction maps P. and Pjá respectively. The projection into this interaction space provides measure of client relationships for the following reasons
1. Common feature capture. The interaction space captures the common or comparable features between clients, analogous to how principal component analysis (PCA) captures the most important features of a dataset.
Consistency enforcement. Our approach enforces consistency between the projections of local models onto the interaction space. This is mathematically represented by the sheaf Laplacian term O² which penalizes discrepancies between the projections of local models.
3. Collaboration encouragement. By minimizing the sheaf Laplacian term, local models are en- couraged to collaborate effectively, leveraging shared information to improve overall performance.
A.3 Restriction Maps and Their Interpretations
The restriction maps, represented by matrices Pij, are fundamental to our sheaf-theoretic approach. These maps project local models O, onto the interaction space F(e). The intuition behind these maps can understood as follows
Feature selection. P. acts as a feature selection matrix, identifying common or comparable features between clients and j.
2. Information sharing. The restriction maps facilitate information sharing between clients by pro- jecting local models onto common space, enabling effective collaboration even when local models have different dimensions or feature sets.
Model comparison. In heterogencous settings where clients have different model sizes, traditional FL methods relying on direct model aggregation or comparison fail. The restriction maps allow for meaningful comparisons by projecting onto a common space.
22

---

Published in Transactions on Machine Learning Research (06/2025)
A.4 Dimensional Considerations and Trade-offs
The dimensions of the restriction map Ps are determined by the dimensions of the local model 0 and the interaction space F(c). If 0, E IR and the interaction space has dimension daj, then Pij is of size dı d₃. The choice of dij involves trade-off
Smaller dy: Results in more compact representation of shared information, leading to communi- cation savings.
Larger dy: Allows for more flexibility in capturing relationships between local models but increases communication costs.
In practice, the choice of di can be guided by factors such as the estimated overlap in feature spaces between clients, computational resources available, and the desired balance between model expressiveness and communication efficiency.
A.5 Comparative Advantages over Traditional FMTL Methods
Our sheaf-theoretic approach offers several advantages over traditional FMTL methods
1. Heterogeneity handling. Unlike many traditional FMTL methods that assume homogeneous models across clients, our approach naturally accommodates heterogeneous model architectures and task objectives.
Nuanced relationships. Traditional methods often assume uniform relationships between clients. Our approach allows for more nuanced modelling of inter-client. relationships through the interaction space and restriction maps.
B Interpretation of di and Pij
In this appendix, we provide an interpretation of the restriction maps P.j Fige for the case when the local models are given by linear logistic regression. We describe how to choose dij and P.j in meaningful way and the constraints that can be imposed them.
Consider a network of N clients, where each client i has local model parameterized by 0 E R24, In the case of linear regression, the local model of client is given by f((0;; = 0 a,, where æ, E Rd: is the input feature vector. For logistic regression, the local model is given by fa(O,; x.) = (0. x;), where a(.) is the sigmoid function.
Interpretation of P.y. Consider two clients i, V such that (i.j) E. The restriction maps Psj and Pj aim to capture the relationships between the local models 0, and 0, by projecting them to common interaction space F(e). In the context of linear or logistic regression, P.j and P3s can be interpreted feature selection matrices that identify the common or comparable features between the two clients.
Let Paj E [ddjjdd and Pj: E Rdijxd; be the restriction maps for clients and j, respectively, where dj = dim F(e) is the dimension the interaction space. The restriction maps should satisly the following condition
P.o,, R2 P,OO,
(21)
which ensures that the projected models in the interaction space are comparable. To impose the condition in 21 we consider the following regularizer term
Qi P.jO. PP,00-
(22)
23

---

D Supporting lemmas Published in Transactions on Machine Learning Research (06/2025)
which we aim to minimize. By adding the regularizer terms for all neighboring clients and then summing over all clients, we obtain the overall regularizer
Q EE =ljjEM
I.,0. P,,0,1²;
(23)
which is exactly the sheaf quadratic form for the choice Fige P.j for e E
Choice of diy. The dimension of the interaction space, dij, determines the size of the restriction maps Piy and Pji. In practice, dej can be chosen based on the number of common or comparable features between clients and j. A smaller value of dy implics a more compact representation of the shared information between the two clients, while a larger value allows for more flexibility capluring the relationships between the local models.
C Real-World Applications of Sheaf-FMTL
In this appendix, we explore practical scenarios where task similarities are inherently defined within vector spaces, making them well-suited for the application of Sheaf-FNTL. By examining representation learning and feature vector similarities in multi-modal tasks, we illustrate how our sheaf-thcoretic framework effectively captures and leverages complex task relationships. These examples showcase the applicability of Sheaf-FMTL. in diverse FL environments.
C.1 Representation Learning and Embedding Spaces
In many ML applications, tasks are associated with high-dimensional data that can be effectively represented through embeddings in vector spaces. These embeddings capture semantic, syntactic, or feature-based relationships between tasks, facilitating the modeling of task similarities as vector operations. For example, if we consider the Natural Language Processing (NLP) field, then tasks such as sentiment analysis, topic classification, and namod entity recognition can be embodded in a semantic space using techniques like Word2Vec or BERT. The proximity of these task vectors in the embedding space reflects their semantic relatedness. Similar NLP tasks often share underlying linguistic structures. By representing these tasks vectors, Sheaf-FWTL can capture and leverage their shared characteristics to enhance collaborative learning,
C.2 Feature Vector Similarities in Multi-Modal Tasks
In multi-modal learning scenarios, tasks often involve integrating and processing data from different modal- ities (e.g., text, image, audio). Task similarities can be defined based on the feature vectors extracted from these modalities, enabling Sheaf-FMTL to model interactions across diverse data sources. For example, if consider multi-modal sentiment analysis, then tasks that analyze sentiment from text, images, and audio can have their respective feature vectors embedded in a unified vector space. The similarities between these feature vectors can indicate shared sentiment characteristics across modalities. Sheaf-FKN can utilize these vector similarities to facilitate collaborative learning, enhancing sentiment detection accuracy by leveraging cross-modal information. Another example is healthcare applications, where tasks involving the analysis of patient data. from various sources (e.g., medical imaging, electronic health records, genomic data) can define task similarities based the integrated feature vectors representing different data modalities. By modeling these similarities in vector space, Sheaf-FMTL can improve personalized treatment recommendations through effective knowledge sharing across related healthcare tasks.
Lemma 3.
0TH O
Σ c=(i,))88
| (0.) - Fzge (0,)
Σ (=()))E
P.:0: P3:0,112.
(24)
24

---

Published in Transactions on Machine Learning Research (06/2025)
Using the block matrix structure of Ly from equation 3 follows
Proof.
expand the quadratic form
0- 1,x 0
ΣΣ
0 0,
JEV
Σ of
P.P.P. 02- Σ 0. P,,O, -I(E))
P.J.P.OOO
iEV
JEN;
E
(O P.T.P.P P. +
P000 P, -0 p P,.0,)
(JIEE
E
P.,0,11 + P. - 2 (P..O..
EE(J)EE
Σ
P.,,0 P
(25)
(=()))E
Recalling that Fige and Pi are used interchangeably to denote the restriction map from vertex i to edge e == (i,5), we obtain
L=
Σ P.,0, - P,,0,1²=
E
FFae (0,) - Fjge (0,))l
(26)
(=(i,))E
(=())))
Lemma 4.
ker(L=) arg min Q= arg min 07 L 0. OECO(T) (EC*(F)
(27)
Proof. Let f kor(Lz). Then, L = 0. By the definition of Q+(0), we have
Qx(0) 0T L=0 0.
Therefore, 0 E arg mrg)) (=(
Conversoly, let 0 E arg (p (0). Then, Q5(0) = 0. which implics
0 0= Q)(
Σ
Fige (0,) Fyga (0,)11
(28)
EF(j))EE
Thus, we get
Fize (0.) = Ffge (0,) Ve= (i.j) EE
(29)
This concludes the proof.
E Proof of Theorem
We start by introducing the matrices Júj Rdixxd, having all its entries equal to one. Then, let us define the block matrix H such that H.j Jıj if (i,j) E E, and Hig = 0, otherwise. Then, 0 and P can be updated using the following updates
g = gh ar (V + X(P
(30)
ph+1 H o (p*
(31)
25

---

Published in Transactions on Machine Learning Research (06/2025)
where @ is the Hadamard product and the matrix H is introduced to preserve the block structure of P by zeroing out the entries that do not correspond to edges in the graph.
Next, we analyze the convergence of the Sheaf-FMTL. algorithm by studying the descent steps in 0 and P separately. This approach allows us to establish bounds on the decrease of the objective function (e P) in each descent step.
Theorem 2. Let Assumptions 1 and 2 hold. Assume the learning rales and n salisfy the conditions
2 < NL and 7) xnz respectively. Then, the averaged gradient norm is upper bounded follows
2.
K-
1 (V p+
pk (V( PO) v(*),
(32)
k=0
where p min {a (1 x²4 .7 (1 =ADi
} and u is the optimal value of M.
Proof. Using the Lipschitz continuity of the gradient of S(0)
NL | -
f(0h+1) S ff(0))++ (Vf(0*), gh+l 8h)
(33)
Adding and subtracting ( to both sides of the inequality, we get
( (p)
= s((tt) X ( P**
$ f(0)) + (Vf(0"), gh+| 0%) +
NL
1 1 \
g ( T
-(00) + (V/(0') 0h+1 0h) + NL 1 0h112+ + A o p
2
A 0 ( p
(PA) ph
NL
gh++1 -0 + \ -0')T pp'"" ght
= =((0") + (Vi(0"). gh+1 - +
2
2
ト/2 (p
= - p (O* 0 -0h) + NI. 2
1 141
(34)
where the last equality follows from the definition (e, P). Using the update rule for g we have
(o P*), o++1 0%)
<VV(()*) + A(P*) +XP") =-a( A(Pb)T p "
+
= = pT2
(35)
On the other hand, we have
100 - 0+
= aa V + X(P
=V p+
(36)
Substituting
35
and
36
back into 34
obtain
V/ @ p*) (( pk) (1 aNL
VoV (2)))
12
(37)
26

---

Published in Transactions on Machine Learning Research (06/2025)
By choosing NL. we ensure that the term 1 - a&L is positive.
From the definition of (O, P), we have
= f(0ht! X ( (ptti)"phtlghht ( -
(38)
Using the update rule for petl i.e. pett H (P* (APiget we can write
1pk-1sTphtt
-H o (pk q/Pp (/ o (P* mpPp g (
3 <Pp (
(39)
where the inequality follows from the fact that the Hadamard product with H zeros out some entries, which can only decrease the Frobenius norm.
Expanding the right-hand side, get
(P* - nXP' k+1 (0 " (p
= -( p p +1222(Ph)3 pht (
(40)
Substituting this back into the expression for (0t++ pk+1), we obtain
vi0ht1 pk+1)
< +(( + 10
2
11
php (
(41)
2
Since the gradient of V with respect to P is given by VpV(O, P) APOO" we can compute the following norm
/VP((O 1P)
(0 (1Pi)117(()) pt = -XP(O++IT(PP)) ( (0
(42)
where we have used the cyclic nature of the trace operator Tr(-). Therefore, we have the following bound
v0t+1 1p++1)
((t+t pt) -n 1 (10 |/ (/ , 12
1 pb) 17 ( 1- n>D3
/| pp)11
(43)
2
where we have used Assumption
Hence, ensure (1
) is positive, we choose the value of r to be 17 R... Combining the inequalities
2
37
and
43
get
v( p
< (000 ph) -a -1())(1 )_.NN.
( 2 7(1 nADS IV V (44)
27

---

Published in Transactions on Machine Learning Research (06/2025)
Summing up these inequalities from k 0 to K 1, obtain
vOOKK pk)
vi(00, po) -a (1 aNL
K-1 ( -1/ (1 NADz )
K-1
V VP pl0)13
(45)
2
Let y be the optimal value of v, and we define p - min {a (1 a 11] (1- =ADi }} Then, rearranging the terms, we can write
1 (V((0), pd) TOOK pk() S pk
1
VV ph1112 - pk
(V(00) (P) - vV"),
(46)
K
K=
where we have used that 1r* S (Ok pk) and | |² V |1 (0 pa) J
F Additional Details on Datasets
A summary of the datasets and the tasks used in Section 1.2 is presented in Table 7 These datasets are real-world datasets created in federated environments with varying degrees of heterogeneity. A detailed description of the datasets, along with their specific data partitioning schemes, is provided in Table S To further quantify the non-IIDness in our data partitions, we have incorporated quantitative metrics assossing the degree of non-IIDness across different datasets in Table g
Rotated MNIST (R-MNIST). Following similar techniques as outlined in (Liu et al. 2022a). we shuffle and then evenly separate the original MNIST dataset between 40 clients. Next, we randomly divide the clients into four groups, each containing 10 clients. We then apply rotations of {0p 90" 180", 270"] to each group respectively. Therefore, clients within the same group share identical image rotations, resulting in the formation of four distinct clusters. The MNIST dataset is available under the CC BY-SA 3.0 license.
Heterogeneous CIFAR-10 (H-CIFAR-10). The original CIEAR-10 dataset split among 30 clients, and heterogeneity is introduced by assigning each client random number of samples from 5 randomly selected classes out of the 10 available classes, following a similar approach as in (T Dinh et al.
2020 Liu et al. 2022a The CIFAR-10 dataset is available under the MIT license.
Human Activity Recognition. The dataset is composed of data gathered from the accelerometers and gyroscopes of smartphones used by 30 individuals, each performing one of six activities: walking, walking upstairs, walking downstairs, sitting, standing, or lying down. In this dataset, the data from each individual/clien is treated as unique task, with the primary objective being to differentiate between these activities. To identify each activity appropriately, feature vectors with 561 elements representing various time and frequency domain variables are used in the analysis. The dataset (Anguila et al. 2013) is licensed under a Creative Commons Attribution 4.0 International (CC-BY 4.0) license.
Vehicle Sensor. The dataset involves collecting data from network of 23 wireless sensors, includ- ing acoustic (microphones), seismic (geophones), and infrared (polarized IR sensors), strategically placod along a specific road segment. This dataset aims to facilitate binary classification to identify two types of vehicles: the Assault Amphibian Vehicle (AAV) and the Dragon Wagon (DW). Each sensor, treated as a unique task or client, gathers acoustic and seismic data encapsulated in a 100- dimensional feature vector, representing the recordings as vehicles pass by. The Vehicle dataset was originally made public by its authors as rescarch datasct (Duarte & Hu 2004
Google Glass Eating and Motion (GLEAM). The datasct is collected using Google Glass from 38 individuals. It captures high-resolution sensor data identify specific activities such as eating. This extensive dataset, consisting of 27,800 entries, each with 180-dimensional feature
28

---

Published in Transactions on Machine Learning Research (06/2025)
Table 8: Data partitioning strategies across datasets where ne is the number of training samples client
Dataset
Data split
Domain
Label
distribution
distribution
R-MNIST
min nı 1500
4 groups with distinct rotation angles
Uniform distribution
max ni = 1500
within each rotation group
H-CIFAR-10
min Rg
1515
Not explicitly
Each client has data from 5 random classes
max 1839
divided into domains
HAR
min ny 210
All activity classes per client
Uniform across activity classes within each client
max ni 306
Vehicle Sensor
min Ti 872
Same label set with
Balanced across vehicle classes per sensor
max T2 = 1933
different feature distributions
GLEAM
min n2 699
All activity classes with uniform distribution
Balanced between eating and non-eating classes
max T& 776
School
min Vi = 15
Shared regression task with uniform feature sets
Continuous largets with varying distributions per school
max n4 175
vector, records head movements for binary classification to determine if the wearer is eating or not. The data includes accelerometer, gyroscope, and magnctometer readings, analyzed for statistical, spectral, and lemporal characteristics to distinguish eating from other activities like walking, talking, and drinking. The GLEAM dataset, released by its original authors (Rahman et al. 2015). is available for non-commercial use.
sceks to forecast the exam results of 15,362 students from 139 secondary schools. The dalaset contains information for each school, with student numbers ranging from 22 to 251, and each student is described using a 28-dimensional feature vector. This vector contains information about the school's ranking, the student's birth year, and the availability of free meals at the school. The datasct has been made publicly available in (Zhou et al. 2011
School. The dataset, originally introduced in (Goldstein 1991
Table 7: Summary of the datasets and tasks used in our empirical setup.
Dataset
Task
# Phients/Taskee
Input Dimension
Model
R-MNIST
Classification
40
28 28 x
convolutional neural network
H-CIFAR-10
Classification
30
32 32 x 3
convolutional neural network
HAR.
Classification
30
561
multinomial logistic regression
Vehicle Sensor
Classification
23
100
multinomial logistic regression
GLEAM
Classification
38
180
multinomial logistic regression
School
Regression
139
28
linear regression
G Supplementary Experimental Results
Figure 0 compares the performance of our proposed Sheaf-FMTL method with dFedU on four datasets: (a) the Vehicle dataset, (b) the School dataset, (c) the HAR dataset and (d) the GLEAM dataset using 7 = 0.31. The experiments evaluate the performance of the model terms of communication rounds and transmitted bits. Sheaf-FMTL and dFedU consistently achieves the highest accuracy and fastest con- vergence, demonstrating an average improvement of 5-10 percentage points D-PSGD and local across
29

---

Published in Transactions on Machine Learning Research (06/2025)
Table 9: Degree of non-lIDness across datasets.
Dataset
Non-IID Metric
Description
Heterogeneity Type
R-MNIST
Rotation angle
High domain heterogeneity with 4 distinct rotation groups
Feature distribution skew
variance
H-CIFAR-10
Label distribution
High label distribution skew among clients
Label distribution skew
HAR
Inter-client variability
High heterogeneity due to unique individual data per client
Concept shift
Vehicle Sensor
Feature distribution
High feature heterogeneity across different sensors
Feature distribution skew
GLEAM
Label distribution
Low heterogeneity with balanced labels
Minimal (almost IID)
School
Continuous targets variance
Moderate to high heterogeneity based on inter-school performance variability
Concept shift and quanlity skew
datasets. For instance, the Vehicle dataset, Sheaf-FMTL reaches 88% accuracy compared to 82% for D-PSGD and 82% for local. While Sheaf-FMTL requires more communication rounds for the Vehicle dataset to achieve similar test accuracy compared to dFFdUU, the performance gap between the two methods narrows as the number of communication rounds increases. However, when examining the number of transmitted bits, Sheaf-FWTL demonstrates a clear advantage, requiring fewer bits reach higher accuracy levels. For example, Sheaf-FWTL with 7 0.1 reaches a test accuracy of 0.85 with just 72 transmitted Kbits, while dFedU requires over 450 Kbits approach this accuracy. Similar trends are observed for the other datasets. Sheaf-FWTL recovers the same performance dFedU in terms of test accuracy with slight drop in test accuracy for the HAR and School datasets. Hence, the sheaf-basee approach effectively captures the hetero- gencous relationships among clients, achieving the same test accuracy using fewer transmitted bits than the baseline d.edU.
30

---

Published in Transactions on Machine Learning Research (06/2025)
0.9
0.8
1e:tt Accuracy
0.7
Sheaf-FMTL (V 0.3)
dFedU
Sheaf-FMTL (Y 0.1)
Sheaf-FMTL [y 0.3)
0.6
D-PSGD
Sheaf-FMTL (y= 0.1)
0.5
Local
D-PSGD
50
100
150
200
250
500
750
1000
1250
1500
Communication Rounds
Transmitted Bits (KB)
(a)
(b)
Test Ansuacy
Sheaf-FMTL (y
dFedU
Sheaf-FMTL 0.1)
Sheaf-FMTL (Y 0.3)
D-PSGD
Sheaf-FMTTL (y 0.1)
Local
0.2
D-PSGD
50
100
150
200
0
10000
20030
30000
40000
50000
Communication Rounds
Transmilted Bits (KB)
(a)
0.4
dFedU
dFedU
Sheaf-FMTL 0.3)
Sheaf-FMTL 0
Sheaf-FMTL (y 0.1)
Sheaf-FMTL (y 0.1)
Teat MSE
0.3
P-DSGD
PD
0.2
0.1
20
40
60
80
100
200
400
800
800
1000
Communication Rounds
Transmitted (kB)
0.8
Test Anturacy
dFedJ
0.6
Sheaf-FMTTL (y= 0.3)
dFedU
Sheaf-FMTL (y 0.1)
0.4
Shesf-FMTL (y
D-PSGD
Sheat-FMTTL (
Local
D-PSGD
40
1500
2000
Communication Rounds
Transmitted (KB)
(8)
(b)
Figure 6: Test/ accuracy function of the number of communication rounds and the number of transmitted bits for the Vehicle dataset (first row), the HAR dataset (second row), the School dataset (third row), and the GLEAM dataset (bottom).
31


Reviews:
[{"review_id": "BEUjDKHFLa", "reviewer": "Reviewer_UVHQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n* The proposed method and its motivation sound reasonable.\n* This paper provided comprehensive experimental results and demonstrated the effectiveness of the proposed method.\n* This paper analyzed the convergence rate of the proposed method and showed that the propose method converges to the stationary point.\n\n## Weakness\n* The reviewer is wondering if the optimal $P_{ij}$ in Eq. (17) is trivial and zero for all $i$ and $j$ since we have\n\n\\begin{align}\n\\sum_{i} f_i (\\theta_i ) \\leq \\sum_{i} f_i (\\theta_i )  + \\frac{\\lambda}{2} \\sum_i \\sum_j \\| P_{ij} \\theta_i - P_{ji} \\theta_j \\|^2\n\\end{align}\n\nfor any $P_{ij}$.\nThe motivation to add this regularization term is clear, and the reviewer agrees with the authors that it can help the node find nodes that have similar datasets. However, from the optimization perspective, $P_{ij}$ in Eq. (17) seems to have the trivial optimal solution. The reviewer would be happy if the authors clarified this point.\n\n* In Algorithm 1, when $P_{ij}^0 = 0$ for all $i$ and $j$, $P_{ij}^{k}$ seems to be always zero for all $k$. Due to this reason, the reviewer is wondering if the initial $P_{ij}$ is very important to enjoy the benefit of the proposed method. Could the authors kindly provide more explanation about the effect of the initial parameter?\n* Related to the above comments, it would strengthen the paper to visualize $P_{ij}$ obtained in the experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper extends the framework of federated multi-task learning (FMTL) and proposed Sehaf-FMTL.\nShaef-FMTL contains FMTL, FL, personalized FL, and hybrid FL as special instances.\nThen, this paper proposes the optimization method described in Algorithm 1 to solve Sheaf-FMTL, showing that Algorithm 1 converges to the stationary point.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the above weakness section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RLcUpJrcUi", "reviewer": "Reviewer_mUSs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel sheaf-based framework for federated multi-task learning and tackles an important problem of data heterogeneity.\n2. The proposed method is communication efficient as the model parameters are projected into a low dimensional space before communicating them to the neighbors.\n3. The paper presents convergence analysis for an L-smooth loss function (average squared norm of the objective function).\n4. Experimental results show that Sheaf-FMTL has comparable performance to dFedU at much lower communication cost.\n5. The paper is well-written and easy to follow.\n\nWeaknesses and questions:\n1. The proposed algorithm has high memory overhead ($\\sum_{j \\in N_i} d_{ij}*d_i$). In reality, the decentralized (cross-device) setups are resource-constrained and can not tolerate memory overheads. \n2. The proposed algorithm has compute and latency overheads as every client needs to compute the edge projections and update its compression matrix.\n3. The experiments are conducted on small-scale datasets and models. Therefore, the impact of the proposed algorithm at scale is unknown.\n4. For the classification tasks, do all the clients have the same classifier structure? \n5. The paper claims \"Sheaf-FMTL algorithm preserves client privacy\": This is not clear as the proposed algorithm is simply projecting the weights into a lower dimension and doesn't provide any theoretical guarantees for privacy. With this argument, any kind of gradient compression method such as sparsification also preserves client privacy. Please clarify what the authors mean by  \"preserves client privacy\". \n6. What graph topologies are used in Figure 3, Figure 4, and Table 3?\n7. How were $d_{ij}$ s chosen? \n8. Is the memory usage reported in Table 3 peak memory or total memory?\n9. Using advanced optimizers such as Quasi Global Momentum (QGM) with D-PSGD can easily improve performance with heterogeneous data. I am curious about the author's thoughts on integrating Sheaf-FMTL with QGM.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel sheaf-based framework for federated multi-task learning that effectively tackles challenges arising from data and sample heterogeneity across clients. It presents a convergence analysis of the proposed algorithm with the average squared norm of the objective function. Experimental results demonstrate that the proposed algorithms' performance on some benchmark datasets is comparable to dFedU but at a lower communication budget.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dfc6FVYTdO", "reviewer": "Reviewer_Ci2G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper presents a comprehensive theoretical framework.\n- The overall presentation is clear and well-structured.\n- The framework successfully incorporates conventional FL algorithms as special cases.\n- The algorithm effectively supports scenarios where nodes utilize models of varying dimensions, showing potential for addressing model heterogeneity challenges.\n\n## Weaknesses\n- Several critical experimental details are currently in the appendix rather than the main text, including the choice of the model, methods for introducing feature heterogeneity, test metric definitions, and the composition of the test set.\n- The evaluation would benefit from more comprehensive experiments. While the paper suggests generalizability to deep neural networks, the current experiments focus exclusively on logistic regression models. Information is compressed based on the dimension of the projected space, leading to the communication savings. It is important to demonstrate that the technique is generalizable to deep models.\n- The algorithm uses a hyperparameter $\\lambda$ whose range can be [0,inf). The paper lacks a discussion on how to choose this hyperparameter.\n- Important baselines are missing from Section 4.2. Since this is the scenario when nodes have the same number of parameters, I believe it is important to show the conventional Federated Learning, D-SGD [1], and EL [2]  as a baseline. Furthermore, since the test sets are (I believe) heterogeneous, it is important to show the scenario where there is no collaboration to establish the benefits of FMTL.\n\nI should note that I am not an expert in the theoretical aspects of this work and would be happy to reconsider my assessment based on the authors' responses and feedback from other reviewers with theoretical expertise in this area.\n\n## References\n[1] Lian, Xiangru, et al. \"Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent.\" Advances in neural information processing systems 30 (2017).\n\n[2] De Vos, Martijn, et al. \"Epidemic learning: Boosting decentralized learning with randomized communication.\" Advances in Neural Information Processing Systems 36 (2023): 36132-36164.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission presents Sheaf-FMTL, a theoretical framework that addresses decentralized federated learning challenges in the presence of feature heterogeneity. The authors frame this problem as a form of Federated Multi-Task Learning (FMTL), where participating nodes learn slightly different but related tasks. The paper demonstrates how this framework theoretically encompasses FMTL, conventional FL, and hybrid FL approaches. Empirical evidence shows that Sheaf-FMTL can achieve better convergence on decentralized FL tasks in the presence of feature heterogeneity and nodes with varying number of model parameters.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Important recommendations\n1. Consider moving key experimental details to the main text, including:\n    - The dFedU weight configuration (aij = 1, ∀(i, j) ∈ E)\n    - Model specifications and dataset heterogeneity implementation\n    - Hyperparameter selection (λ values)\n    - Learning rate details, if space permits\n2. Enhance Section 4.2 with additional baseline comparisons (FL, D-SGD, EL, and a no-collaboration scenario) to better establish the benefits of FMTL.\n3. Expand experiments to include datasets and larger models from established benchmarks such as LEAF [3] or Fedscale [4] - both Sections 4.2 and 4.3.\n4. Provide a discussion on optimal choice of λ. Would it not depend on the connectivity of the graph? If so, including experiments with varied graph topologies would strengthen this discussion.\n5. When discussing deeper models, clarify the implications of nodes having different model sizes. Would the approach remain effective across different model and layer architectures?\n\n## Not necessary but good to have\n- Include background on data heterogeneity types to better contextualize the paper's contributions.\n- Consider discussing the compression benefits of using smaller dimensions in the projected space.\n- In Table 2, adding concrete numbers on FLOPS and Memory Usage will make the paper stronger.\n- While it is not necessary, open-sourcing the implementation code and experimental setup would greatly benefit the research community.\n\nI am open to reconsidering any of these recommendations if the authors can provide convincing arguments for why certain suggestions might be unreasonable or outside the scope of the current work.\n\n## References\n[3] Caldas, Sebastian, et al. \"Leaf: A benchmark for federated settings.\" arXiv preprint arXiv:1812.01097 (2018).\n\n[4] Lai, Fan, et al. \"Fedscale: Benchmarking model and system performance of federated learning at scale.\" International conference on machine learning. PMLR, 2022.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)