Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data
Authors: Xuanyu Chen
NAN YANG
Shuai Wang
Dong Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed Learning
Heterogeneous and Unlabeled Data
Self-Supervised Learning
Federated Learning
Decentralized Learning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We theoretically characterize the robustness of different distributed self-supervised learning frameworks under non-IID unlabeled data, and validate the derived insights through extensive experiments.

Abstract: Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theoretical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.

Published as a conference paper at ICLR 2026
UNDERSTANDING THE ROBUSTNESS OF DISTRIBUTED LEARNING FRAMEWORKS AGAINST
NON-IID DATA
Xuanyu Chen', Nan Yang' Shuai Wang", Dong Yuan
'School of Electrical and Computer Engineering. The University of Sydney NNottwwestern Polytechnical University [xuanyu, chen, n. yang, dong. yyuan]esydneyy edu..u {s @mag) nwpu. edu.
ABSTRACT
Recent research has introduced distributed self-supervised learning (D-SSL) ap- proaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theo- retical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inher- ently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL.) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to- global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.
INTRODUCTION
Deep learning advancements have been driven by large-scale datasets, seen in the training of LLMs, which require billions of data points (Hoffmann et al. 2022 Rae et al.) 2021). However, real-world data is often decentralized, such as surveillance footage from distributed security cameras. This abundance of unlabeled distributed data has spurred interest in distributed self-supervised learning (D-SSL) (Zhuang al. 202 Wang al. (022), which extends self-supervised learning (SSL) to decentralized settings. Existing D-SSL frameworks can generally be distinguished in two aspects: differing by the adopted supe-supe learning (SSL) method or by the applied distributed framework. Self-supervised learning (SSL. is a widely used technique to learn representations without human-labeled annotations by solving pretext tasks that generate supervisory signals from raw data (Gui et 2024). Depending on the approach used to generate supervisory signals, SSL methods are broadly categorized into Contrastive Learning (CL) and Masked Image Modeling (MIM) (Liu et al.
2021 Zhang et al. 2022), with representative methods like SimSiam (Chen & He 2021 and MAE
(He et al.] 2022). On the other hand, federated learning (FL) and decentralized learning (DecL) are two main frameworks in training models with distributed data /Verbraeken al. [020 Sun et al.) 2024). FL aggregates local models via a central server (McMahan et al. 201 Zhuang et al. 202 1a), while DecL. enables direct inter-client communications for aggregating models, enhancing privacy and avoiding the dependence on the central server (Tang et al. 2022 Ayache & El Rouayheb, 2019).
One unique challenge of D-SSL research is handling highly heterogeneous data on clients. Distributed data among multiple clients are normally non-independent and identically distributed (non-lID), leading performance degradation (Zhu et al. 202I). To tackle this challenge, previous works
"Correpooning Authors: Nan Yang and Dong Yuan.

---

Published as a conference paper at ICLR 2026
proposed advanced D-SSL algorithms with robustness to heterogeneous data. Notable examples include FedU (Zhuang et al., 2021a). Orchestra (Lubana et al. 2022). and L-DAWA (Rehman et al. 2023). However, despite continuous algorithmic innovation, there is still lack of theoretical understanding of this heterogeneity problem. For example, FedU was designed within the FL framework, but how would its robustness to non-III) data change if deployed in a Decl. framework D-SSL algorithms are primarily
without coordination from the server? Similarly, state-of-the- based on CL, while the adaptation of MIM methods to distributed settings remains under-explored. Could D-SSL based on MIM offer greater robustness to non-IlI data than CL-based methods? These confusions converge into a fundamental research question affecting the advancement of D-SSL:
How robust are different D-SSL frameworks against data heterogeneity?
To address this question, this paper aims to provide theoretical understanding of how different D- SSL frameworks behave under heterogeneous data. We construct mathematical models in a simplified non-IID setting and rigorously analyze the representability of local and global representations learned by these algorithms. Our analysis reveals two key insights: (i) D-SSL algorithms based on Masked Image Modeling (MIM) are inherently more robust than those based on Contrastive Learning (CL), although their robustness still degrades under severe divergence between local and global distributions; and (ii) the robustness of decentralized SSI. improves with the average connectivity of the network, which suggests that decentralized SSL is only as robust as federated D-SSL in the limited case of full connectivity (i.e., a fully connected network). Building on these insights, we also explore how theoretical results can inform algorithmic design. As illustration, we refine the MIM objective with the additional alignment regularization, which we call MAR loss, to encourage local-to-globa representation consistency. Finally. we conduct extensive experiments
ResNet (He et al., 2016) and Vision Transformer (ViT) (Dosovitskiy et al. (020) across a variety of distributed settings and benchmark datasets to validate our theoretical findings and to demonstrate the usefulness of MAR loss as practical example.
In summary, main contributions are listed below:
1. We develop a rigorous theoretical analysis of distributed self-supervised learning (D-SSL) under non-IID data, showing that MIM- D-SSL is inherently more robust than CL- based D-SSI..
2. We establish the relationship between network connectivity and robustness, proving that decentralized SSL benefits from higher connectivity and that federated SSL. is no less robust than decentralized SSL.
3. We introduce MAR loss as an illustrative case study demonstrating how theoretical results can guide algorithmic design, by refining the MIM objective with alignment regularization.
4. We conduct extensive experiments across model architectures and distributed settings, which validate our theoretical insights and further confirm the effectiveness of MAR loss.
2 RELATED WORK
Self-Supervised Learning. Self-supervised learning (SSL) leverages unlabeled data by generating pseudo labels from raw inputs to learn meaningful representations (Gui et al., 2024). Vision-base SSL methods are typically categorized into contrastive learning (CL) and masked image modeling
(MIM) (hhan et al. 2022 Liu et al., 202T). CL learns representations by maximizing the similarity
between positive pairs (i.e., similar data points created by data augmentation) and minimizing
between negative pairs (i.e., data pairs created by other data points) (Chen et al.] 2020 He et al.
2020). Recent methods like SimSiam (Chen & He| 202T) and BYOL (Grill et al. 2020) advance the original contrastive loss by removing terms related to negative pairs, which improves stability and reduces batch size dependence. MIM, in contrast, randomly masks out patches of input images and predicts the missing parts, learning representations through reconstruction loss (Bao et al.) 2021 Zhou et al. 2021 Xie et al.) 2022 He et al. 2022). Although different in formulation, recent studies have shown that many MIM methods have close connections to CL (i.e., their objectives can be directly re- formulatedd as contrastive loss (Zhang et al.] 2022] Kong et al., 2019)). In this work, we aim to figure out which SSL paradigm is inherently more robust against data heterogeneity.
2

---

Published as a conference paper at ICLR 2026
Distributed Learning. Distributed learning enables collaborative model training across multiple clients without sharing data. Two dominant frameworks in this area are: federated learning (FL), which uses central server to coordinate and aggregate models (McMahan et al.) 2017a), and decentralized learning (DecL.) where clients exchange models locally with neighbors ( (Tang et al. 2022] Ayache & El Rouayheb] (019).. While FL is more widely adopted (Zhang
al. (021) for better convergence and training effecctivnnsss, DecL offers benefits in scalability and privacy. Recent studies have started comparing these two frameworks (Beltrán et al. [023] Hegedůs et al.) 2021). For example, Sun et al. explored which leads to better generalization and the impact of network architecture on generalization (Sun et al. 2024). However, the relationship between network architecture and the non-IID robustness in distributed settings is still unclear. Our work addresses this gap by providing both theoretical analysis and empirical findings to clarify this relationship.
Distributed SSL. Distributed SSL (D-SSL) integrates SSL with distributed frameworks to leverage unlabeled, decentralized data while preserving privacy (Zhuang et al.) 202 Yang et al. 2023). A core challenge is learning robust representations under data heterogeneity (Zhu et al. 20221) Prior work has primarily focused on algorithmic solutions such as FedU (Zhuang et al. 202 la) and L- DAWA (Rehman et al.) 2023). Although some studies also provide theoretical analyses, their purpose is to demonstrate the validity of the proposed algorithms rather than to advance the understanding of the robustness variance between different D-SSL frameworks (Lubana al., [0222] Jing al.) 2024). The most relevant theoretical work is by Wang et al., who showed that SSL is more robust than supervised learning in distributed settings (Wang et al. 2022). Unfortunately, their study only analyzed a specific case of D-SSL where CL is combined with FL and did not extend it to other types of D-SSL frameworks. In contrast, our work delves deeper into these differences, shedding light on understanding the insensitivity of various D-SSL approaches under heterogeneous conditions.
3 PROBLEM SETUP
To provide theoretical insights on understanding this central question, we first introduce our problem setup about distributed training and D-SSL with heterogeneous data.
3.1 DISTRIBUTED TRAINING
Distributed Setting. Consider a distributed scenario consisting of connected network of N clients, represented as a graph G = (V,E), where V is the set of clients and & is the set of edges denoting direct communication links between clients. The connectivity of the graph is captured by a matrix A E R.NN referred to as the adjacency matrix, where A, denotes the set including client i [N] itself and its neighbors shown by E, |Ai] represents the size of this neighborhood set or the connectivity of client i, and |A| - - == (lA,)) is the average connectivity. Hence, distributed training conducted through DecL satisfies Vi E [N]2 < |A, < N. In contrast, FL relies on a central server that aggregates local models from all clients and broadcasts the global model back to them in each round, as in FedAvg (MeMahan et al., 2017a). This architecture effectively enables every client to communicate with all others through the server, which corresponds decentralized topology where Vi E [N], [A,] = N. A more formal specification of the graph structure and the mixing-weins -weight conditions for this distributed setting is provided in Appendix A.6.I
fully connected
Objective of Distributed Optimization. To utilize different clients to learn useful representations, distributed training generally optimizes the below global objectives:
N Σ (WW)) N
W Dec = min
1 N
Σ (WW)) JEA:
W min W
1
A,
where Lj is the objective of local SSL on client j. WDee Dee and WFed denote the global objective of Decl. and FL., respectively. In particular, at each iteration of Decl., each client conducts local updates using the local dataset and aggregates the updated local model with those from neighbors [Tang et al.) 2022). For generating the global model for downstream tasks, there will be an additional aggregation on all local models after all iterations. Differently, the optimization of FL involves each round of model aggregation only on the central server (McMahan et al., 20)7 Then, the server broadcasts the global model to all clients for the next round of training. Note that the FL framework does not need another aggregation between all local models since the updated global model on the server can be used directly for fine-tuning.
3

---

Published as a conference paper at ICLR 2026
3.2 RIGOROUS ANALYSIS OF D-SSL ON SIMPLIFIED NON-IID SETTING
Non-IID Client Data. D-SSL involves all clients collaboratively training a global model by leveraging
their local unlabeled datasets [D,),=] and communicating over the graph G. Since sharing data is prohibited to protect privacy, the heterogeneity across these distributed data sources generally
leads to a performance drop in many distributed applications (Zhuang et al.) 202 McMMhan
et al.) 2017a). Two common types of data heterogeneity are: feature heterogeneity and label heterogeneity (Zhu et al. 2021). In this paper, we follow previous works (Wang et al. [022 Liu et al., 2022) model a simplified but formal label non-IIDness between local datasets as follows. The global data distribution D
U Dj across clients is assumed to contain unlabeled data from 2.V classes. For the dataset on client i, the local data distribution D, is constrained and imbalanced on three classes, with most samples belonging to classes 2i - 1 and 2i, while the re- maining very few samples come from
ex+1..dd
01
e1
eN
Class 25-1
e
03...d
N-1
Closs 25
Clicnt 1
Client
Client
Class by
Figure I: Illustration of the constructed heterogeneous distri- bution for local data on clients. Each client holds two unique data classes.
the class h; E [2N] \ {2i 1, 1.2.). Specifically, for a sufficiently large positive integer d > 0, let x E Rd ~ D, be the data points in the local dataset and , ed be the standard unit-norm vectors of the d-dimensional Euclidean space. For class 2i 1, we set (2i-1)) e; EN == (2- + je (2( where and jr are two positive hyperparameters, q sampled uniformly from {0, 13 and & (O, from Gaussian distribution. Likewise, for class 2i, we define (2(2)) E;- EN 2k 19 (ex + (26) The size of the data from classes 2:- - and 2i are equal and both grow in polynomials of d. For infrequent class h;, the samples generated as: (hhr)) en, + pg(he) and the amount of data is sublinear in d, denoted as (dd)) with a E (0, 1)). Furthermore, we assume all N local datasets to have an equal total number of samples, i.e., Di = D22 ... |. To facilitate understanding, overview of this non-lID data distribution in Figure [ Next, we consider CL and MIM as two main paradigms of SSL and formulate CL and MIM, respectively.
provide an
CL Formulation. For CL., we adopt the more advanced Simsiam Chen & He 2021 which trains with only the positive pairs (ga(x), (Ir), 96 (1:). where ga(+) and gb(-) are random augmentations drawn from SimSiam's ' augmentation policy (e.g., Gaussian noise, flipping). Consider a linear embedding function fw fx) W x, where the weight matrix W satisfies W E Igexd and e 2.N according to the distributed settings, the local objective on client i is defined as:
LCL = -E (WW +
IIWTWI
Eq. 2 captures the SimSiam loss by utilizing the negative inner product (a, b) to measure the distance between the positive pairs. This objective also excludes a feature predictor for simplicity and includes a regularization term | ( for more mathematically tractable, similar to previous works (Wang et al. [022 Liu et al. 2022). Note that (2) stands for general form of Simsiam loss due to the wide class of augmentation functions (Gui et al. (024). For detailed and tractable theoretical exploration,
we consider the linear formulation of data augmentation and further differ CL by the similarity between (-) and (.). In particular, for the case where the positive pairs are generated by similar augmentations, the objective becomes Ler, = EE D. [W (W +
{'))] + JIWWTW W I where& E, ~ (O, I) are random noise sampled IID from the Gaussian distribution. On the other hand, when (-(-) and (-(-) are different, we define the loss as C'c,. EEz~D, [(W(x
E)) (W (Hz))] + WIWWTW [?.. where H pd denotes linear image transformation (c.g.. rotation, translation, etc.). The formal conditions on are given in Appendix A.6.1
MIM Formulation. For MIM, random binary mask m E {0, 1]d (created by uniformly sampling 0 with probability p, i.e., mask ratio) is applied to partition the input æ into two complementary views: the unmasked part 21 @ and the masked part xy = z (1 m) satisfying 21 22 z. Then, we train an encoder- -ecoddr model f = fa far where the encoder fa encodes the input x to a latent representation z fe (x1) and the decoder fa decodes z back pixel space reconstruct the
4

---

Published as a conference paper at ICLR 2026
masked part 22. Hence, considering linear encoder and decoder with embedding matrix We Rexd and Wa E pake, the local objective of MIM is given by
LMIM = Egw De sese - =211² = EE (r m1-
where the mean square error (MSE) loss is utilized to enforce the reconstructed image to be similar to the original image, and @ denotes the Hadamard product. Recent studies have focused the connection between MIM and contrastive losses and found that the MIM reconstruction objective
admits alignment between the masked and unmasked parts (Zhang al. 2022 Kong al. 2019). Based on these results, we adopt an alignment-style formulation of Eq.
3 with == W E
CMIM = E, ((W & (W) @ (1 (1mm))))+++ |WTW1
which implicitly aligns the masked and unmasked views in the embedding space. The regularization term W/TT is also introduced to ensure well-posed quadratic form and improve the traceability.
THEORETICAL INSIGHTS
In this section, we use the above problem setup to model different D-SSL frameworks and compare their robustness to heterogeneous data. Differences in applied SSL and network architecture lead to distinctions in learned representations, which can be further explored to determine variance in robustness. Due to page limitations, the complete proof for our analysis is provided in Appendix A.6
4.1 ANALYSIS OF REPRESENTATION LEARNED BY D-SSL
We begin our analysis with the definition of the representability of the learned representation.
Definition 4.1. (Representability Vector (RV)). Let {ei...., eaf be the standard basis of R4, Let
W .001,..,,1,, [W]]' E Rexd be the feature matrix learned by the linear embedding function (w(x) Wz, where S d. For row space R = row(W) S Rd, we denote the representability of R as vector (e .... [e>) where [Rx)) ) is the projection of e onto for k E [d]. Hence, we have [T [ex)]] = (e where {M1,..., Ve is any orthonormal basis of R.
The intuition behind this definition is that for any input vectors x E Rd, the learned feature space should have a good representation of the standard basis vectors, él, ed, perform well. In particular, these basis vectors should have large projections onto the feature space. The introduction of the representability vector allows to guantitatively assess the feature space learned by different D-SSL frameworks. Similar definitions and notations have also been used in previous works studying the feature space of SSL (Wang et al. 2022, Liu et al. 2022). Based on this definition and the above problem setup, we establish the following theorem for D-SSL. based on MIM pre-training
Theorem 4.2. (Representability of Distributed MIM). Consider a distributed scenario consisting of N (d clients and following the above non-iid setup with d' and = d For distributed SSL that utilizes Masked Image Modeling (MIM) as the pre-training approach, with high probability, the following statements hold:
N be the local RV learned on client i, then we have 1. Let r.M = (r.,...,r
0)d-3) 2p(1 2p(1-p)dō +0(d
S rii < 1, where i E [N]Vk.
2. Let Pra
[ ..... plllT be the global RV learned through DecL, then we have 1 8
2p(1-r)(1-1) A/A +) S FM I 1; while for the global RV RVTM FM = [ F. MMT
Old $)
learned through FL, we have 1 2p(1-p)d5 - +
SPM Fed S 1.
Theorem 4.2 shows the status of the feature space learned by distributed MIM with different objectives
(i.e., local decentralized global vs federated global). Note that for each provided representability vector, find unique lower bound and shared upper bound (considering
= 1). The
5

---

Published as a conference paper at ICLR 2026
distance between the lower and upper bound states how much the learned representation fluctuates in the e unit directions, c1,..., Ce. associated with data generation. Therefore, the smaller the distance, the less sensitive the representation space is to the non-llD distribution of local datasets clients. In other words, the corresponding D-SSL is more robust to heterngeneity.
By similar proof, we derive the representability vectors for D-SSL with CL pre-training follows.
Theorem 4.3. (Representability of Distributed CL). Consider the same distributed scenario in Theorem 4.2 For distributed SSL that utilizes Contrastive Learning (CL) as the pre-training approach, with high probability, the following statements hold:
1. Let Let = 11 Tiel be the local RV, then we have Old d3 d3+0(8 S Sr S and
t
1 1 m(H)at + I rre I $ for similar and dissimilar augmentations, respectively.
2. For the global RV Dec = 1F.....55E F learned through DecL, we have O(d
O(d 8)
Thee 1 and in(D((-1/1AA)d +old-3) S & I for similar and dissimilar augmentations; whilefor FFed [PP ...., pel learned through FL, Old $)
U-1/IAna
Old $)
we have
db -ed 4Ola 8) SFFod S and m(H)d -edd ) by S Sfre 1.
Theorem 4.3 demonstrates that the local and global feature spaces learned by distributed CL are distinct from those learned by distributed MIM. However, it is not obvious which feature spaces hold a smaller gap between the lower and upper bounds. To determine which type of pre-training less sensitive to data heterogeneity, we further compare their global feature spaces learned in DecL and FL framework, respectively, and summarize the results in the following theorem.
4.2 MIM IS INHERENTLY MORE ROBUST THAN CL WITH HETEROGENEOUS DATA
Theorem 4.4. Let s = max Fk mingelo) Fi be the sensitivity of distributed SSL to hetero- geneous data x E Rd,, measured as the spread of the leading coordinates of the learned global representabilitty vector For any network architecture, distributed SSL satisfies the following prop- erty: lima-roo > sM where d and M represent the sensitivities of distributed SSL adopting contrastive learning and masked image modeling as the pre-training approach, respectively.
The main intuition for the greater robustness (or smaller sensitivity) of distributed MIM is that CL learns representations from aligning features of the positive pair generated from the original data through data augmentation, whereas MIM aligns features of the reconstructed and the raw data to learn representations. Although the applied augmentation generally does not lead change in data labels (Chen et al. 2020 Chen & He 2021), the output is still different image. In contrast, the masking operation splits the original image into the masked and unmasked parts, but a portion of the original data is retained in both parts. As a result, CL. learns local representation with greater randomness, and that additional randomness is also biased by local labels. Considering that data heterogeneity already exists among clients, the global representation learned by distributed CL is less uniform than that learned by distributed MIM.
4.3 IMPACT OF THE AVERAGE CLIENT CONNECTIVITY ON NON-IID ROBUSTNESS
Next, we shift our focus to another dimension that distinguishes D-SSL algorithms and address the question: how does the network architecture affect the robustness of the feature space learned by D-SSL? The tool for solving this question is again the bounds of the representability vector. For the
DecL setup where clients directly communicate with their direct neighbors, Theorem 4.2 and 4.3 have implicitly shown the answer.
Corollary 4.5. For any SSL pre-training approaches, if the distributed scenario is fully decentralized (i.e., without a central server), the robustness of distributed SSL against heterogeneous local data improves with the average connectivity |A] between clients in the network.

---

Published as a conference paper at ICLR 2026
also implies that the robustness of D-SSL conducted in federated setup should be no worse than in a fully decentralized network. Consider the best case of the network topology, where each client can communicate with all other clients in the network. In this case, each client receives model aggregated by the local models from all clients, which is exactly the global model distributed by the server in the federated setup. We can continue exploring to verify that this intuition is correct. Theoretically, combining Theorem 4.2] Theorem 4.3] and Corollary [45) we arrive at another main theorem addressing the question introduced the beginning of this section.
Corollary 4.5
Theorem 4.6. For any SSL pre- ave-training paradigms, distributed SSL satisfies the following property: ((k)
lim Dec 2 syeal, where DDe = maxxe(() Dec minngee] DDe denotes the sensitivity of distributed SSL performed in the DecL (i.e., clients directly communicate with neighbors), and
((R)
Fedd = maXke[c] ((R) minggle] Fed represents the sensitivity of distributed SSL performed in the FL setup (i.e,, all clients are indirectly connected through the central server).
_(k)
This thcorem further demonstrates the robustness trade-off between applying SSL in federated and decentralized frameworks. For less concern about the impact of data heterogeneity, we should conduct distributed SSL in a federated setup (often also referred to as federated self-supervised learning (Zhuang al., 2021b Lubana al. [2022] Rehman et al. 2023)). However, the decentralized case is more common in reality, as it is challenging to provide central server that can be trusted by all clients and has stable communication with them. Then, can consider increasing the average connectivity between clients to minimize the negative impact of heterogeneous data on training (e.g.. identifying under-connected clients and creating additional direct communication links).
MAR LOSS: AN ILLUSTRATIVE CASE STUDY IN ENHANCING ROBUSTNESS
The preceding analysis has addressed the main focus of this paper by establishing theoretical insights into the robustness of different D-SSL frameworks under heterogeneous data. As a further step. we illustrate how these insights can guide a more robust algorithmic design. In particular, our results show that although distributed MIM is fundamentally more robust than CL., training dynamics are dominated by the client-specific covariance, causing local encoders to drift toward different directions before aggregation gradually mitigates this effect. This observation motivates us to refine the MIM objective with additional term that explicitly and dynamically promotes consistency between local and global masked representations, which we term MAR loss. The integration of MAR into both federated and decentralized frameworks is summarized in Algorithm [ and Algorithm
Formally, MAR loss augments the MIM objective with an alignment regularization term:
AMAR Ez=D, Exy [Ifa(f((=))) 22ll2 + +2(00 A-MMMD(Z,, . 3)
(5)
where 2, fe((1)) and denote the local masked and global representations, and 7/ > 0 is dynamic weight for alignment. The alignment regularizer based on Maximum Mean Discrepancy (MMD), a widely used measure of distributional discrepancy in machine learning (Gretton et al.) 2012 [i et al., 2017] Gong al., 2016)) MMD compares whether two distributions and Q differ by mapping samples into reproducing kernel Hilbert space (RKHS) and evaluating differences in their feature means. Typically, MMD adopts a Gaussian kernel k( x') = exp(-1ix - x'11 /
In MAR, we employ an adaptive version (A-MMD) to compare the feature spaces of local and global representations more robustly. Unlike prior FL works that use vanilla MMD (Ma et al. 2024:] Hu al.) 2024 Liao et al.) 2024b), A-MMD selects the kernel bandwidth automatically rather than fixing
it. Given batches of local and global embeddings of equal size B, A-MMD is computed as:
B
2
A-MMD(2, (2,, 8) B(B - 1)
E( 21,6)
k(Eas >b(Eas mo)
k(2,,e Eo), (6)
B
agd
with the adaptive kernel defined as k(2, 2²) ) = exp(- 2(mean...
=01l)= This data- data-driven choice ensures stability across non-lID clients by scaling the kernel to the observed embedding distribution.
||8-2'11
Finally, to balance early-stage consensus and late-stage efficiency, we design the regularization weight (6)
it to decay smoothly from mmax to 7min- We adopt cosine schedule based on client participation:
Tt 7min + (7max 7mmnn) 1 + COS T:
(7)
7

---

Published as a conference paper at ICLR 2026
where w/ counts the number of times client has been selected up to round and Q controls the decay horizon. In Decl. where all clients participate every round, one can simply set Ω = T. In FL with partial participation. a practical choice the expected number of selections per client, or I as default. This schedule applies stronger alignment when client divergence is most pronounced, and gradually relaxes toward "/nin as training progresses, ensuring dynamic robustness gains.
EXPERIMENTS
In this section, we conduct extensive experiments validate the correctness of our derived theoretical insights and evaluate the effectiveness of the MAR loss in improving the robustness of distributed MIM against data heterogeneity. We first introduce the experimental setup. Then we assess our results in different datasets, model backbones, and distributed settings.
6.1 EXPERIMENTAL SETUP
Datasets and Distributed Simulation. We pre-train our models on the Mini-ImageNet dataset (Vinyals et al. 2016), which contains 60,000 images extracted from ImageNet (Deng al.] 2009). To simulate a distributed scenario with label non-IIDness, the dataset is partitioned by sampling the class priors of the Dirichlet distribution (Hsu al. 2019 More heterogeneous division can be made with a smaller Dirichlet parameter during sampling, while the IID case is simulated with a very large a. Besides, we follow prior works to simulate feature heterogeneity by uniformly dividing datasets and applying unique data augmentation for each client (Wang et al. 2022 Zhu et al. 2021). Hence, the local labels are kept the same but features are skewed into different domains before training. Furthermore, to simulate DecL, a connected network with the number of clients and the average connectivity as inputs and return the adjacency matrix A. For FL, we additionally assume central server connecting with all clients. After pre-traiin pre-training, the models' backbones are fine-tuned on benchmark datasets, including CIFAR- 10, CIFAR-100 (Krizhevsky et al. 2009). and ImageNet. The fine-tuming accuracies used for analysis.
use the Erdōs-Rényí model (ERDdS & R&wi, 1959) to initialize
Implementation Details. For our experiments, we use ResNet (He et al. 2016) and Vision Trans- former (ViT) (Dosovitskiy al. 2020) as the model architecture. Following the problem setup in theoretical analysis, we select Simsiam (Chen & He 2021) and MAE (He et al. (022) as the repre- sentatives of CL and MIM pre-training, respectively. In original works, Simsiam used pre-train ResNet models, while MAE is used to pre-train ViT's We implement two new SSL baselines to show that our theoretical insights apply any model architecture. One uses Simsiam to pre-train ViTs, and the other one pre- are-trains ResNet through MAE. Furthermore, we follow the classical distributed algorithms, D-PSGD (Lian al.) 2017) and FedAvg (McMahan et all 2017a), implement the DecL and FL. frameworks, and then implement our FedMAR and DecMAR algorithms based on these frameworks. All our codes are implemented in Python using the Pytorch framework and executed on server with 4 NVIDIA@ RTX 3090 GPUs. The detailed training setup and server configuration can be found in Appendix A.2 because of page limitations.
6.2 EMPIRICAL STUDY
Insensitivity Superiority of Distributed MIM. Table [ compares the impact of data heterogeneity on the pre-training effectiveness between distributed MIM and CL. With highly heterogeneous data, the learned local feature space will be significantly different across clients, resulting in greater divergence between local and global feature space and larger drop in the performance compared to the IID setup (Zhuang al. 2021a b Lubana al., 2022). Across various datasets and backbone architectures, we observe that distributed MIM consistently exhibits smaller gap between IID and non-lll> settings compared to distributed CL. The experimental results align with Theorem 4.4 verifying that MIM is less sensitive than CL when handling heterogencous data in distributed scenarios. To further substantiate this theoretical insight, we also visualize the local and global feature spaces learned by distributed MIM and CL and compute the ly-norm weight distance between their local and global models. Please see Appendix A.3.T for these external experimental results.
Impact of Average Connectivity on Non-IID Robustness. We verify second insight by setting up decentralized networks with different average connectivity |A]. For the same |A]. we consider two cases: (1) a general case where the number of neighbors [A,] varies across clients, and (2)
8

---

Published as a conference paper at ICLR 2026
Table Fine-tuning accuracy (%) of backbones pre-trained by different D-SSL algorithms. All results provided in this table are the mean of three trials (L/non-IID) = Label Non-IID; F/non-IID = Feature Non-IID). The values in brackets denote the gap between IID and non-III performance.
CIFAR-10
CHEAR-1O 100
ImageNet
IID
L/non-IID
Finon-IID
IID
L/non-IID
Flaon-IID
IID
L/bea-IID
F/aeo-IID
Simsiam CNN
86.03
84.33 (11,70)
84.62 (1144)
58.91
(L1.11)
57.81 (11.10)
46.74
46. (4.66)
46.41 (10.33)
MAE- + CNN
87.28
86.97 (10.311
86.17 (11.11)
57.86
57.77 (10.09)
((0.66)
45.88
45.87 (10.01)
45,80 (1.0.08
72.32
(J2.82)
(11.66)
(15.11)
(J5.51)
61.97
(12.11)
59. (L2284)
MAE+ VIT
69.90
68.20 (U1.70)
69.32 (10.58)
50.04
48.95 (L1.09)
(J,0.4)
62.69
62.25 (10044)
62.51 (10.18)
61.0
Federaled Methods
Decentralized Methods
51.5
60.8
50.8
50.5
80A
50.0
80.2
Fedaested
00.0
Decartraliadd (Unfiem)
FedMIM
DecMIM DesMAR
Deoentralibed (Genera)
10
12
14
18
20
Average Connactivty
(A]
Non-IID Level
Non-IID Level
(a)
(b)
Figure 2: (a) Impact of the average connectivity between clients on the non-IID robustness. Models are pre-trained in network with 20 clients and then fine-tuned on CIFAR-100. The blue line shows the results of DecL, and the orange line shows FL results. (b) Comparison of MAR and MIM loss on robustness to data heterogeneity in federated and decentralized settings.
uniform case where all clients have the same connectivity, i.c., vi E [. A). Additionally, we set up an FL scenario with 20 clients training in parallel per round. Figure 2a shows that Corollary 4.5 correct. We can observe that the fine-tuning accuracy of decentralized SSL increases with A. Moreover, Figure 2a provides empirical evidence for Theorem 4.6 We find that pre-training in FL is no less robust than DecL against heterogeneous data. Additional results using alternative consensus matrices for DecL are given in Appendix A.3.2 and confirm the same robustness ordering.
Effectiveness MAR loss. To further illustrate the practical relevance of our analysis, we evaluate MAR loss against the standard MIM objective in both FL. and Decl. frameworks under varying degrees of heterogeneity. Figure 2b shows that, as the level of non-IIDness increases (i.e., the Dirichlet parameter decreases from I to 0.001), the fine-tuning accuracy of all methods declines. Nevertheless, models pre-trained with MAR loss consistently outperform those trained with the vanilla MIM loss across all non-lID levels. This trend holds in both FL and DecL settings, suggesting that MAR loss can effectively reduce the sensitivity of distributed MIM to heterogeneity.
comprehensive evaluation, extend the comparison recent federated SSL baselines. We evaluate the effectiveness of our FedMAR by comparing it against several state-of-the-an art (SOTA) federated self-supervised learning (F-SSL) baselines non-IID distributed setting. The SOTA baselines involve: 1) FedU (Zhuang et al. 2021a); Using the divergence-aware predictor module for dynamic updates within the self-supervised BYOL network (Grill al. 2020);
Besides, provide
4) FedEMA (Zhuang al. 2021b): Employing EMA of the global model to adaptively update online networks; 5) Orchestra (Lubana al.] 2022): Combining clustering algorithms with Federated Learning for better model aggregation. 6) FeatARC (Wang et al. 2022): Combing clustering techniques with feature alignment; 7) LDDAWA (Rehman et al. 2023) Smartly aggregating models
according to the angular divergence between local models; and 8) FedU2 (Liao et al. 2024a): Optimizing training with the flexible uniform regularizer and efficient unified aggregator. Following prior works (Zhuang al. 2021aa Rehman al. 2023), we simulate highly heterogeneous scenario with 100 clients sampled from a Dirichlet distribution with a = 0.1. In each round, 5 clients are randomly selected and each conducts 10 epochs of local training for 200 rounds in total.
Since most baselines employ ResNet-18 18 (He et al., 2016) as the backbone, we first implement FedMAR with ResNet-18 18 for direct comparison. As shown in Table 2 FedMAR employed

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of FedMAR with the state-of-the-art F-SSL methods on the Non-iid version (a 0.1) under cross-device (n 100) settings. Each method was pre-trained with Mini-ImageNer Dataset. The table shows the mean fine-tuning accuracy (%) of three trials.
Method
Architecture
Params
GFLOPS
CIFAR-10
CIFAR-100
ImageNet
FedU
ResNet-18
38.47M
7.40
72.02
38.44
65.10
FedEMA
ResNet-18
38.47M
7.40
70.73
40.78
65.24
Orchestra
ResNet-18
11.84M
7.31
88.87
70.11
65.02
FeatARC
ResNet-18
11.70M
1.83
89.60
64.11
68.17
LDAWA
ResNet-18
15.39M
1,83
89.95
68.96
51,43
FedU2
ResNet-18
15.39M
1.83
82.39
55.49
45.27
FedMAR(OOrs)
ResNet-18
22.50M
3.64
92.70
70.82
65.36
FedMAR(OUrs)
Tiny-ViT
11.60M
0.88
90.03
71.28
75.99
ResNet-18 18 achieves higher accuracy on CIFAR-10 10 and CIFAR-100 while obtaining comparable results ImageNet. This indicates that MAR loss can provide tangible improvements even when using the same CNN backbone as prior methods. To further examine the generality of MAR, we also evaluate FedMAR with a lightweight Vision Transformer backbone (Tiny-ViT). Importantly, this model has a comparable number of parameters and GFLOPs to ResNet-18 18, ensuring fairness in comparison. In this setting, FedMAR employed on Tiny-ViT achieves superior performance on all three benchmarks, surpassing CNN-based baselines while maintaining lower computational cost. These results suggest that MAR loss particularly effective when applied to transformer- -bsse models in federated self-supervised learning,
limited to convolutional architectures and can be
Extended Study on MAR. In addition to the comparative experiments. we also conduct ablation studies on the main components of MAR loss, including the alignment term and its dynamic weighting. The detailed results of these analyses are provided AA.41 and AA..2] respectively. The reported results have confirmed that each component contributes to the performance improvement led by MAR loss. Finally, we assess the practical feasibility MAR by analyzing its privacy concerns and communication overhead. Since MAR communicates only masked embeddings, the additional communication overhead remains modest, and raw data are never shared. Privacy is therefore also preserved through masking and can be further strengthened with differential privacy. A detailed discussion is reported Appendix A.5
7 CONCLUSION
In this paper, investigated the robustness of distributed self-supervised learning (D-SSL) under heterogeneous data. Our theoretical analysis shows that MIM-based frameworks achieve greater robustness than CL-based ones, and that the degree of robustness in decentralized learning closely tied to the average network connectivity, with federated learning being no less robust than decentral- ized learning. These findings provide principled foundation for understanding how algorithmic choices and network structures affect distributed learning with unlabeled and heterogencous data, Beyond the theory, we also illustrated how such insights can inform practical design. As a case study, we introduced MAR loss, a refinement of the MIM objective with alignment regularization, which serves to demonstrate the applicability of our analysis. Extensive experiments across model architectures and distributed settings validate our theoretical predictions, and further confirm the utility MAR loss in practice. We hope that our results can serve as a theoretical grounding and guiding framework for future developments in distributed self-supervised learning.
ACKNOWLEDGMENTS
This work is partly supported by the Australian Research Council Linkage Project (Grant No. I. IP2200208893..
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ghadir Ayache and Salim El Rouayheb. Random walk gradient descent for decentralized learning on graphs. 20/9 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 926-931. IEEE, 2019.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. 2021.
arXiv preprint
Enrique Tomás Martínez Beltrán, Mario Quiles Pérez, Pedro Miguel Sánchez Sánchez, Sergio López Bernal, Gérôme Bovet, Manuel Gil Pérez, Gregorio Martínez Pérez, and Alberto Huertas Celdrán. Decentralized federated learning: Fundamentals, state of the art, frameworks, trends, and chal- lenges. IEEE Communications Surveys & Tutorials, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, (13):211-218, 1936.
P ERDdS and A R&wi. On random graphs i. Publ. math. debrecen, 6(290-297):18, 1959.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference machine learning, pp. 2839-2848. PMLR, 2016.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural Information Processing Systems, 33:21271 122244, 2020.
Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey on self- suleevupere learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
11

---

Published as a conference paper at ICLR 2026
István Hegedus, Gábor Danner, and Märk Jelasity. Decentralized learning works: An empirical com- parison of gossip learning and federated learning. Journal of Parallel and Distributed Computing. 148:109-124, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 30016-30030, 2022.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non- non-identical data distribution for federated visual classification. arXiv preprint
2019.
Kai Hu, Yaogen Li, Shuai Zhang, Jiasheng Wu, Sheng Gong, Shanshan Jiang, and Liguo Weng. Fedmmd: a federated weighting algorithm considering non-iid and local model deviation. Expert Systems with Applications. 237:121463. 2024.
Shusen Jing, Anlan Yu, Shuai Zhang, and Songyang Zhang, Fedsc: Provable federated self-supervised learning with spectral contrastive objective over non-iid data. arXiv preprint arXiv:2405.03949, 2024.
Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. mutual information maximization perspective of language representation learning, arXiv preprint arXiv:1910.08350, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnahás P6ezos. Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017.
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh, Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? case study for decentralized parallel stochastic gradient descent. Advances in neural information processing systems, 30, 2017.
Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Fengyuan Yu, Huabin Zhu, Binhui Yao, Tao Wang, Xiaolin Zheng, and Yanchao Tan. Rethinking the representation in federated unsupervised learning with non-iid data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 22841-22850 2024a.
Xinting Liao, Weiming Liu, Pengyang Zhou, Fengyuan Yu, Jiahe Xu, Jun Wang. Wenjie Wang, Chaochao Chen, and Xiaolin Zheng. Foogd: Federated collaboration for both out-ff-distributioo generalization and detection. Advances in Neural Information Processing Systems, 37:132908- 132945, 2024b.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations, 2022.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE transactions on knowledge and data engi- neering, 35(1):857-876. 2021.
Ekdeep Lubana, Chi Ian Tang, Fahim Kawsar, Robert Dick, and Akhil Mathur. Orchestra: Unsu- pervised federated learning via globally consistent clustering. In International Conference on Machine Learning, pp. 14461-14484. PMLR, 2022.
Xiao Ma, Hong Shen, Wenqi Lyu, and Wei Ke. Enhancing federated learning robustness in non-iid data environments via mmd-based distribution alignment. International Conference on Parallel and Distributed Computing: Applications and Technologies, pp. 280-291. Springer, 2024.
Leland McInnes John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 28220033226 2018.
12

---

Published as a conference paper at ICLR 2026
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication effiiiiennf learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017a.
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.0696: 2017b.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 12.11 1446, 2021.
Yasar Abbas Ur Rehman, Yan Gao, Pedro Porto Buarque De Gusmão, Mina Alibeigi, Jiajun Shen, and Nicholas D) Lane. L-dawa: Layer-wise divergence aware weight aggregation in federated self-superviseed visual representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16464-16473, 2023.
Yan Sun, Li Shen, and Dacheng Tao. Towards understanding generalization and stability gaps between centralized and decentralized federated learning, 2024. URL
abs/2310. 03461
Zhenheng Tang, Shaohuai Shi, Bo Li, and Xiaowen Chu. Gossipfl: A decentralized federated learning framework with sparsified and adaptive communication. IEEE Transactions on Parallel and Distributed Systems, 34(3):909-92) 2022.
Joost Verbracken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan Rellermeyer. A survey on distributed machine learning. Acm computing surveys (csur), 53(2): 1-33, 2020.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake. Does learning from de- centralized non-iid unlabeled data benefit from self supervision? arXiv preprint 17019:2210.10947, 2022.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE transactions on information forensics and security, 15:3454-3469, 2020.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9653-9663 2022.
Nan Yang, Xuanyu Chen, Charles Liu, Dong Yuan, Wei Bao, and Lizhen Cui. Fedmae: Federated self-supervised learning with one-block masked auto-encoder. arXiv preprint 133. 2023.
Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey federated learning. Snowledge-Basee Systems, 2167:0 2021.
Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. Advances in Neural Information Processing Systems, 35:27127-2711 2022.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 1xXi::2//100783: 2021.
Hangyu Zhu, Jinjin Xu. Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Neurocomputing, 465:371-390, 2021.
13

---

Published as a conference paper at ICLR 2026
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsuper- vised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2021a.
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. In International Conference on Learning Representations, 2021b.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 FULL PSEUDOCODE OF D-SSL WITH MAR LOSS
Algorithm 1 FedMAR Algorithm
Input: initial model W number of local updates E, number of training rounds T, learning rate 7) the upper bound of regularization weight 7max the lower bound 7min Output: optimized global model W
1: for = ......,
do
2: if 0 then
3: server broadcasts 8 to ~ [N]
4: else
5: computes 7tt (0) by 7max and 7min server (shown in Eq. 7
6: server broadcasts W ;, 7t
(8)
~ [N]
7: end if
8: for client i E C in parallel do WT <- W
9:
10:
ift then
W i < SGD(W Lo Tb, E, (MIM)
12:
else
Wt i, t SGD(W n7, CMAR(Z, )))) (shown in Eq.
14:
end if
15:
sends Wi i,E 2, to server
16: end for
17: 2 [C] EiEc 2
18: W + C Eiec W
i
19:
end for
Algorithm 2 DecMAR Algorithm
Input: initial models W number of local updates E, number of training rounds T,, learning rate 7, the upper bound of regularization weight Tmaax, the lower bound Ymin
Output: optimized global model W
1: for = 0, T do
2: for client E [N] in parallel do
3: if = 0 then
4: send i to its neighbors
5: else
computes 7i (8) by Tmax and 7mia for each neighbor (shown in Eq. Eq.
6:
send W Ri, 7t to its neighbors
(i)
7:
8:
= Ai Ejea, Zj
9:
end if
10:
W.t < A. Ejen, Wt-1
11:
if = then
W 1. 2, SG:(WW: f,u 1), E, LMIM)
13:
else
W! WEE t SGD(W) t, E, AMAR(E, (19)
(()) (shown in Eq.
14:
end if
16:
end for
end for
18: WT
N/4 EiE[M] W
15

---

Published as a conference paper at ICLR 2026
A.2 DETAILS ABOUTT EXPERIMENT SETUP
In this section, we have provided two tables to present our experiment setup. Table 3 shows the experiment details, which include the specific settings for the model architecture, dataset, scenario, and training. Table I demonstrates the setup of the running environment, including the configuration of our test server.
Table 3: Settings of Experiments.
Details
Model Architecture
ResNet, Vision Transformer (ViT)
Number of layers in ResNet
18
Number of blocks in ViT
5
Pre-train Method
MAE, Simsiam
Pre-train Dataset
Mini- Mini-ImageNer CIFAR- 10/100, ImageNet (1e5 (IID,, 1, 0.1, 0.01, 0.001} {1, 0.1, 0.01, 0.001}
Fine-tune Dataset
Non-IID Options (i.e. the value of a))
Options for the 7 used in MAR loss For Federated Learning (FL.):
Number of clients
100
Number of sampled clients per round
5
Number of local training epochs
2 100
Number of total training rounds
For Decentralized Learning (DecL):
Number of clients
20
Options for average connectivity
3,5, 10, 20 (equals to FL)
Number of local training epochs
1
Number of total training rounds
25
Fine-tum tunint Epochs
50/100 101FAR--0/100,... 20/100 (ImageNet)
Preitrain Batch Size
128
Fine-tune Batch Size
256 CCIFAR-10/100 1024 (ImageNet)
Base Learning Rate
1.5e-4
Table 4: Settings of Running Environment.
Config
Details
Server GPU Count
Server GPU Type
RTX 3090 (24GB)
Server CPU Type
AMD EPYC 7282 16-Core
CUDA
12.4
Framework
Py PyToc
16

---

Published as a conference paper at ICLR 2026
A.3
EXTERNAL EXPERIMENTS
A.3.1
FEATURE SPACE VISUALIZATION AND MODEL DIFFERENCE
ViT Client 1 Local Model
MAE+VIT Client 1 Local Model
VIR+ Client 1 Local Model
15
20
des5
cass
diss 9
Cass
class
dea 5
class
class
da67 7
dass
class
dass
class
4.0
15
1à
VVT Client 100 Local Model
MAE+VVTT Client 100 Model
MAR+ Client 100 Local Model
15
dows dacs 8
clacs
class
cass
clasd
dass
Class
decs
Coss
class
dans
class
class
Simsiam+ViT Global Model
MAE+VIT Global Model
MAR+ Global Model
Class
dass
clsss
class
dea
cless
Class
dass
cass
Class
doss 9
clas
Class
-2
5.5
255
06
25
56
75
130
125
150
Figure 3: Visualization the feature space of local and global model in Non-IID setting. Each column stands for D-SSL framework (i.e., pre-training ViT by Simsiam, pre-training ViT by MAE, and pre-training VIT by MAR). The first row shows the local feature space from client 1, the second row shows the local feature space from client 100, and the last row shows the global feature space.
Besides Table demonstrating the non-IID robustness of distributed CL and MIM by the gap in fine- tuning accuracy, we further explore the differences in their learned features empirically. Specifically, we simulate heterogeneous setting with 100 clients using Dirichlet sampling with 0.1. For each D-SSI. framework, we obtain three pre-trained ViT backbones: (1) global model trained using across all clients; and (2) two local models trained solely on data from client and client 100, respectively. To compare their learned feature spaces, extract the encoder features of each model. These high-dimensional features are first projected to 20 dimensions using principal component analysis (PCA) and then embedded into 2D space using Umap (Melnnes et al. 2018) for visualization.
Figure presents the feature of local and global models learned by each D-SSL method. Each column corresponds to method, while each row shows features from specific model (client 1, client 100, and the global model). We observe that for distributed MIM methods, the local features aligned with each other and also closer the global features, suggesting more consistent
17

---

Published as a conference paper at ICLR 2026
representations across heterogeneous clients. In contrast, distributed CL exhibits greater divergence between local and global features, indicating that is inherently more sensitive to data heterogeneity.
To provide more guantitative comparison, also show the weight differences between local and global models in Table In particular, we compute the layer-wise &2 2-rnorm difference between local and global model weights and report the across all layers. The results show that distributed MIM methods (MAE and MAR) yield significantly lower weight distances compared to distributed CL, reinforcing the observation that MIM leads more stable and consistent model updates in the presence of non-lID data.
Table 5: Weight distance between local and global models learned from different D-SSL methods.
12-Norm Difference
SimSiam VIT
MAE
VIT
MAR + VIT
local vs local 100
45.37
36.10
35.75
local vs global
40.34
31.57
31.38
local 100 vs global
38.39
31.77
31.25
A.3.2 EFFECT OF DIFFERENT CONSENSUS MATRICES ON ROBUSTNESS THEORY
Table 6: CIFAR-100 Accuracy (%) of decentralized MIM under different consensus matrices. Results are averaged over three test runs.
Method
Avg Connectivity 2s
Avg Connectivity 10
FL (reference)
52.72
52.72
DecL with Data Size Weights
46.62
52.28
DecL with Degree Normalized
45.34
52.08
DecL with Doubly Stochastic
45,22
52.18
DecL with Push Sum
45.19
52.22
This experiment examines whether the robustness findings in Section remain valid under different choices consensus matrices in decentralized learning. The theoretical bounds link robustness to the average connectivity of the network graph, while the average connectivity is closely related to how efficiently information mixes across clients. If various consensus rules produced qualitatively different mixing behavior, they could in principle affect robustness. To test this, conduct two groups of experiments on decentralized networks with 20 clients under strong non-lID conditions using a 0.1. In the first group, the network has an average connectivity of around 5, while in the second group, connectivity is increased to about 10. Within each group, we pre-train distributed MIM using four commonly adopted consensus schemes, including data size weighting, degree normalized averaging, doubly stochastic matrices, and push sum, and compare all results against federated learning baseline with the same number of clients. The results in Ta a consistent pattern. When connectivity low, all decentralized variants suffer noticeable accuracy loss relative to federated learning, and the specific consensus rule makes only minor differences. When connectivity increases, all decentralized variants recover to level that is close to the federated baseline yet never surpass it. These findings confirm that the qualitative ordering predicted by the theory persists. The choice of consensus matrix influences only constant factors in mixing but does not overturn the robustness relation that federated learning is at least as robust as decentralized learning.
A.4 ABLATION STUDIES ON MAR
A.4.1 ABLATION ON ALIGNMENT METRIC
Our MAR loss (Eq. 5 involves two key components: the dynamic regularization weight y and the A-MMD distributional penalty used to align local and global representations. To understand their impact, we perform ablation studies on each component. We first evaluate the contribution of the alignment metric.
For baselines, we consider two commonly used choices in prior work: cosine similarity, which has been widely adopted in federated SSL studies for enforcing alignment between local and global
18

---

Published as a conference paper at ICLR 2026
Table 7: Evaluation of different alignment metries for MAR loss on CIFAR-100. We report
accuracy (%) under three settings of fixed yy: 1c- 1,-1, 1c-2, and (degenerate to vanilla MIM).
Metric
Y= le-1
7= le-2 2
0
Cosine Similarity
51.71
52.47
51.45
Vanilla MMD (o =
51.79
52.12
51.45
A-MMD (median σ)
52.42
54.13
51.45
A-MMD (mean σ)) [Ours]
54.09
54.39
51.45
feature spaces (Wang et al., 2022), and vanilla MMD with fixed kernel bandwidth, which has Hu
also been explored in recent federated learning works (Ma et al., (2024) Hu et al., 2024 Liao et al.) 2024b). On top of these, we evaluate our adaptive variant A-MMD, where the kernel bandwidth is chosen automatically based on either the median or mean of pairwise distances. As shown in Table 7 A-MMD consistently outperforms cosine similarity and vanilla MMD across different values. Between the two adaptive variants, using the mean pairwise distances provides slightly better performance, and adopt this as default design.
A.4.2 ABLATION ON REGULARIZATION WEIGHT
Table 8: Evaluation of regularization weight for MAR loss on CIFAR-100.
Weight Schedule
Acc(%)
51.50
le-1
54.09
= le-2
54.39
1e-3
53.55
7: le-1 - 1e-3 (cosine decay)
54.91
Next, we analyze the impact of the regularization weight y by fixing the alignment metric to A-MMD. Results in Table show that using a large weight (? 1) degrades performance, as the alignment term overwhelms the reconstruction objective. Conversely, very small weights such as 7 1e-3 reduce MAR to near-vanilla MIM objective and fail to deliver sufficient robustness gains. Moderate fixed values such as y 1c-2 and y 1c-1 yield stronger results, but still remain below our proposed dynamic schedule.
Notably, the cosine decay schedule that smoothly decreases y from 1c- 1e-3 achieves the best performance (54.91 ()). %). This validates our intuition behind dynamic weighting: stronger alignment is most beneficial in the early stage when client divergence is high, while gradually relaxing the weight avoids excessive penalty in later stages. These findings highlight the importance of the dynamic design in MAR loss, which not only achieves higher accuracy but also improves training stability.
A.5 DISCUSSION ON PRIVACY AND COMMUNICATION OVERHEAD OF MAR
When deploying MAR loss in practice, natural concerns arise regarding the potential privacy risks and the additional communication associated with sharing local representations, We provide both quantitative and qualitative analyses below show that these costs remain modest and manageable.
Privacy considerations. The information communicated by MAR is limited to local representations 2 (21) derived from the unmasked portion of the input. Because MIM typically adopts high masking ratio (e.g., 75% in MAE (He et al. 2022)), most raw content remains hidden and the embedding dimensionality substantially reduced, which mitigates potential leakage. For stronger guarantees, MAR can be further combined with standard Differential Privacy (DP) mechanisms McMMhan et al. 2017b Wei et al.) 2020) by perturbing embeddings before transmission, e.g..
2: f=(e (0, of) with calibrated to satisfy (c, 5)-P.
Communication overhead. In addition to the standard model updates (e.g., gradients or weights), MAR transmits compact masked embeddings computed from the unmasked portion of each input. This is the sole extra payload introduced by MAR. For instance, in the MAE (ViT-B/16) setting on ImageNet with 75% masking ratio, each image has 196 patches, of which 49 remain visible. With
19

---

Published as a conference paper at ICLR 2026
hidden size 768 and batch size 256, this yields about 49 x 768 x 256 float values (z 36.8 MB in float32). By contrast, a full model with 86M parameters is = 328 MB, so the additional cost from MAR is only ~ 11% under this configuration. Crucially, in cross-device settings where small batches are common, this extra cost decreases proportionally with the batch size: at B=128 is R 18 MB (~5%), at B-64 it is z MB 3%), and B=32 it drops around 1%. These calculations indicate that the MAR-induced overhead remains acceptable in realistic deployments. Moreover, MAR is optional: when minimal communication is the overriding priority, one can simply use the standard MIM objective, whose effectiveness is explained by our theory, at zero additional cost. When small extra cost acceptable, MAR offers corresponding robustness gains while keeping the overhead low.
20

---

Published as a conference paper at ICLR 2026
A.6 FULL PROOF FOR THEORETICAL ANALYSIS
A.6.1 FORMAL ASSUMPTIONS
To make the theoretical analysis fully transparent and self-contained, we first summarize here all assumptions used deriving the main results. These assumptions complement the problem setup in Section 3 and reflect the standard modeling choices commonly adopted in theoretical studies of distributed and self-supervised learning.
Assumption A.1. (Communication Graph). The distributed system modeled as a fixed and connected communication graph G = (V. E) with N = [V] clients. Each client communicates only with its neighborhood A, [j (1,)) Ui) We assume 2 S [A:] & for all i E [N]. The average neighborhood size |A| N >E>! |A;] is used as a measure of connectivity.
Remark A.2. This formulation encompasses decentralized learning on arbitrary connected topologies and federated learning the fully connected special case (i.e., by the help of the central server, all clients can indirectly communicate with each other so there exists |A,| = N for all E [N]
Assumption A.3. (Consensus Weights). During decentralized aggregation, each client i forms a mixing vector wt {wi, IEA, satisfying:
*
Wij > 0 only if E A, reepelig sparsity);
.
EjEA, Wij (row-stochasticity)
Remark A.4. The above conditions represent the standard requirements for decentralized model aggregation: each client averages only over its local neighborhood and the mixing vector is row- stochastic. This formulation covers commonly used consensus rules in decentralized optimization, including uniform averaging (McMahan et al., 2017a), sngree-normalizer weights (Lian et al. (017). and symmetric doubly-stochaste schemes (Tang et al. 2022]. Our analysis relies only on these basic structural properties, while more general mixing operators could in principle be incorporated by extending the corresponding aggregation step. Exploring such extensions is an interesting direction for future work, but it is not required for the results presented here.
Assumption A.5. (Non-degenerate Embedding). Throughout the analysis, we focus on non-trivial stationary points of the regularized objectives, where the embedding matrix W e Rexd satisfies W * 0 and rank(W) c.
Remark A.6. The trivial solution W = 0 does not minimize the reconstruction or alignment terms in either the MIM CL objectives, and corresponds representation carrying no information. Therefore, this assumption is generally satisfied in the theoretical analysis of self-supervised learning (Liu et 2022, Wang et al. 2022).
Assumption A.7. (Independence Local Sampling.) For each client the local dataset D; consists of [D, independent samples drawn from its local distribution D;-
Remark A.8. The independence assumption is the minimal condition required for spectral norm bounds of empirical covariance matrices. does alter the established non-IID structure across clients, but ensures that the empirical covariance on each client concentrates around its population counterpart. This is standard assumption in the theoretical analysis of distributed learning (Wang et al. 2022 Tang et al. 2022).
Assumption A.9. (Dissimilar Image Transformation for CL.. When the two augmented views (go (x), 9b (xx)) used in CL. are generated from dissimilar transformations, model (x())) by linear operator H E ddxd acting on the input space. In the theoretical analysis, H enters only through the quadratic form relevant. Let S = span(er..... e denote the class-dependent subspace in the non-lID generative model, and let P be the orthogonal projection onto S. We assume that
Hx, and therefore only its symmetric component Hsym (H H ) is
tr( tr(PH P) 0.
Remark A0 The above condition ensures that the transformation H preserves nontrivial energy on the class- dependee semantic subspace S. It is a mild requirement and is common to hold in standard contrastive learning augmentations (Chen et al.) 2020 Chen & He 2021), including rotations, flips, translations, crops, blurs, and color jittering. These transformations perturb the input in ways that do not cancel class-discrininative directions, ir( Pyymm remains strictly positive in practice.
21

---

Published as a conference paper at ICLR 2026
A.6.2 LEARNED REPRESENTABILITY FOR DECENTRALIZED MIM
This section provides the full proof of Theorem 4.2
Proof. We begin by formulating the representability of local representation. Then, we derive the global representation based on the local feature. Since is different from decentralized learning in the updates, establish the global representation for each distributed framework, respectively.
For local feature space. According to the alignment-style loss function MIM shown in Eq. and by the definition of Kronecker product, we have
CMIM = Berm==BBlllle ([WWac W
= ([((((ddig(vve(x))) (W(ne)))((())))((((((()) vee(1 - m)))] |
Define
a (tiag(vec(z)) vec(m),
b diag(vec(z)) -ee(1 m).
(9)
so that the above loss becomes
- + |
(10)
Using the fact that
a WTW = tr(a W = tr(WTWbaa) = tr(Wha
(11)
we obtain
aa WTb = ae
(tr(Wha = W(ba + abT).
(12)
Together with the gradient of the complete objective becomes
0 (
W 2WWTW,
(13)
aW
0L
WE (ba
T + abT] + 2WWTW.
(14)
aw
Setting the gradient to zero yields
WE [haT + abT] = 2WWTW.
(15)
Under Assumption A.5 multiplying both sides the left by the Moore-Penrost pseudoinverse W+ reduces Eq. 15) to the stationary condition
1 E 2
[a [baT + abb = WTW.
(16)
Let X.M represent the left-hand side of this equation. Consider the binary matrix m used for masking is sampled uniformly from the binomial distribution with probability p, we establish
E(diag( (vec( (ddaeeemmmTdiivveeeee X
+ diag (vec (ec)))))))) Tdidia (vec
(17)
[D
2p (1 - p)
Σ
(diag (vec (I:,)) diag (vec (I:,))
DD,l
where RA Dr. matrix for the learning with local dataset client i. Based the setup of data generation in Section we also derive the following expectation of X.M with dt and μ d
1 I/ai (diag (vec (mzz)) diag (vec (TT denotes the empirical covari-
22

---

Published as a conference paper at ICLR 2026
E [X:]] =
diag
2p -p) + + 21 p) + + d (
s.n 2p - p) + +
1 term
N termss
0 (d 0 d
(
(18)
d-N lerms
diag
(0 (1
2p (1 -
-l- p) + (110))))00(100
2p d + +0 ( 0d
Next, consider the fact that up to a positive scaling and an additive constant, the regularized MIM objective can be rewritten the Frobenius-norm objective L(W) | WTW Thus, minimizing LMIM solves the Frobenius-nome best rank-c approximation problem for X.M According to the Eckart- Eckart-Young-Mire Young-Mirsky theorem (Eckart & Young 1936). we notice that the row span of the optimal W E Rexd is the span of the eigenvectors corresponding the first eigenvalues of MM M X.M. Denoting the set of orthonormal eigenvectors of x.M as ".1::.. Vi,d
we have X.M
Ej=1 o (uNf)) where Aog := (x,²)) the j-th largest eigenvalue of X:M Therefore, the inequality below is satisfied:
c[ ex co
( ex
Σ (e
(19)
X > (c
for any with & E [N] Mo.. On the other hand, under the data construction described in Section 3.2 the number of samples on each client equals to the sum of the samples from frequent classes and the rare class. Since each of the two frequent classes grows in polynomials of d, while the amount of data from the rare class (dd)) with a E (0, 1), the local sample size satisfies |D; = O(d)) with 3 Z 1. Based on this sufficiently large sample size and Assumption A.7| the matrix concentration bounds (Vershynin) 2018) implies that the spectral norm satisfies X| E X < O (d- with
probability least 1 -ee Building Weyl's inequality, obtain that with high probability,
PaM ALE
[
|X
E
[x 122 1 ( (
(20)
By combining Eqs.
18
19
and 20 we can derive the below lower bound for e e[X.Mea ex:
exxMe e = E X M
e + t [ - E [XM]] ex
2 2p p) 2p d di + 0 (
I| -E [XM]]
(21)
> 21 - - 0 (1
-
23

---

Published as a conference paper at ICLR 2026
which is led by the fact that || as follows:
||||| for symmetric X. Likewise, we prove the upper bound
x, M ck = e [x + [ - [ [I en
2p (1-11pp) -1 p) d +0 ( + +|* - -E [x;"]]
(22)
- 2p (1 d + (
Moreover, we notice from Eqs.
and 19 that the below statements hold for X,1
X 2 A (B (xx -
- ( 2 2p (1- p) - (d
(23)
M S A ( [ + (d 2p (1 - p) + (1=3)
With Eqs.
21
23 we further establish
2p - di - 0 d (d
Σ 22
2p (1 - p) db + 0 0 d
5=1
2p (1 -) d + d-
20 d-
(24)
2p p) d +0 dd
2p (1 p) d +0 d
0 (
2p p) d3 + d-
This completes the proof for local representation.
For global feature space. Since the local goal can be equivalently re-formulated as | WMM-W 1
by Assumptions A. and A.3 we re-write the global goal D-SSL for DecL framework (shown in Eq. as
min W
1 N
Σ KE(N]
A
Σ
-
(25)
JEA:
Note that the following function holds the same minimizer as Eq. 25
minl|
Σ
Σ
XMM
WTW
N
A
NE[N]
jEA;
1
min| W N
Σ
X.
WTW
(26)
1E[N]
min
X W
XM
2
M denotes the empirical covariance matrix for training with the local where X.M EjEA. [A.] ;
datasets across the local datasets on client and its neighbors. So, finding the optimal W for DecL is equivalent to solving 24. Following the derivation of Eq. (8) and linearity of expectation,
24

---

Published as a conference paper at ICLR 2026
establish
E X = diag
(.., 2p (1 - p) (11-11)
d5 +
1
+
+0
d +
................................ 2p
p) d + + (d
[A,
[
jEAMA
5 term
(27)
terms
0 (d. (d
d-N terms where we prove with the fact that
(IAx) 1) 2p (1 p) p) +2p (1 -1-)) p) + 11
[Ai
(14,1 -1)2p (1 - p) di + 2p (1 p)
+0 (d-8)
|Ail
(28)
2p (1-p) (1-
1
d 2p (1 -p) p)
0 (d=
|Aiil
AA
2p (1 p)
((1
1
di +
1 IAil
+ 0 (
|4d
With Eq. we can also have
XM
=
-diag
2p (1 p)
d + (r ..... 2p (1 - p)
(1- A
(1-
d8 +
dd #
A
(29)
N terms
..., 0 (d -..0(aa))) (d=t))
where we consider
1 N
Σi=1 [A,] =
=|Å| and the fact that
2p (1 - p) ((1
dd
+
+
0
(d ))
1=
A.l
[A.]
N
N
N
2p (1 p)
1
di
+
Σ
1
+ +0 (a
N
[Asl
N
A
i=1
i=1
(30)
= =2p (1 - -p)
1
d
+
+
(d
[
A
2p (1 - p)
d
+
0 (
A
Through similar proof from Eq [N]:
21
to Eq
23
we prove that the following statements hold for all
i
e ek 2 2p (1 -p)
1
1
dt o
(d+
A
(31)
e e ev < 2p (1 p))
1
1
d +
+0(10
(d
A
25

---

Published as a conference paper at ICLR 2026
M 2 \1
R
XM
( +0(1+8)
+
=p -p)
1
di 0 dd
A
(32)
M
E
XM
+ +0
(
2p (1 p)
1
d +0 (d
A
which then implies:
2p (1 -p)
[1- [
di 0 (d
dt +0 (d +
j=1
2p (1 - p)
(1-
2p (1 -p)
d + +0 d-
20 (d-3)
A
(33)
2p (1-)))
1
d3 0
2p p) (1-1#) [AT
1
d3 +0 (d-3)
A
0 (d-3)
d3 + + (d-8) (
2p (1 p)
1
[
The proof for the global featured space learned in the decentralized learning framework has been completed. Next, consider federated learning (FL) special case of decentralized learning with
Vi [V], |A| = N. The global of FL thus:
1 N
min
Σ KE(N
1 - WTW
(34)
This is similar to solving
min
Min|XXM - -PXM-WWWIB, W
(35)
where XM == N/2 EiENN X.M denotes the empirical covariance matrix for learning with the global dataset. Then, we derive
E XMM diag
2p (1 p) - 0 (ddtt) +0 (1-3)
(36)
2p p) dt - (atit) + O (d-3) ,..O ( (d=))
where we adopt N ((d)b) have used the fact that
(N 1) (1 p) d + 2p p) + NO
N
(0 (d( d -1) 2p (1 p) dt + 21 (1-pppp) -p)
+
(37)
θ
= 2pp (1 p) (1
( 10
de
+
1 10
+ 0 (d
2p (1 p) d3 0
+ +0 (
26

---

Published as a conference paper at ICLR 2026
Again, by similar arguments from Eq.
21 Eq.
23
we further prove
2p (1 p) d θ
(d
(d=3
Σ (efo
2p (1 p) d3
o
d
+ + d-33
21 p) di + 2
+ 3
20 (d
(38)
2p - p) di d
+0 d
p - p)
+ +0 +0(
0 (
=1
2p (1 p) d) - 0
+ 0 (d)
which completes the proof of this theorem.
27

---

Published as a conference paper at ICLR 2026
A.6.3 LEARNED REPRESENTABILITE FOR DECENTRALIZED CONTRASTIVE LEARNING
This section provides the full proof of Theorem
Lemma A.11. (Representabilitty of Distributed CL under Similar Augmentations), Consider the same distributed scenario in Theorem 4.2 For distributed SSL that utilizes Contrastive Learning (CL) in pre-training and generate positive pairs through similar augmentations, with high probability, the following statements hold:
1. Let r = - TiAT
be the local RV learned on client If positive pairs are generated Old 3) d3 d3 S T S 1, where i [N]]kk
by similar augmentations, we have
2. Let Bec = [ 1..... TelT [ be the RV learned through the global objective of framewook then we have
Old b)
S 50 S
(1 (a jd to
3. Let F.M Fed = [PM .... be the RV learned through the global objective of FL framewok,
we have 1 d3 e(d200200006 5) S 1 S. 1.
Old
Proof. Following the proof in A.6.2 we first discuss local representability learned by distributed contrastive learning and then derive the global representation based these local features. Since federated learning differs from decentralized learning in terms of updates, we construct separate global representations for each distributed framework.
For local feature space. Let learning (CL) shown in Section 3.2 we obtain
& and = + g'. Based on the loss function of contrastive
1 [TT
LCL EZ=DD,(((( (W) (x +\)
(39)
La + |
=
By the same derivation between Eq.
11 -1. 14h. the gradient of the above function is
dCCl aw = WE
+ T)T (z (xx + 2WWTW.
(40)
To find the minimizer of LCL, we solve for
WB (2 + (2) + T) + (x + +)x + wwW 0.
(41)
Under Assumption A.5 it leads to
2xx + T + +g'xT + =(E') + ExT + + E(EE) = WTW.
(42)
Similarly, let X. represent the left-hand side of this equation. We then establish
X²C
Σ
[2 + #1sell El + xis(E')T E.gel, + EL + +E,(5)),)
(43)
28

---

Published as a conference paper at ICLR 2026
where X represents the empirical covariance matrix for the local feature learned by CL client 1.
Considering that E, ~N( f), we also derive the following expectation of X
(X: = diag
72 + 0 0.0003 (d3) 20 (d ,
0 (.. + 20 (d-b) y...y 72 00 )
20 (a (d=))
5t
N terms
...20 20 (d + O (d- ,..., 20 (d (d..))...., +
d-N toms
ddagg (d + (d (0(+t)
+ ( d + 0 (d ( (d+t))
(44)
Next, using similar arguments from Eqs.
to 23 we arrive at the below results:
d8 0 (d (
S <elife S +
(45)
d3 - 0 ( s N < d +0 (133)
With these inequalities, derive
db 0 (
d3 00
1=1
(46)
0 (
d8 + (
which completes the proof of the local part.
For global feature space, Since the local goal can be equivalently reformulated as || - WTW by Assumptions and | and A.3 the global goal of distributed contrastive learning the decentralized
learning (DecL) framework is given by
min W
Σ N
A
X - WT
(47)
N
JEA,
Furthermore, we find this is equivalent to solving
minl W
XC - WTW
N
A
iE
N
JEA,
=min|
X
WTWI?
(48)
M
=
X WTW
29

---

Published as a conference paper at ICLR 2026
Again, using similar arguments from Eq.
27
to Eq.
33
further establish
d 0 (d
A
(E)0))>
N
1
1
d3 + +0 d +d(tt)
[
(49)
0
d
dd + 0 (d+))
The proof for the global feature space learned in the DeeL framework has been completed. Next, denote federated learning (FL.) as special case of decentralized learning with Vi, |A,] N. The global objective of FL is expressed as
min X WTWIL
(50)
where we denote XC = N EiENN X By similar arguments from Eq. (36 Eq. 38) we have
d
θ
d
d
d
θ
d
+ 0
d
di + d
dd
20 d
(51)
di @
d
d
d3
d
+0
d
(d
di
+
d
which completes the proof of this lemma.
Then, we start to prove Theorem 4.3 as follows.
Proof. Lemma A. demonstrates the learned local and global representations of distributed when positive pairs are generated by similar augmentations. For the other case using dissimilar augmentations, we adopt a similar process to derive the local and global representations.
For local feature space. Define a = + and h learning (CL) with dissimilar augmentations in Section 3.2
H.. According to the loss function of contrastive we have
1 2
Cec E (w + E (W +
WTW
(52)
- |WTWI
The minimizer of this loss function is
dL aw
- WE Har( + E)T + + ()++))=TTT) 2WWTW= 0.
(53)
Rearranging it under Assumption A.5 derives
E Hxxx TT Hx WTW.
(54)
30

---

Published as a conference paper at ICLR 2026
Let X denote the left-hand side of the above equation. Hence,
x
1 [Ux( + E)T + (x [)=
1
2
D.
(55)
1
[Hanaa]a + Hra + mngel HT + HT)
2 D.
Similarly, based on the formulation that NO 1). can be written as
db and μ d= the expectation of X;"
B(X;")= tr(
(H) + (
(HH)r² +0 + (d
diag
temn terms
N
..... ( ((.t)),....0
o(r (((t))....0t
+ diag
terms
= ddiag (I((H)d& + + (att. ,..., tr(H) + 0 (d
tr(H)d + to o (d=)))
(56)
Following the proof process from Eqs.
20 to 23).
the following inequalities can be found
|X - E [x(9 so |4% - ArB [x["
unH)ddt - 0 (d S <ep ex S u((H)dd +o(
(57)
((H)d 0 (d= s y ((H)d +0
0
However, unlike the previous proof, there exists a potential issue that the image transformation matrix H may lead to the case that X not a square matrix. Then we denote X;" = - A iij" where of and of left and right singular vectors produced by SVD decomposition. So, we have
C'
xX ex
Σ
Eusletuf
(58)
C² <
letu u exl
1,1
1=1
which further leads to
trH)dd 0 (d (d
d
le u 1 2
tr(Hdd + +0 dd
1=1
(59)
o ( (d-
(n)dd + 0 dd
31

---

Published as a conference paper at ICLR 2026
For global feature space. By similar augments from Eq. the global representation learned through the decentralized learning framework, we establish
27
Eq. 33 and based on (5. for
tr(H)
d 0 (d
Σ exl
u
" j
2
ir(H)
1 [
d8 0 (d-b)
(60)
0 (dtbb)
=1
(r(
1
dš + +0 (d+t)
|
On the other hand, for the global objective of the federated learning framework, we follow the arguments from Eq. (36) to Eq. (38) to derive
tr(H)d3 θ d
0 d
e u
of exl
urH)dd3 O d
0
d
(61)
0 (
tr( (H)d3 - θ d
0 d=
Combining Lemma A.1I Eq.
Eq. (0) and Eq. (6] completes the proof.
A.6.4 PROOF FIRST THEORETICAL INSIGHT
This section provides the full proof of Theorem 4.4
can find the main difference between the global representations lies in the lower bound. For the global feature learned in the decentralized learning (DecL) framework, denote the sensitivity of D-SSL below:
Proof. According to Theorem 4.2 and Theorem 4.3
0 (d-
Dec
(62)
di +0 (a-8)
2p - p)
1 |7]
O (d=b)
(63)
Dec
d3 0 (d
1 A
o (
Ca Dec
(64)
di O (d-3)
tr(H)
|
where M represents the sensitivity of MIM-based D-SSL heterogeneous data, & represents the sensitivity of CL-based SSL with similar augmentations, and Ca represents the sensitivity of CL-based SSL with dissimilar augmentations. Then, we compare the magnitude of s Dec and Cs by solving the following equation:
C1
Dec .
32

---

Published as a conference paper at ICLR 2026
0 (d
(
M Dec
C2 Dec
di 0 (d=3)
(1- |
1 d - -0
0
d
d3 0 (d=)) (d ((1-
d3 +0(+3))
(65)
[
[
dt 0 (d=t))
(11 - di + + (d)
[3]
Consider the dimension d of the Euclidean space is very large so that d - oc. Then, we have
lim [s sBaol=
d
0 (
(11-
1
d - (d
(d )) - (a-t)) ((1-)) |
d +0 (d (a
A
lim d+00
d3 o dd ) ((1 (11- (²
d8 0 (d=))
(66)
d'b
A
lim
d=000
2
(d')
Due to the fact that 2 <
A
N, we prove
lim d=
Dec - Stal <0.
(67)
Similarly, under Assumption A.9
we determine if Dec is less than C2 Dec as follows
0 (d
C2 lim
tr(H)
1
|
de + +0 (d-b)
d
Dec M
lim d000
00;
(68)
d
8 Dec
0 d
d
2p(1 p)
(
1
d3 + +0 (d-3) +0(d-)
[
which implies
lim [e - Dee < 0. d=000
(69)
Combining (77 and 69 arrives
lim Ms d
Scel < O,
(70)
where S.e denotes the sensitivity of Ca-e SSL to heterogeneous data. On the other hand, for the federated learning (FL) framework, denote the following sensitivity of D-SSL:
d
8 Fee
(71)
2p (1 - p) d
d
+
+0
d
33

---

Published as a conference paper at ICLR 2026
(
8 Fee
(72)
d3 θ d
+ +0 d
0 (
C2
(73)
Fed
tr (n(H)d+
θ d
+0 d-+
The difference between M and C1 is given by Fed
(
M
C1
Fee
FFed
2p (1 p) θ d 2
d
1 0 d d
0 (
1-
0 (P-2)
( (d 1
+0 (d)))
1-
+0 (
(1
(d
+ 0 (0())
(d
dt - 0 20
+ (
(74)
For the above result, let d + oc, we can establish
0 d #
+
d
-0
(d- (
lim Med - SPed d=100
lim
lim
<0 0
(75)
dt
O
2d
+
0
d
dt
Then for the comparison between Wed and C2
Fedd under Assumption A.9
we have
d
C2
tr(H) d3
-
d
+0
d
d
lim
Fed
lim d=000
(76)
Fed
0 d
d
d θ d
0 d
With Eqs. 75 and 76
we find
lim IsMed Feal < 0.
(77)
Combining Fq.77 and (7. completes the proof.
34

---

Published as a conference paper at ICLR 2026
A.6.5 PROOF OF SECOND THEORETICAL INSIGHT
This section provides the full proof of Corollary 4.5 and Theorem 4.6
Proof. For the decentralized learning (DecL) framework, we notice from Eqs. 662 their denominators both include the term
66 and (64) that
Since |A| is proportional to [
derive
[
that A is inversely proportional to M C1 Dect Dec and C2
DDect which completes the proof of Corollary we compare the robustness of distributed MIM
Next, by similar proof from Eq. to (q.(7))
between DecL and FL framework by solving
o (d-8)
0 (
M M
$ Dec
FFed
2p (1 pp)
(1-
1
d + +0 (d
2p(1 - p) d d
+ 0
d
(78)
This is equivalent to solving
1
d (2p p) -0(dt)) )
2p (1 -p-))
1
[
(79)
= 2p - p) d8
2p (1 - p)
di 2p (1 p) d 0 d (d
A
Due to the fact that
lim d-000
2p (1 -2)
2p (1 p) p),33
-2p (1 p) db 1 =d())
<0,
(80)
A
have
lim M
sM 0.
(81)
d=000
Similarly, for CL-based SSL, we have
o dd-
C1
Ci
(82)
Dec
Fee
d O (d
d o
+ 0 d
o (
0
C2 Dec
C2 FFed
(83)
d 0 (d-
tr(H)d - θ d
+ 0 d
tr(
1
Under Assumption A.9 the above results imply that
lim Dec d=
C1 0,
(84)
lim Is C2 d+00
Dec
C3 &ed > 0.
(85)
With Eqs.
84) and 18
we find
lim S &f 0.
(86)
d=300
35

---

Published as a conference paper at ICLR 2026
Combining Eq. with Eq. 86 derives
lim Dec > sped].
(87)
Note that Eq. (8) holds for decentralized learning setups in which each client has an inconsistent number neighbors. However, there exists an optimal case for decentralized learning, denoted by vi, [A,] NN. In this case, the global objective of decentralized learning be re-formulated as follows:
(88)
N NE JE[N]
*E[N]
This equation is exactly the same the global objective of federated learning shown in Eq. Therefore, we know the below statement holds:
lim d
[De = SFed);
(89)
when Vi E [AV] |A,] ... Combining Eq. 87 and Eq. (8) completes the proof.
36


Reviews:
[{"review_id": "ErHoMGdGdN", "reviewer": "Reviewer_gLn9", "summary": "The paper analyzes robustness in distributed self-supervised learning under non-IID client data, comparing contrastive learning (CL) and masked image modeling (MIM) in both decentralized (peer-to-peer) and federated (server-aggregated) settings. The main theoretical result is that MIM is inherently less sensitive to client heterogeneity than CL. The analysis also shows that robustness in decentralized SSL improves with higher average graph connectivity, and that federated SSL is at least as robust as decentralized SSL. Motivated by the theory, the authors introduce MAR: a MIM objective augmented with an adaptive MMD alignment term whose weight decays over training to encourage early consensus. Experiments on standard vision benchmarks indicate that MIM and MAR degrade less from IID to non-IID splits and that MAR provides consistent gains with modest communication overhead.", "questions": "1) Can you state the graph/weighting assumptions explicitly in the main text (not just appendix) as a numbered assumption set\n2) Do the robustness results still hold under alternative consensus matrices (doubly-stochastic, row/column-stochastic, push-sum for directed graphs)? If not, what changes in the bounds?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "*  Strong theoretical analysis. Comparative guarantees showing MIM’s robustness advantages over CL; useful system-level insight about connectivity and federated vs. decentralized training\n* The paper establishes the relationship between network connectivity and robustness. This provides useful system-level guidance: increased connectivity improves decentralized robustness.", "weaknesses": "* Restrictive weighting model: only uniform neighbor averaging; no analysis or experiments with alternative consensus matrices (e.g., doubly-stochastic, push-sum), adaptive/learned weights, or time-varying/intermittent connectivity.\n* Topology realism is limited: results are tied to average degree rather than spectral gap/mixing rate; experiments rely on simple random graphs, not clustered or heavy-tailed real networks.\n* Sensitivity to MAR hyperparameters (mask ratio, batch size affecting MMD, decay schedule) is underexplored.\n* Limited empirical evaluation breadth: few baselines/datasets and sparse ablations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cl9Huohxod", "reviewer": "Reviewer_PoK1", "summary": "This paper presents a theoretical analysis of the robustness of Distributed Self-Supervised Learning (D-SSL) frameworks against non-IID data. The authors' primary contributions are twofold: (1) A theoretical argument, based on a simplified linear model, that Masked Image Modeling (MIM) is inherently more robust to data heterogeneity than Contrastive Learning (CL). (2) A theoretical claim that the robustness of D-SSL improves with network connectivity, implying that Federated Learning (FL) is no less robust than Decentralized Learning (DecL). To illustrate the practical implications of their theory, the paper introduces MAR loss, a refinement of the MIM objective that adds a local-to-global representation alignment regularizer using Maximum Mean Discrepancy (MMD). The authors conduct extensive experiments to validate their theoretical claims and the effectiveness of MAR loss.", "questions": "1. Could you elaborate on why the insights from a linear model $f_W(x) = Wx$ are expected to hold for highly non-linear models like ViT and ResNet? Is there any specific property of SSL objectives (MIM/CL) that makes this simplification more valid than it would be for, say, standard supervised learning?\n\n2. The analysis relies on a specific data generation process where each client has nearly orthogonal data classes. How does this simplified model capture the complexities of real-world non-IID data, such as the Dirichlet distribution used in your experiments, where clients share overlapping classes with different priors? Could the theory be extended to more realistic data distributions?\n\n3. The use of MMD to align distributions in FL is an established technique. Could you please clarify the core novelty of MAR loss beyond the combination of A-MMD and a decay schedule? How is the design of MAR loss a direct and unique consequence of your theoretical findings (e.g., Theorem 4.2), rather than a general-purpose regularizer that could be applied to many FL scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses a critical and underexplored question in the D-SSL literature: the fundamental robustness of different frameworks (MIM vs. CL, FL vs. DecL) to non-IID data. Attempting to build a theoretical foundation for this problem is an ambitious and valuable goal. The questions asked are highly relevant to the community.\n\n2. The experimental evaluation is comprehensive. The authors validate their claims across multiple model architectures (ResNet and ViT), SSL paradigms (SimSiam and MAE), distributed settings (FL and DecL), and benchmark datasets. The inclusion of experiments on feature space visualization and ablation studies on MAR loss strengthens the empirical part of the paper.", "weaknesses": "1. The entire theoretical analysis (Theorems 4.2-4.6) is built upon a set of highly simplified and potentially unrealistic assumptions. The analysis is restricted to a linear embedding function $f_W(x) = Wx$. Modern SSL methods, especially those using ViT and ResNet backbones, rely on deep, highly non-linear transformations. It is not at all obvious why insights derived from a linear model would generalize to these complex settings. The paper fails to provide any justification for this leap of faith, other than \"mathematical tractability.\" This severely limits the generality and impact of the theoretical claims. The non-IID data generation process described in Section 3.2 is artificial. Data points are constructed from orthogonal basis vectors ($e_i$), and each client $i$ is assigned data from two specific, unique classes ($2i-1$ and $2i$). This is a toy problem that does not reflect the complexity of real-world label heterogeneity, where classes can have semantic overlap and distributions are far more nuanced. There is a significant disconnect, as the theory is derived for this synthetic setting, while experiments use the more standard (and complex) Dirichlet distribution.\n\n2. The paper presents MAR loss as an \"illustrative case study,\" but it is also listed as a main contribution. The idea of using a distributional alignment penalty like MMD to enforce consistency between local and global models in FL is not new (e.g., work on FedMD, FedProx, etc., which use similar ideas for knowledge distillation or regularization). The contributions of MAR loss—using an adaptive MMD kernel and a cosine decay schedule for the weight—are incremental engineering refinements rather than a fundamental algorithmic innovation. Framing this as a key contribution overstates its novelty and distracts from the paper's main focus on theoretical understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M8ZUWaOKxE", "reviewer": "Reviewer_MVSv", "summary": "The paper studies how different distributed self-supervised learning (D-SSL) choices handle non-IID client data. It analyzes a simplified label-skew model (each client i mostly has two classes plus a rare third) and defines a Representability Vector (RV)—the squared projection of standard basis vectors onto the learned feature subspace—to quantify “robustness” of different methods. The main theoretical claims are: (i) MIM-based D-SSL is asymptotically less sensitive to non-IIDness than CL-based D-SSL (Theorem 4.4), and (ii) robustness improves with the average client connectivity; FL is no less robust than DecL (Corollary 4.5, Theorem 4.6). Building on this, the authors propose MAR and report empirical trends on Mini-ImageNet pretraining with CIFAR-10/100 and ImageNet fine-tuning.", "questions": "Answers to the following questions may change my assessment:\n\n1. Please provide a rigorous derivation (or inequality bounds) showing that the reconstruction loss in Eq. (3) is equal to (or tightly bounded by) the alignment-only objective in Eq. (4) plus a data-independent Frobenius regularizer. If it is merely a lower/upper bound, how do the subsequent minimizer and eigenspace arguments change? \n\n2. In Appendix A.7, how do you get from the true gradients of $-\\mathbb{E}\\|(W a)^\\top b\\|^2$ (and the CL analog) to the identities $W^\\top W = \\mathbb{E}[\\cdots]$ (Eqs. (9)–(10), (35), (47))? Could you rewrite the proofs using the form $-\\mathrm{tr}(W^\\top A W)$ with $A=\\mathbb{E}[aa^\\top bb^\\top]$, leading to $A W = W W^\\top W$, and then justify the Eckart–Young step from there?\n\n3. Should the rare-class mean be $e_{h_i}$ (not $e_i$) and should the sum over offsets use $q_k$ (independent Bernoullis) rather than a single $q$? If yes, how do these corrections affect the covariance calculations and bounds?\n\n4. Please formalize $s=\\lceil\\bar r\\rceil-\\lfloor\\bar r\\rfloor$: which coordinates of $\\bar r$ are considered, and is it a max–min over all $d$ or only the first $c$? How sensitive are Theorems 4.4 and 4.6 to this choice?\n\n5. What properties of $H$ are required (e.g., PSD, orthogonal, $\\operatorname{tr}(H)\\ge 0$) to ensure the denominators in Theorem 4.3 and Eq. (54) are positive and the inequalities hold?\n\n6. Please specify the per-client sample sizes (as functions of $d$) and the independence assumptions used for concentration. Can you restate the bounds explicitly in terms of $n_i$ (client-level sample sizes) rather than only in powers of $d$? (Appendix A.7)\n\n7. Do the key inequalities (MIM < CL sensitivity; FL ≥ DecL) persist under non-linear encoders/decoders (e.g., Lipschitz networks) and realistic augmentations? Any partial evidence or counter-examples? (Secs. 3–4)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a central D-SSL design question and provides actionable hypotheses . The empirical section illustrates these points across backbones and topologies. \n- The RV/sensitivity notion is intuitive for comparing learned subspaces across frameworks.", "weaknesses": "Below are the most critical technical issues I found while reading the proofs and setup.\n\n1) Incorrect “equivalence” of the MIM objective (Eq. (3) → Eq. (4)). \nEq. (3) is an $\\ell_2$ reconstruction loss $\\|W(x\\odot m)-(x\\odot (1-m))\\|^2$. Eq. (4) replaces it by a negative squared inner product $-\\|(W(x\\odot m))^\\top(x\\odot(1-m))\\|^2 + \\tfrac12\\|W^\\top W\\|_F^2$, stated as \"equivalent\". This drops the $\\|W(x\\odot m)\\|^2$ and $\\|x\\odot(1-m)\\|^2$ terms that do affect the optimizer (the latter is data-dependent), and it adds a data-independent Frobenius penalty as a surrogate. No derivation is provided for this equality, only a citation to qualitative connections between MIM and contrastive alignment (which do **not** imply equality of objectives). The subsequent analysis treats (4) as the exact objective. This is a fundamental mismatch that invalidates the later first-order conditions and eigen-structure arguments built on top of (4). \n\n2) First-order optimality conditions are derived with wrong shapes and unjustified rearrangements.\nIn Appendix A.7, the gradient of the (modified) losses is set to zero and directly rearranged to $W^\\top W = \\mathbb{E}[\\cdots]$ (Eqs. (9)–(10) for MIM; Eqs. (33)–(35) for CL). However:\n  - For MIM, the objective involves an expectation of a *squared scalar* $\\|(W a)^\\top b\\|^2$. Its gradient has the form $-2A W$ with $A$ built from $\\mathbb{E}[aa^\\top b b^\\top]$, not $-2 W A$. The step \"$-2W\\,\\mathbb{E}[\\cdots]+2WW^\\top W=0 \\Rightarrow W^\\top W=\\mathbb{E}[\\cdots]$\" reverses the multiplication order and implicitly assumes invertibility/commutativity that does not hold. (pp. 19–20)\n  - For CL, Eq. (33) expands $\\|(W(x+\\xi))^\\top(W(x+\\xi'))\\|^2$ and then differentiates as if $\\mathbb{E}[x^\\top W^\\top W x]$ behaved like $\\mathbb{E}[x^\\top x]$, ultimately yielding Eq. (35): $W^\\top W = \\mathbb{E}[x^\\top x + x^\\top \\xi' + \\xi^\\top x + \\xi^\\top \\xi']$. This is **dimensionally inconsistent**: the RHS is a scalar, while $W^\\top W$ is a $d\\times d$ matrix. The derivation implicitly \"pulls out\" $W^\\top W$ from inside a squared scalar and replaces it by a scalar expectation. The CL proof with dissimilar augmentations (Eqs. (45)–(47)) suffers from the same issue.\n   \nThese steps are central: they enable the replacement of each loss by $\\|X - W^\\top W\\|_F^2\\$ and the invocation of Eckart–Young to claim the row-span of \\(W\\) equals the top-eigenspace of \\(X\\). Because the stationary conditions are not validly derived, the RV bounds that follow are on shaky ground. (Appendix A.7) \n\n3) Ambiguities and likely typos in the non-IID data model. \n- In the rare class definition, samples for client $i$ are given as $x^{(h_i)} = e_i + \\mu \\xi$ (p. 4). This reuses $e_i$ rather than $e_{h_i}$ (where $h_i\\in[2N]\\setminus\\{2i-1,2i\\}$), effectively collapsing the \"rare\" class onto the client's own axis instead of its designated class. This undermines the intended label heterogeneity.\n- The offset term $\\sum_{k\\neq i} q\\,\\tau e_k$ uses a single Bernoulli $q$ inside a sum, not $q_k$; thus the sum is either 0 or $\\tau\\!\\sum_{k\\neq i} e_k$, which is likely unintended.\n\n4) Definition and use of the Representability Vector (RV) are inconsistent.  \nDef. 4.1 defines $r=[\\|\\Pi_R(e_1)\\|^2,\\dots,\\|\\Pi_R(e_c)\\|^2]$ where $R\\subset\\mathbb{R}^d$ is the row-span and $c=\\text{rows}(W)\\le d$. Projecting only the first $c$ basis vectors is ad-hoc; later theorems index $k$ in ways that mix client and feature indices (e.g., \"where $i\\in[N]\\setminus k$\" in Theorem 4.2.1). The definition of \"sensitivity\" $s=\\lceil\\bar r\\rceil-\\lfloor\\bar r\\rfloor$ also lacks a precise norm (max–min over which coordinates?), yet it drives the main claims.\n\n5) Assumptions needed for concentration are not spelled out. \nThe proofs use matrix concentration to bound $\\|X - \\mathbb{E}X\\|$ \"with probability $\\ge 1-\\frac12 e^{-d^{1/10}}$\" (p. 20) but the number of samples per client, independence across samples/classes, and dependence on $d$ are unspecified beyond \"polynomial in $d$\" / $O(d^\\alpha)$. Without explicit sample sizes, the rates in $d$ (e.g., the $O(d^{-2/5})$ and $\\Theta(d^{7/20})$ terms) are hard to justify and compare. (Appendix A.7)\n\n6) Handling of the augmentation transform $H$.\nFor the \"dissimilar\" CL case, bounds scale with $\\operatorname{tr}(H)$ (Theorem 4.3). But $\\operatorname{tr}(H)$ can be zero or negative (e.g., certain rotations or permutations), which can flip signs in denominators and invalidate inequalities used to argue robustness ordering. The required conditions on $H$ (PSD? orthogonal? non-negative trace?) are not stated.\n\n7) Modeling choices weaken the external validity of the theory. \nThe linearization of encoders/decoders and of augmentations, and the unusual scaling $N=\\Theta(d^{1/20})$, $\\tau=d^{1/5}$, $\\mu=d^{-1/5}$, seem chosen to make algebra convenient, but the conclusions then rely on delicate powers of $d$ (e.g., $d^{2/5}$ vs. $d^{7/20}$). It is unclear how stable the ordering (MIM < CL sensitivity; FL ≥ DecL) is beyond this stylized regime (e.g., different $p$ schedules, non-linear encoders).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)