Modular Federated Contrastive Learning with Twin Normalization for Resource-limited Clients
Authors: Azadeh Motamedi
IL MIN KIM
Venue: Accepted by TMLR ()

Abstract: Despite recent progress in federated learning (FL), the challenge of training a global model across clients, having heterogeneous, class-imbalanced, and unlabeled data, is not fully resolved. Self-supervised learning requires deep and wide networks, and federal training of those networks induces a huge communication/computation burden on the client side. We propose Modular Federated Contrastive Learning (MFCL) by changing the training framework from end-to-end to modular, meaning that instead of federally training the entire network, only the first layers are trained federally through a server, and other layers are trained at another server without any forward/backward passes between servers. We also propose Twin Normalization (TN) to tackle data heterogeneity. Results show that ResNet-18 trained with MFCL(TN) on CIFAR-10 achieves $84.1\%$ accuracy when data is severely heterogeneous while reducing the communication burden and memory footprint compared to end-to-end training. The code will be released upon paper acceptance.

Published in Transactions on Machine Learning Research (11/2024)
Modular Federated Contrastive Learning with Twin Normalization for Resource-limited Clients
Azadeh Motamedi
19am43@queensu.co
Deportment of Electrical and Computer Engineering
Queen's University
I--Mi Kim
Almin. klmin.kim Wxseernsuunccs.cco
Department Electrical and Computer Engineering Queen's University
Reviewed on OpenReview: https: // openreview. net/ per/ffrum? id= Flanywl't 14
Abstract
Despite recent progress in federated learning (FT.), the challenge of training global model across clients, having heterogeneous, class-imbalanced, and unlabeled data, is not fully re- solved. Self-supervised learning requires deep and wide networks, and federal training of those networks induces a huge communication/ computation burden on the client side, We propose Modular Federated Contrastive Learning (MFCL) by changing the training framo- work from end-to-end to modular, meaning that instead of federally training the entire network, only the first. layers trained federally through server, and other layers trained at another server without any forward/ baakrd/dbbwwdrddd passes between servers. We also propose Twin Normalization (TN) to tackle data heterogencity. Results show that ResNet- 18 trained with MFCL(TN) on CIFAR-10 achieves 84.1% % accuracy when data is severely heterogeneous while reducing the communccation burden and memory footprint compared to end-to-end training. The code will be released upon paper acceptance.
1 Introduction
Federated Learning (FL) has been introduced as a framework to learn from the data located at multiple clients without transferring the client's raw data to a central location, called the server McMahan et al. (2016 2017); Konečný et al. (2016a 6a b). In each FL round, each client's local model is trained on its dataset, and the local models' parameters are sent to the server to be aggregated and broadcast. back to the clients. The distribution of local datasets plays vital role in orchestrating the real-world implementations of FL. Simply aggregating the model parameters trained on heterogencous datasets does not improve the global model accuracy unless additional tricks are adopted such as regularivation ion techniques in local objectives of
the clients Luo et al. [2021) 0s Dieuleveut et al. (2021) 9: Mu et al. 2001]; ; Li et al. 2020 2021 Durmus et al. 2021 b; Yu et al. (2020b), adaptive aggregation in the server Wang et al. 2020a) da sharing on the clients or server Zhao et al. )2018); Hao et al. (2021), and data augmentation Shin et al. (0220) to reduce the gap between the local and global models. Mcanwhile, addressing the challenges of data labelling in FL, some self-supervised methods have been proposed considering data heterogeneous clients Lubana al. (2022 Lu
et all (202)); Makhija et al. (2022); Huang et al. (2022); Miao & Koyuneu (2022); Yu et al. (2020a) In
particular, contrastive Chen et al. (2020) and non-contrastive learning Grill et al. (2020): Chen & He (2021)
methods have been applied to FL Zhang et al. (2020b): Zhuang et al. 2202 2022). The existing techniques
are mostly based on training one or even Iwo deep networks al each client, adopling various methods such as moving average update at local models Zhuang et al. (2022), tuning the communication frequency between the clients and server basod on the divergence local and global models Zhuang
(2021), or aggrogating and sharing the clients' representations based on their distance Zhang et al. (2020b) ensure that clients collaboratively learn from the unlabeled data Shi et al. 20021 For detailed review, Appendix A.1.

---

Published in Transactions on Machine Learning Research (11/2024)
3
(a) E2E traditional FL (b) E2E resource compati- (c) E2E resource compatible
(d) Modular resource com- ble FT. (supervised) Horvath FL (supervised) Kim et al. patible FT. (self-snpervised)
Konečný et al. 2016albb b).
et al. 2021)).
(2022): Hong et al. (002)). [ours].
Existing resourco-compatible schemes and the proposed method. The red arrows indicate the forward and backward passes through all layers. The parameters of the bold connections are transmilted to the server in each FL round, as indicated by the blue and green arrows. (a) Clients send all parameters to the server in end-to-end (E2E) FedAvg, (b) Clients send parameters of some filters (a vertical cut) of the model to the server in E2E HeteroFL. (c) Clients send parameters of some layers (a horizontal cut) of the model to the server in F2F Split-Mix. (d) Clients send parameters of some layers (a horizontal cut) of the model to the first server (gServer) in the proposed modular scheme. After the initial layers are trained, the rest of the model is trained on the representations received in the rServer. The gray and pink arrows show the forward and backward passes on the client side and rServer, respectively. The white arrows show the transmission of the representation vectors, which is done only once.
Figure 1:
Typically, self-supervised learming relies on a large amount of data, deep and wide models, and/or large batch sizes to achieve good performance, which means high computing power and/or large memory fool prints are nceded Zhuang et al. (2022). Such resources are often unavailable on clients' devices, especially in cross- device FL scenarios. To address these issues, researchers have proposed different scaling methods for clients to have different-sized local models depending on their available computation resources (Figure 1 In Diao et al. 2020); Horvath al. (202T), the authors suggested dividing the network based on its width and assigning diffcrent number of filters to different clients based on their computational resources (Figure 1b In more recent schemes, the network is divided based on its depth, and different numbers of modules (sub- models) are assigned to different clients Kim et al. (2022); Hong et al. (2022) (Figure Lcp. However, there are mainly three limitations in these resourc--copptiblle architectures. First, in those schemes, only supervisod loss functions are considered. Shallow or narrow networks are typically insufficient for self-supervised tasks because they lack the capacity to learn rich, informative representations from complex or highly diverse data, particularly in scenarios involving heterogeneous or dasss-mbalaaced data. In FL, where data heterogeneity is major challenge, shallow networks are unlikely to extract representations that generalize well across different clients, especially when each client has unique data distributions. Self-supervised learning methods, either contrastive Chen et al. 22200) or non-contrastive Grill et al. (020)). require networks with enough depth and capacity to align representations from diverse data sources effectively.
Second, the performance of those schemes has not been comprehensively studied under severe heterogeneous and class-imbalanced (CIB) data scenarios. If some clients with unique data cannot afford to train a deep and wide sub-model, the global model cannot learn those unseen distributions. The performance degradation might be even worsened when the local data is CIB the number of clients is large (e.g., more than 10) Hong et al. (2022). Third, although those schemes are proposed for resource- resource-imiiedd clients, the computation (and communication) burden on the side of clients is still high Kim et al. (0022). Moreover, clients still need large memory to implement the entire graph to backpropagate the gradients from the last layer to the first layer of the whole network, which is needed for any type of end-to-end (E2E) learning.
Although E2E learning performs well on numerous tasks, in some cases (e.g., ill-conditioned problems), E2E learning might converge slowly, converge into possibly local optima, face vanishing gradients Shalev-Shwartz
2

---

Published in Transactions on Machine Learning Research (11/2024)
et al. 2017], be expensive/diffccut to implement due to its scrial nature. As alternative approach to F2F learning, modular learning has recently been studied Pfeiffer et al. (0023) Belilovsky et al. (2019); Wang
et al. (2021b), in which a model is split into multiple modules, which are trained on distinct loss functions and the gradients of each module are not necessarily backpropagated to the other modules. Existing modular learning methods can be calegorized into two groups from the perspective of gradient backpropagation (BP) (i) Brain-inspired BP-free approach Mllng et al. (2021) that trains the network based on Hebbian rules and (ii) BP-limited approach Löwe et al. (2019) that limits BP to few modules using various techniques.
In this paper, inspired by modular learning, and considering the restrictions of communication, computa- tion and memory resources on the side of clients for adopting self-supervised learning, we propose new and effective FL method, namely Modular Federated Contrastive Learning (MFCL). In MFCL, instead of federally training an entire model across the clients, the model is split (at a cut-off layer) into two modules that are separately trained (Figure ldp. The layers up to the cut-off layer are called the client module. This module is foderally trained across clients through the gradients server or the gServer. The remaining layers are called the representations server module the rServer module. The rServer module is trained at the rServer The data representations are extracted from the clients' unlabeled data by the federally trained clients' module and transmitted to the rServer. The rServer module is trained based on self-supervised con- trastive learning on the representations received from the clients at the rServer. In MFCL, the two modules are trained separately (through two distinct servers) without any aawaad/bbackwar passes between them, which is essentially the second type of modular learning (i.e., the BP-limited approach).
To address the issue of data heterogeneity in FL, we propose a new normalization method, namely Twin Normalization (TN). Batch Normalization (BN) fails when the mini-batch data is CIB Wu & He (2018))
The performance of BN can further deteriorate in FL under severe heterogeneous data Diao et al. (2020). TN, unlike BN, is a comple-based normalization which normalizes two augmented views or two eramples of one sample and doesn't rely on mini-batches size, which makes it more efficient in heterogeneous CIB data, Specifically, TN is designed to avoid some the key drawbacks of BN in the context of FL, especially in heterogeneous and CIB scenarios. Similar to Group Normalization (GN) Wu He (2018), TN normalizes across group of channels, making it robust in scenarios with small mini-batches. However, compared to GN, TN leverages more diverse information within each group of channels, as it normalizes the channels of two augmented views of the same sample together. This results in more accurate mean statistics for those groups, enhancing the robustness of the normalization. TN effectively captures both inter-example and intra-example relationships, improving model performance. Our contributions are as follows:
Considering the resource constraints of clients, we propose modular framework to perform con- trastive learning the server. MFCL reduces communication overhead and allows the rServer to conduct contrastive learning more balancal (CB) data, improving the model's generaliza- tion.
The proposed MFCL addresses the issue of limited memory and computing resources at the client devices by shifting the eesource-intensvv contrastive learning to the Server. This makes the system more feasible for real-world deployment on resouree-constranne clients.
We propose TN, for contrastive learning in FL, which performs better than BN and Group Normal- ization (GN) in severe data heterogeneous scenarios.
Through experiments, we demonstrate the effectiveness of the proposed MFCL, especially with TN, which achieves robust, stable, and state-of-the-rt performance on severe heterogeneous and CIB data while only a small-size client module is trained federally across clients.
IIn FL literature, leveraging two (or more) servers has been demonstrated to be useful for addressing (j) data heterogeneity
Caldarola al. (2021 Ghosh et al. (020), (ii) data privacy Thapa
2022 Li ct al. 2022 (ii)) communication and
computation efficiency Khan al. [2021] In MFCL,, adopting two servers ensures data privacy, which will be later discussed.
(2020ah: Waag et al. (201)) and (Iv) personalization Mansour et 2020
3

---

Published in Transactions on Machine Learning Research (11/2024)
Contrmtte Client Represrntation Mini-buich
rServer
Recosstracted
Aegroonadine Is
Augacutation
Modello
Decoder
B
Discriminative Medule
[00 In
Aupaestation 2,
Augmentation
gm
Mxx
Client Input Mini-batch
(a)
gServer
0 f
ID"
Contrastive ršerver Mini-batch
Σ expe
Dm'
MENT
e
0
ffed
of
ofed
ffed
Representation vector, zij
Representation vector, zij
Representation vecior, 2
1oo f Totl,
0 ( 11oF1.
gl fiot
fiurle
f (ur]e
fiorle
Uniabelled Clirat D
Untabellled Client 2D
Unlabelled Clicat M, DM
Unlabelled data set Client 2
Unlabelled data set Client 232
Uniabelled data set Cleent O,
(b)
Figure 2: (a) Client module and decoder. Client creates two augmented versions X7 j 1,2 of xf" E Dm
and constructs contrastive client input mini-batches nmn x x x1,2. .., k.2 (b) Training
phase 1: Client module//decoders are trained federally across clients through the gServer. (c) Training
phase 2: Clients generate representations 2 f(or). ( = 1,2 from the trained client module and send
them to rerverr The rServer constructs B from the received representations and updates ils parameters x by minimizing contrastive loss.
2 Proposed Modular Federated Contrastive Learning
In the literature, there are a few works on contrastive serming--ased FL, and in those works, the contrastive loss is optimized at each client device. However, considering the memory and computing resource limitations of the clients, might. be difficult or, times, even impossible to perform contrastive learning at the clients Li ct al.] (2014), bocause the contrastive loss typically benefits from the larger model, larger batch size, and/or longer training time Chen et al. (2020); Zhuang el al. (2022). Besides, directly optimizing the contrastive loss across clients as in the traditional FL (i.e., averaging all parameters of clients' local models trained with contrastive loss) may filter out the subtle yet useful information of the difference between the weights corresponding to the augmented versions of data, which is important in determining the similarities of examples. To address these issues, in our proposed MFCL, the contrastive loss is optimized at the rServer. Specifically, instead of federally training the entire deep network across clients on a contrastive loss, only a part of the model is trained by a contrastive loss at the rServer on data ropresentations received from the clients' unlabeled data extracted by the client. module. The client module at. the gServer and the rServer module the rServer are trained separately without any forward/backwart passes between them
2.1 Training Procedure for the Proposed MFCL
Training Phase 1 for Client Modules: During the first stage of training, the client modules are federally trained over M clients with coordination of the gSeveer The client module, denoted by (-) consists of the first several layers up to a cut-off layer of the entire model. Here, [0Th denotes the parameters of the client module at client at FL round t. For unsupervised (federated) training of the client module, we 2In this work, we assume the two servers do not communicate (or, do collude).
4

---

Published in Transactions on Machine Learning Research (11/2024)
introduce the decoder (+)r a mirror structure of the client module, with a proper loss (e.g., cross- entropy or mean squared error (MSE)); as shown in Figure a()) Here, [05))a denotes the parameters of the decoder at client m at FL round + For normalizing the layers' outputs of the client module and its decoder, one could use the widely adopted BN and it would work if the clients' data were homogencous and CB. However, in practical FI, the clients' data are heterogeneous /CBB and thus, BN often does not work well. To address this issue, given the special structure of the contrastive mini-batch at the clients, we propose using a TN layer after each convolutional layer in the client module and after each transposed convolution layer in the decoder. The details about TN are provided in the next section. The architectures of the client module and its decoder for R8sNet-18 and ResNet-50 are described respectively, in Tables 9 and 10 in the Appendix.
At the beginning of the first FL round, the gServer sends the initial parameters to a set of selected clients. These clients train their modules on their datasets. Specifically, at the end of FI. round t- 1, the selected clients send the updated parameters, 1- m E TT- to the gServer, and the gServer aggregates the pa- rameters Ofed
Σ Ewer, D Emer_n 1 00- In FL round 1, the gServer broadcasts the aggregated
parameters, 0f-11 to a set Tr of clients. Each client m E ne locally updates the received averaged parameters ffed to 0 with the following objective:
min Exn Dm Looms 9f
(1)
where Loss (:,-) is the loss function to optimize 0.nh and (-) denotes data augmentation as follows. For 1, of x." E Dm training the client modules/ pondules/decoders, client m first creates two augmented versions x 1
where xx" denotes the i-th data sample of the local dataset D of client m. Combining the augmented versions of K data samples, client m constructs the contrastive client imput mini-batches of size 2K denoted
by DT = ( -(T,] (x K. xT,2........ xx,2)
Training Phase 2 for the rServer Module: The second stage begins when federated training of client modules and their decoders is finished, say at FL round T. Then, the decoder is discarded and, supplying two augmented versions J 1,2, of cach data sample xx" D to the trained client module, client produces representation vectors, = Sori, (X]). 1,2, called the contrastive client representation mini- batches, and transmits them to the rServer. Combining the representations received from at least Q S. M clients, the rServer constructs the contrastive rServer mini-batches B", such that each mini-batch 8 is large enough for contrastive learning. In general, the rServer representations are much more CB, than the data at each client (see Figure G in the Appendix). At the rServer, the rServer module, Mxx: parameterized by x"; is trained by minimizing the contrastive loss on contrastive rServer mini-batches, 1 (see Appendix A.2). Figures bb) and (cc) show the two training phases of MFCL, respectively.
Note that MFCL, is designed to result in single global model. To that end, when the training is done, the client module and the rServer module form single unified global model. To deploy this unified model, the (finally trained) rServer module should be broadcast to the clients. Algorithm I Toutlines the detailed process of the MFCL framework.
3 Normalization for the Client Module DDebodr
Without any normalization, CIB data in the mini-batch can increase the variance of the mini-batch gradients. Therefore, the output distribution of layers may diverge, which leads to lower accuracy Wu & He (2018). In the following, we first briefly discuss BN and GN. We then propose our normalization method, TN.
For simplicity, the model parameters of the whole autoencoder (i.c., the client module and it's decoder) are denoted by 0m". (0 and [0 represent the model parameters of the client module and decoder parts of 0Tr, respectively. WW used Avg McMahan al. (0166) to aggregate the clients" module parameters at the gServer. Other hyperparameters and augmentation techniques are detailed Tables 12 and 13 in the Appendix.
5

---

Published in Transactions on Machine Learning Research (11/2024)
3.1 Batch Normalization (BN)
The strong point of BN is that each channel is normalized over the entire mini-batch lolle & Szegedy (2015). BN works well in contrastive learning when each contrastive mini- batch is large and CB (this is the case for the contrastive mini-batch at the rServer). However, when the mini-batch size is smaller and/or the data of each mini-batch. is CIB, the batch statistics estimated by BN are biased and inaccurate. Therefore, using BN for training the client modules/decolee on the heterogeneous and CIB data FT. leads to biased models, which are unable to accurately learn the features of those classes having only small number of samples.
3.2 Group Normalization (GN)
The dependence of the batch-normalized outputs of cach layer on the entire mini-batch makes BN powerful: however, it is also the source of BN drawbacks. GN, proposed for distributed training, normalizes a group of channels of sample together without utilizing the entire mini-batch WW & He (2018). Specifically, GN randomly chooses group of A < C' channels, which may have different scales, and normalizes them together. GN is applied to FL to address the issue of heterogencous data across clients and works better than BN Zhang et. al. 2021); ; Yu et al. (2021 2020b). The reason is that in FL, the model parameters (e-g., the filters' parameters in convolutional neural networks) are averaged over multiple clients. Averaging makes the scales of filters smoother. Applying smoother filters on channels leads to smoother features and normalizing sample over a group of its channels, in GN, becomes meaningful. One limitation of GN is that it relies limited information to normalize each sample over a group of A channels, compromising the precision in estimating the statistics of each sample to make the statistics independent of the mini-batch size.
3.3 Twin Normalization (TN)
We propose TN as a solution to address the lack of sufficient information in GN for accurately estimating the statistics of a sample across a group of its features. Since in contrastive learning, there are two augmented views of each sample, it is possible to benefit from the rich and diverse information of those two examples and normalize them together. TN normalizes positive examples together, which increases their similarity and further differentiates them from negative ones. For mathematical formulation, we let w f= 1...., K;j 1, 2.2:c c 1,..., Ci denote the attributes of channel at layer 1 for the j-th augmented version, x 1,2, of the i-th imput data sample, x,", in the client's 's mini-batch. We first determine the mean μ 4nd and variance (a ; 2y of A channels corresponding augmented versions of cach input the I-th layer as follows:
Wij
adl
and (a 20
1
c.l I
14 1112
(2)
2A
2A
J=1 cε
J=1
cE
where || denotes the le norm and 4 C {l...., Ci} denotes the set composed of A channels at layer L.
Then, we perform the normalization as follows:
w 0.m w ui tnd
c.l
+ 0mm!, for E t
(3)
where w is normalized w for c E Di. Also, 0. and 0 denote trainable scaling factor and shift, respectively, for layer of the client's 's module and decoder at FI. round t.
TN ensures that the attributes of examples are normalized independently of other examples in the client's 's contrastive mini-batch. The pros of adopting TN are follows: (1) Unlike BN, TN is not sensitive to mini-batch size, making it more suitable for FL where mini-batches can be small and vary across clients. (2) TN's ability to normalize augmented views together enhances the similarity between positive examples and separates them from negative examples, which is crucial in contrastive learning. This improves the generalization of models trained in FL with heterogeneous CIB data.
Although TN works well with heterogeneous CIB data, BN still works better than TN when the data is homogeneous and balanced. Therefore, for training the reever module at the server, we still benefit
6

---

Published in Transactions on Machine Learning Research (11/2024)
Xx
K
K
GN
(Proposee)
Figure 3: Normalization techniques for the client 2K, represents the elements of cach contrastive client mini-batch. The y-axis, denoted by C, represents the channels and the z-axis, denoted by (X, Y), represents the spatial axes of images' pixels. Only the misaligned blocks with black tops are simultaneously normalized together with the same mean and variance.
decoder of each client. The x-axis, denoted by
from BN loffe & Szegedy (2015). This is because, compared to each client's contrastive input/repreentt /repeeeerrennnoonnnn mini-batches, the rServer's 8 contrastive representation mini-batches can be constructed as large as needed and much more resources are typically available at the rServer. Thus, the contrastive rServer mini-batches are much better CB and large enough to warrant. BN.
3.4 The comparison between BN, GN, and TN
With some inspiration from BN and GN, TN is uniquely adapted for federated contrastive lcarning. Specif- ically, BN relies on the whole mini-batch, and the size of the mini-batch plays an important role in the success of BN. TN is designed to avoid some of the key drawbacks of BN in the context of FL, especially in heterogeneous and CIB scenarios. TN, unlike BN, is a sample-based normalization which normalizes two ex- amples of one sample and doesn't rely on mini-batches size, which makes more efficient in class-imbllanced and heterogeneous data settings. GN normalizes across group of channels, making it robust in scenarios with small mini-batches. Compared to GN, TN has more diverse information per group of channels TN normalizes the group of channels of two augmented views of the same sample logether, making the mean of those groups more accurate. TN effectively captures both inter-ezample and intra-ezaman relationships, improving model performance. In neither BN nor GN, two examples of one sample are normalized together as in TN. Figure 3 shows the differences among BN, GN, and TN.
In summary, while BN struggles with heterogeneous data due to its reliance on batch-wide statistics, and GN, though more robust, lacks inter-example normalization, TN combines the strengths of both. TN retains GN's independence from mini-batch size while having inter-example (not inter-sample) relationships, in a controlled manner by focusing on augmented pairs. This makes TN particularly well-suited for federated contrastive learning in heterogeneous enviromments, where it improves model performance without suffering from the drawbacks of BN and GN
4 MFCL vs Federated Split Learning
4.1 Methodological Comparison
The proposed MFCL might appear to be similar to Federated Split Learning (FSL) Thapa et al. 2022 in that we also split the network at a cut-off layer into the client module and the rServer module. However, MFCL and FSL differ significantly in major aspects. First, in FSL, the smashed data (i.e., the output of the cut-off layer) must be communicated between the clients and server as many times as the number of FL rounds. In contrast, in MFCL, the data representations are transmitted from the clients to the rServer only a single time, after the clients' module and docoder are trained. Therefore, MFCL imposos a significantly lower data traffic burden compared to FSL. Second, unlike FSL which backpropagates the gradients of the smashed data from the server to the clients during the training, MFCL trains two separated modules without
Note that while TN particularly beneficial in federated contrastive learning, is not necessarily limited to this context. TN could extended any learning tasks that benefit from augmented data samples and the data CIB.
7

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: The accuracy performance of MFCL with various cut-off layers for ResNet-18 on CIEAR-10 with
M 10 clients, and ResNet-50 on Imageent with M 100 clients (5 0.1).
3
4
5
6
7
ResNet-18 (on CIFAR-10) ResNet-50 (on Tiny-ImageNell
59.1
84.5
82.4
74.3
61.5
52.3
47.6
25.8
30.6
32.7
33.9
33.7
32.2
28.5
any gradient/weigh transactions. Last, but not least, MFCL can be more advantageous than FSL in terms of data privacy, which is discussed in what follows.
4.2 Data Privacy Comparison
In MFCL, only the representations of the harshly augmented data are transmitted (disclosed) only to the rServer. This means that the gServer has access only to the parameters of the client modules poodules/decoderss not the representations produced by the client modules, whereas the rServer has access only to the data representations, not the paramcters the clients' modules coddules/dcodderss decoders. In FSL, the server has access only to a part of the model and this setup provides a certain level of data privacy Wang et al. (2023b); Turina et al. (2020); Jeon Kim (2020); Thapa et al. (2021); ): [L et al., (2022). Comparing MFCL with FST. MFCL, distinct training phases on two distinct servers. On
offers more enhanced data privacy than FSL due top of that, in MFCL, the data representations are sent to the rServer only single time. In contrast, FSL entails the transmission of smashed data in every FL round throughout the training process, which makes it and gServer do not have
more vulnerable to data leakage. Morcover, in MFCL, the client module/decode any gradient transactions with the rServer, which has access to the data representatioe ions, ensuring that the rServer does not have the gradients of representations. By contrast, in FSL, the server has smashed data as well as the corresponding gradients, making the whole system even more vulnerable. MFCL bolsters data privacy by reducing the risks associated with data leakage.
5
Experiments
5.1 Setup and Performance Metric
We perform our experiments on CIFAR-10, CIFAR-100 Krizhevsky 2009), and ImageeeeeNett/ Le & Yang (2015) with ResNet-18 and NesNet.50. For ResNet-18, 18, the first two layers, and for ResNet-50,, the first 4 layers are chosen as the client module (i.e., the encoder part) and we design the decoder part as a mirror structure of the encoder. Table [ presents the results of choosing the cut-off layer al different locations for the two networks. In both cases, the client module and its decoder are composed of convolutional layers and TN layers (See Tables 9 and 10 in the Appendix for more details). The rServer module is composed of the remaining layers of the network, which is trained the received representations with contrastive loss.
We implemented MFCL with low 2.14 following the standard structure of FL McMahan et al. 2017)) and the contrastive learning Chen et al. 2020). To construct heterogeneous and CIB datasets over the clients, we used Dirichlet distribution, with concentration parameter (5 > 0) Yurochkin et al. (2019); Wang et al. (2020b). Tf g set to a smaller value, the data becomes more heterogeneous and CTB, whereas when E - 0o, the data becomes homogeneous and CB (See Figure 7 in the Appendix). We used a single NVIDIA GeForce RTX 3090 GPU to simulate the clients and rServer modules. We evaluated the classification accuracy of MFCL based on the linear evaluation method Chen et al. (2020). Specifically, layers as described in Table in the Appendix on top of the rServer module, and one lincar Dense layer on top of the projection layers for linear evaluation. In training phase 2, the rServer module, projection layers, and linear (classifier) layer can be jointly trained as long as the gradients are not backpropagated from the lincar (classifier) layers to the rServer module and the projection layers. Alternatively, the rServer module and the projection layers could be trained first, and then, the Dense layer on top of the frozen (trained) Server module and projection layers can be fine-tuned using labelled data.
used non-linear projection
8

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Accuracy comparison MFCL and baselines on CIFAR-10 with = 10 on ResNet-18, CIFAR-100 with M 100 on ResNet-18, and Tiny-Image Iny-ImmgeNet with M 100 on ResNet-50. The best performance is highlighted in boldface. For SSFL methods, the top-1 accuracy with linear evaluation is reported. For MFCL, BN is used at the rServer.
CIFAR-10 10
CIFAR-100
0.01
0.1
200
0.01
0.1
200
0.01
0.1
200
Method
FedAvg
62.3=0.62
92.1 79.2
36.7
48.2
60.5
29.4
36.6
49.5
FedProx
70.8
83.0
93.2
36.9
48.740.21
67.0=0.17
30.1+o.1
36.9 20.25
50.20
MOON
25.4
77.720.31
94.110
37.210
67.620.11 49.1
16
30.720.20
36.720.14
50.620.18
FSL
54.1
66.720.31
77.020.10
30.51o.18
29.4
37.620.0s
19.8
25.420.28
31.0 31.020.211
FeR
25.620.2a
72.110.11
81.910.07
32.410.25
35.010:1
42.320.14
21.720.31
24.320-28
33.810.95
Orchestra
79.62n.0s
82.1
80.7
37.420.13
10.320.09
10.920.07
26.220.07
28.820.08
29.71:0.10
FedEMA
78.8=0.16
81.9+0.09
00.1
37.2+0.00
40.3+0.07
37.6#0.04
27.0+
2
38,9+0.15
FedU
74.7=0.38
77.6#0.30
13.2
32.1
36.7#0.10
39.8=a.13
24.6#0.1s
26.3
30.1
FealARC
83.3=0.34
84.6=0.19
86.7
42.5+0.19
54.610.19
67.320.08
31.62o.31
32.9=0:24
37.5ta
(BCL(BN)
15.3=0.41 32.5+a.s
86.4+0.12
14.1
28.3+0.38
51.3=0..1
12.9
22. 20.22
29.5+t.11
MFCL(GN)
79.320.14
79.8
82.64o.11
41.510.m
43.220.16
20.12
29.64o.18
30.420.12
38.31o.1
MFCL(TN)
84.1 84.6g0.21
85.3
44.420.17
45.520.21
47.820.13
33.1
33.920.14
42.220.18
5.2 Baselines
We consider the state-of-the-art works in self-supervised FL (SSFL) and supervised FL (SFL) that are pro- posed to address data heterogencity. In the field of SSFL, we compare MFCL with FedU Zhuang al. (2021) and FedEMA Zhhuag et al. (2022) SimCLR Chen al. (2020) with FedAvg another baseline, denoted FedSimCLR, We also consider Feat Wang ct al. (2023a), recent approach in decentralized lcarning that aims to address the challenge of learning from decentralized, heterogeneous data using contrastive learning,
FedAvg McMahan et al. (2016), FedProx Li et al. (2020), and MOON Li et al. (2021) are fully supervised FL. FSL Han et al. (2021] is our baseline for federated split learning.
There are two main baselines for implementing SFL in resource-limited clients: HeteroFL Diao el al. (2020) and SpXIt-MIX Hong et al. (2022). Those baselines and the majority of current resource-compatible schemes are in the SFL field, where the number of FL rounds and the computation burden of local updates are completely different from SSFL schemes. For fair comparison, we report the commuicatiin burden, computation burden, and memory footprint from those papers.
5.3 Accuracy Performance
We train CIFAR-10 over M = 10 clients on ResNet-18, CIFAR-100 over M = 100 clients ResNet-18, and Imag-lmageNet over M = 100 clients on ResNet-50. We consider three heterogeneous and CIB scenarios (g = 0.1 and 0.01), and a homogeneous and CB scenario (g = 200). Each scenario is run 3 times, and the mean and standard deviation of the top-1 accuracy on a uniform test set is reported
Improved accuracy in heterogeneous and CIB scenarios. From the numerical results in Table 2 one can that, in severe heterogeneous and CIB datasets, MFCL.(N) significantly outperforms other SFL and SSFL methods. Furthermore, MFCL(TN) outperforms MFCL(BN) and MFCL(GN) in those scenarios. The first reason is that MFCL. benefits from self-supervised learning, which is mathematically and experimentally proved to be more robust to CIB data Liu et al. (2021). In supervised learning, the expected values of the squares of the lengths of the gradient vectors corresponding to different classes are approximately related to the square of the number of samples those classes Anand et al. (1993); Yu el al. (2020c). This means that in supervised learning when the dataset is CIB, the gradients of the classes with more samples are larger
oTe code for these baselines is not publicly available. We implemented them with the parameters reported in their papers. Unless otherwise specified, we assume that all clients participate in rounds (l.e., Q = M).

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Communication burden a, computational complexity 8, and memory y in FedEMA and MFCL.
ResNet-18 trained on CIFAR-10 with M 10 and 5 = 0.1. ResNet-50 trained on Tiny-ImageNet with M 100 and 0.1, Results are reported per client.
ResNet-18
#bits (pl) FL rounds Rep. size #bits (p2)
(bits)
8 (FLOPs)
7 (Byte]
F FedSimCLR
21x
100
21x
72x10
63x10
21x10*
800
0
17x10"
69x10'²
31 31x10
MFCL
11x10"
15
26 10
25x 10
25x10'0
53x10"
11x107
ResNet-50
#bits (p1) FL rounds Rep. size # bbtts (p2)
a (bits)
3 (FLOPs)
Y (Byte)
FedEMA
45x 45x10
100
0
44x 10
33x10'a
17x10"
FedSimCLR
45x
800
0
36x10
20x
62x10
MFCL
23*10"
15
52x10
50x 50x10'
10
70x10"
23x10
'p', #, 'Rep'' stand for 'phase', 'Number of', and "Representations; respectively.
than those of the classes with fewer samples, which leads the gradients being biased toward the class with more samples. In contrast to this, self-supervised learning does not rely on the class labels to calculate the loss, and thus, the gradients are less biased, and the model can better learn the classes with fewer samples. Therefore, with self-supervised learning, we can mitigate the bias of labels.
Second, MFCL benefits from TN. severe heterogeneous and CIB scenarios, addition to the bias of labels, we face what we call the bias of features. The bias of labels can be mitigated by new designs of the network (e.g., MFCL instead of F2E learning). However, the bias of features is the natural result of having clients with heterogeneous CIB data. One effective way to moderate the bias of features is to use properly designed normalization techniques. TN mitigates the bias of features when data is heterogeneous and CIB
Accuracy in homogencous and CB scenarios. MFCL works best when datasets are heterogeneous and CIB, which is a practical scenario. However, in the homogeneous and CB scenarios, SFL methods outperform SSFL and MFCL. The reasons are as follows. First, when clients have homogeneous CB data, the labels are homogeneous, and the gradients are not biased. In those scenarios, SFL works better than SSFL Wul & He 20188. Second, when the mini-batch is CB and large cnough, normalizing the data over the entire mini-batch (i.e., BN) works better than the sample-based normalization (e.g., GN or TN). The reason is that the batch statistics estimated from a large CB mini- batch are more accurate, yielding smaller variations of statistics from one mini-batch to another, and thus, it leads to better generalization performance.
5.4 Client Side Resource Saving by MFCL
Communication burden. In FL, the complexity of communication channels refers to the data traffic and the frequency of communication. In the following, we compare the communication complexities of MFCL and the traditional E2E SSFL methods in both aspects.
Data traffic: Traditional E2F. SSFL methods require sending a huge number of model parameters to the server in each FL round. In MFCL, clients only train a shallow module federally to learn the low-level features, which requires only a few FL rounds. Therefore, the data traffic of training phase in MFCL is far lighter than that of the E2E SSFL methods. Meanwhile, MFCL has additional data traffic in training phase 2, which does not exist in the traditional E2E SSFL settings. Based the simulations, considering realistic scenario (e.g,, using single precision format (32 bits) and adopting error correction code with code rate 3. which is used in all real-world communication systems) to transmit the model parameters in training phase 1 and representation vectors in training phase 2, we found that the overall data traffic of MFCL is lower than the corresponding E2E mothods. Table 3 compares the communication burdon of MFCL with that of Fed In training phase 1, the communication burden, dr, of MFCL is computed by
8Te advantage of also demonstrated by I-S visualization of the learned clusters, shown in Figure g in the Appendix.
10

---

Published in Transactions on Machine Learning Research (11/2024)
on (bits) = (10 + +000)) x 32 Rx
(4)
where 10(=d), |05h, R= 1 and T denote the number of model parameters in the downlink, number of model parameters in the uplink, channel code rate, and number of FL rounds, respectively. In training phase 2, the communication burden, 02, of MFCL is computed by:
3:
02 (bits) - ((2 x p) X 32 x R.
(5)
where, |2n|| is the size of each representation vector from client m and p is the number of representation vectors. The size of one representation vector depends on the kernel size and stride of the last layer of the client module, and the number of data samples in the clients. For example, in ResNet-18, the size of one representation vector is | = 32 x 32 x 64 x 1 where the stride is 1 and the number of filters is 64. For CI over M = 10 clients, | = 5000; and for Tiny-ImageNet over M = 100 clients, | = 1000. Each representation vector contains 2 different augmented versions of each data sample.
MFCL, each client sends p = representation vectors to the rServer (i.e., 16 different augmented samples for each data sample). The total communication burden for MFCL is a æ1 + a2 which is reported in Table 3 Clearly, with more data in clients, the data traffic of MFCL would increase lincarly. However, more data might need larger models in E2E SSFL, which leads to higher data traflic for FedSimCLR
Frequency of communications: Each communication event incurs overhead costs, including channel setup, data ancoding/doooooo decoding, handoff management Kukliński et al. (2014), and session management Hasan et al. 2019). High-frequent communication increases latency and network load, polentially leading lo delays and congestion, especially bandwidth-limitell or high-latency networks like mobile networks. Frequent communication also drains the battery life of mobile devices and raises operational costs for servers and infrastructure Mao et al. (2017). As reported in Table 3 each client utilizing MFCL uses the communication channels 16 times (i.e., 15 rounds plus one time to send the representation vectors), which is approximately 6 times fewer than FedEMA and 50 times fewer than FedSimCLR. This demonstrates that MFCL significantly oulperforms other E2E SSFL methods in terms of communication burden.
Memory and computational efficiency. On the side of clients, the memory footprint to store model parameters, fcature maps, and gradients, an important issuc. The memory footprint y of cach client is computed by
7 (Byles)
|00m + 1gggg + Σ |
x 2K x 4,
(6)
where 10"), |g" 2K, and |mr" are the number of model parameters, the number of gradients, the size of D", and the size of feature maps of different layers of the network, respectively. The feature map size in each layer depends on the kernel size, the stride and the number of filters in the layer. As shown in Table 3 the memory footprint of MFCL the side of clients is much lower than the FedSimCLR. MFCL also has lower complexity in terms of FLOPs, which makes it suitable for resource-limited clients.
MFCL and other resource-compatiin computation burden 8, and memory footprint y are reported for MFCL(TN) and other schemes designed for resource-limited clients in SFL. FedPAC Xu et al. (2023) is another promising baseline to reduce the communication burden and address the data heterogeneity. Instead of updating all clients' classifiers si- multaneously, FedPAC shares only classifiers that are relevant to given client, minimizing communication overhead. However, FedPAC requires more local updates, which increases the computational burden and memory footprint on the client side, making it less suitable for resource-limited clients.
baselines. In Table 4 the accuracy, communication burden a,
One can see that MFCL(TN) has better accuracy in severe heterogeneous data scenarios while reducing the client memory footprint, with an increase in communication burden.
2 Another important aspect for MFCL is that not all clients who participated in training phase necossarily participate in training phase 2. Figure in the Appcndix shows that MFCL is robustly working even in that situation. M0bole devices may move between different wireless network coverages, called the cells, requiring handoffs that involve signaling and maintaining session contimuity. Frequent communication increases the number of handoffs.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Table 4: a, 3, and y in MFCL(TN) and resource com- patible baselines on ResNet-18 18 (M 10).
Table
TN FedSinCLR with ResNet-18 (CIFAR-10, M-10).
Method
ACC
a(bits) 3(FLOPs(Byto)
5 = 0.01
= 0.1
f=0.01
2=20
FedSimCLR MPCL FedSimCL.R MFCL
HeteroFL.
49.3
88.6
15x10
21x10"1 55x 10
BN
65.6
15.3
72.1
32.5
Split-Mix
50.7
88.2
45x10"
27x10º 70x107
GN
68.1
79.3
74.7
79.8
FedPAC
61.7
80.3
12x1010
44x10" 18x 10t0
TN
72.3
84.1
78.2
84.5
MFCL(TN)
84.1
84.5
25x10'0
53x109
11x107
0
10
10
10
10
20
20
20
20
30
30
30
30
10
20
30
0
10 20
30
10
20
30
10
(a)
(b)
(c)
(d)
(e)
Figure 4: Visualization of features learned by MFCL(BN) and MFCL(TN) with clients. Client 4950 airplanes, 50 cars. Client 2: 4950 cars, 50 airplanes. (a) The synthesized test image with the left showing car and the right showing an airplane. The saliency map of gradients for (b) client I with MFCL(BN), (c)
client 2 with MFCL(BN), (d) client 1 with MFC), and (e) client 2 with MFCI(TN)..
5.5 More Simulation Results for TN
Addressing data heterogeneity with TN. Figure 1 shows that TN has better performance over BN in learning classes with smaller number of samples. Let us consider clients each with 5000 samples. Client has 4950 'airplanes' and 50 'cars'. Client 2 has 4950 'cars' and 50 'airplancs'. MFCL(BN) and MFCL(TN) are used lo train ResNet-18 on those clients. When training done, we use the synthesized samples see the gradients' saliency map. Figure 1a depicts a synthesized sample with half showing car and half showing an airplane. Figures 4b and 4c show the gradients' saliency map for MFCL(BN) on clients and 2, respectively. For client 1, 'cars' are rare and the network does not learn good features for them, while client 2 struggles with rare 'airplanes'. Figures 4d and 4e show the gradients' saliency map for MFCL(TN) on the same clients. Clearly, MFCL(TN) has learned better features for both classes.
TN in other SSFL methods. To see if TN is still effective in other contrastive FL settings, we applied TN (instead of BN) to FedSimCLR after the first two Conv2D layers and results are presented Table One can see that (i) TN can improve the performance of FedSimCLR and (ii) MFCL(TN) outperforms FedSimCLR (with BN, GN, TN) when clients have heterogeneous and CIB data.
One of the advantages of MFCL is that the batch size in the client module does not need to be large. The effects of other hyperparameters related to TN, such as batch size and number of channels per group, A, are detailed in Tables 16 and 17 in the Appendix.
5.6 Extra Privacy
The general assumption for preserving privacy FL that the clients are honest, and the server honest but curious Nasr et al. (2018); Le al. (2023). MFCL has two servers: gServer for model parameter transactions and rServer for training on the received representations.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Table 6: The accuracy performance of MFCL with additional privacy provided by SA, DM and both SA and DM. ResNet-18 is trained on CIFAR-10 with M = 10 clients and { = 0.01.
Table 7: Accuracy comparison between FedEMA and MFCL(TN) in the data setting of FodEMA Zhuang et al. (022) with M 5. For CIFAR-10, each client has 2 classes. For CIFAR-100, each client has 20 classes.
SA DM SA+DM No Encryption
CIFAR-10
CIFAR-100
FedEMA MFCL.(TN)
83.3
61.7
MFCL(TN) 84.1 82.4
82.4
84.1
85.6
61.9
5.6.1 Extra privacy-prece preserving mechanisms in training phase and 2
algo- rithms such as Differential Privacy Abbad al. (2016), Secure Aggregation (SA) Bonawitz et al. (2017), or Homomorphic Encryption Zhang al. (2020a) can be applied to the model parameters. We adopted SA in Table 0 to avoid extra computation burden in clients.
Training phase 1. Training phase 1 of MFCL follows the traditional FL. Any
Training phase 2. Since rServer does not have access to the client module's model parameters, recon- structing data from harshly augmented samples using a random decoder is challenging. We also provide extra privacy by applying DataMix Liu et al. (2020b) to MFCL, and the results are listed in Table G
5.6.2 Possible Attack Scenarios
We analyzed different scenarios, considering whether the gServer implements SA, whether the rServer applies DM, and whether the two servers are colluding. SA is adopted to preserve the privacy of the client's 's data at the gServer. Note that if two servers collude, SA cannot provide extra privacy at the rServer 12 The goal is to reconstruct the original sample x from the representations of augmented data with or without DataMix. The simulation results are shown in Table ??.
Case 1: C1D0 (Servers collude and there is no DataMix) In this scenario, the gServer transmits the averaged gradients to the rServer. The rServer, having access to both the gradients and the representations of the augmented data, is capable of reconstructing the augmented data samples. However, more complex challenge lies in reconstructing the original data samples from the augmented data, without direct accoss to the original data or labels necessary to train a network for such reconstruction.
Case 2: C1D1 (Servers collude and DataMix is applied at clients' side) This case is similar to case 1, except that the clients use DataMix. One can see the reconstruction is oven more challenging.
Case 3: CODO (Servers do not collude and there is DataMix) In this case, the rServer does not have the gradients. Therefore the decoding process to reconstruct the augmented data from the representa- tions of the augmented data should start from pure noisy gradients. Again, the more challenging step is to reconstruct the main data from the augmented data, without having access to data samples or the labels.
Case 4: C0D1 (Servers do not collude and DataMix is applied at clients' side) This case similar to the case 3, except that the clients use DataMix. One can see the reconstruction not possible.
5.7 Detailed Comparison with FedEMA
From Table 2 one can see MFCL(TN) works better than FedEMA Zhuang et al. (2022) which is the state- of-the-art baseline for SSFL. In what follows, we compare MFCL with FedEMA from various aspects:
1t is also possible to add more layers to client modules, which creates trade-off between privacy and accuracy (TTbble We implemented the basic form of SA, where gradients are masked on the client side so that the masks cancel out during aggregation. This way, the gServer does not have access to the individual client gradients but still access the aggregated gradients. Alteruative methods, such Secure Multi-Party Computation, or combinations of SA with Homomorphic Encryption Differential Privacy, can ensure that the aggregated gradients remain inacosssible to the greveer. Additioually, employing Trusted Execution Environment (TEE) the gSevee could further enhance privacy without extra computation burden the side of clients.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Table 8: Attack scenarios.
Case
C1D0
CID1
CODO
C0D1
X1 and X2
Reconstructed X
CB dataset in each client. In FedEMA, the client's dataset is assumed to be CB (c.g., each client has classes of CIFAR-10). In Table 2 we utilized the Dirichlet distribution. However, for fair comparison in Table 7 we implemented MFCI.(TN) using the data distribution from FedEMA with clients (i.e., 2 classes per client for CIFAR-10 and 20 classes per client for CIFAR-100). One can see that with 5 clients and CB datasets, the accuracy performance of FodEMA is close to that of MFCL(TN).
Number of clients. As reported in Table 4 of FedEMA Zhuang et al. 20222 the accuracy performance of FedEMA declines with increasing the number of clients, rendering it unsuitable for cross-device FL. MFCL works well with 100 clients (Table 2 or even more, which is more realistic in cross-device scenarios.
Number of samples per client. As reported in Table 12 of [Zhuang et al. (0022) the top 1 accuracy of FedEMA Zhuang et al. (2022) improves with increasing the number of samples per client. The reason is that in FedEMA deep network, such ResNet-50, is trained federally, and having more samples in each client reduces the likelihood of overfitting and enhances the training performance. In contrast, in MFCL a shallow autoencoder is trained federally, making it fcasible to work effectively with a smaller number of data samples, which is more realistic in cross-device FL.
Communication burden and memory footprint. Table 8 presents a comparison of the communication burden, computation burden, and memory footprint between FedEMA and MFCL(TN). The communication burden of F is marginally lower than that of MFCL.(TN) and FedSimCLR. However, FedEMA incurs substantially higher computation burden and memory footprint. The reason is that FedEMA needs to train and store two networks (i.e., online and target networks) at each client, leading to higher computational demands and memory usage.
FedEMA works perfectly well in cross-silo applications where (1) the number of clients is not large and (2) each client has enough data samples and resources. MFCL(TN) is designed for cross-device applica- tions where (1) the number of clients can be large, and (2) clients do not have huge data, communication, computation, and memory resources.
6 Conclusion
To summarize. We have developed MFCL, as a novel approach to FL, transitioning from an E2E learning to a modular framework to leverage contrastive learning in resource-constrainel clients. We also proposed Twin Normalization which is effective addressing the challenges of hetcrogencous CIB data at clients. Our simulations demonstrate that MFCL(TN) achieves superior and more stable performance in highly hetero- geneous and CIB data scenarios. Furthermore, MFCL exhibits reduced communication burden, memory footprint, and computational complexity on the client side. It is noteworthy that MFCL works effectively with group normalization, indicating that it performs well even without twin normalization.
Limitations. MFCL has great potential as a resource-effective approach for cross device FL and can handle severe heterogencous and CIB datascts. However, there exist certain limitations for MFCL. When the number of data samples per client is huge (c.g., in cross-silo FL), the client module should include additional layers to maintain performance. This requirement. compromise the resource efficiency advantages of MFCL.
14

---

Published in Transactions on Machine Learning Research (11/2024)
References
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Nya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference computer and communications security, pp. 308-318, 2016.
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In Proceedings of International Conference on Learning Representations, 2021.
Rangachari Anand, Kishan G Mehrotra, Chilukuri K Mohan, and Sanjay Ranka. An improved algorithm for neural network classification of imbalanced training sets. IEEE Transactions Neural Networks, 4 (6):962 969, 1993.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to ima- genet. In International conference on machine learning, pp. 583 593. PMLR, 2019.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for learning. In proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Securily, pp. 1175-1191, 2017.
machine
Debora Caldarola, Massimiliano Mancini, Fabio Galasso, Marco Ciccone, Emanuele Rodolå, and Barbara Caputo. Cluster-driven graph federated learning over multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7749,2758, 2021.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In Proceedings of International Conference on Machine Learning, pp. 1597-1607, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 15750 15758, 2021.
Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. Xixiu preprint xrXi::2010.0224, 2020.
Aymeric Dieuleveut, Gersende Fort,, Eric Moulines, and Geneviève Robin. Federated expectation maximiza- tion with heterogeneity mitigation and variance reduction. arXiv preprint arXiv:2111. 020083, 2021.
Moming Duan, Duo Liu, Xianzhang Chen, Renping Liu, Yujuan Tan, and Liang Liang, Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions Parallel and Distributed Systems, 32(1):59-71, 01,59-71--0 2020.
Alp Emre Durmus, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. Federated lcarning based on dynamic rogularization. In Proceedings of International Con- [erence on Learning Representations, 2021.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:19586-19597 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 13:21271-21288 2020.
Dong-Jun Han, Hasnain Irshad Bhatti, Jungmoon Lee, and Jackyun Moon. Accelcrating federated learning with split learning on locally generated losses. In ICML 2021 workshop on federated learning for user privacy and dala confidendiality. ICML Board, 2021.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Weituo Hao, Mostafa El-Khamy, Jungwon Lee, Jianyi Zhang, Kevin J Liang, Changyou Chen, and Lawrence Carin Duke. Towards fair federated learning with zero-shot data augmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3310 3319, 2021.
Walid KA Hasan, Yachao Ran, Johnson Agbinya, and GuiYun Tian. A survey of energy efficient iot network in cloud environment. In 2019 Cybersecurity and Cyberforensics Conference (CCC), pp. 13 21. IEEE, 2019.
Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Efficient split-mix federated learning for on-demand and in-situ customization. arXiv preprint arXtt:2203.097/ 2022.
Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 1412876-12888) 2021.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv: 003306 2019.
Wenke Huang, Mang Ye, and Bo Du. Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10143- 10153, 2022.
Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. Advances in Neural Information Processing Systems, $4:30365-30333 2021.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of International Conference on Machine Learning, pp. 448 456, 2015.
Joohyung Jeon and Joongheon Kim. Privacy-sensitive parallel split learning. In 2020 International Confer- ence on Information Nelworking (ICOIN), pp. 7-9. IEEE, 2020.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In Proceedings of International Conference on Machine Learning, pp. 5132 5143, 2020.
Latif U. Khan, Walid Saad, Zhu Han, Ekram Hossain, and Choong Seon Hong. Federated learning for internet of things: Recent advances, taxonomy, and open challenges. IEEE Communications Surveys Tutorials, 1759)9175917799 2021. doi: 10.1109 /00.9///OT..00022 20090400
Minjae Kim, Sangyoon Yu, Suhyun Kim, and Soo-Mook Moon. Depthfl: Depthwise federated learning for heterogeneous clients. In The Eleventh International Conference on Learning Representations, 2022.
Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Dis- tributed machine learning for on-device intelligence. arXie preprint arxiv:1100. 161.00000. 2016a.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communicatio efficiency. arXiv preprint 11i::16100054022 2016b.
A Krizhevsky, Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Slawomir Kukliński, Yuhong Li, and Khoa. Truong Dinh. Handover management in sdn-based mobile net- works. In 2014 IEEE Globecom Workshops (GC Wkshps), pp. 194-200. IEEE, 2014.
Junqing Le, Di Zhang, Xinyu Lei, Long Jiao, Kai Zeng, and Xiaofeng Liao. learning with malicious clients and but-euriottuuuuttuuoeeuuuuuuuuu servers. IEEE Transactions on Information Foren- sics and Securily, 2023.
federated
16

---

Published in Transactions on Machine Learning Research (11/2024)
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Jingtao Li, Adnan Siraj Rakin, Xing Chen, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti. Ressfl: A re- sistance transfer framework for defending model inversion attack in split federated learning. In Proceedings of the IEEF/CVT Conference on Computer Vision and Pattern Recognation; pp- 10194-10202, 2022.
Mu Li, Tong Zhang, Yuqiang Chen, and Alexander J Smola. Efficient mini-batch training for stochastic optimization. In Proceedings of the International Conference on Knowledge Discovery and Data Mining, pp. 661-670, 2014.
Qinbin Li, Bingsheng Hc, and Dawn Song. Model-contrastive federated lcarning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 10713 10722,, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Machine Learning and Systems, 2:429-455 4500, 2020.
Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In Proceedings of NeurIPS Workshop on Distribution Shifts, 2021.
Lumin Liu, Jun Zhang, S.H. Song, and Khaled B. Letaief. Client-edge-cloud hierarchical federated learning.
ICC 2020- 2020 IEFE International Conference on Communications (ICC), pp- 1-6, 2020a. doi: 10.1109 D..110//CCC4027.20000 211.779222..188000
Zhijian Liu, Zhanghao Wu, Chuang Gan, Ligeng Zhu, and Song Han. Datamix: Efficient privacy-prescrve edge-cloud inference. In Computer AssionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-20 28-28, 2020, Proceedings, Port XI 16, pp. 578-595. Springer, 2020b.
Sindy Löwe, Peter O'Connor, and Bastiaan Veeling. Putting end to end-to-end Gradient-isolated learning of representations. Advances in neural information processing systems, 32, 2019.
Nan Lu, Zhao Wang, Xiaoxiao Li, Gang Niu, Qi Dou, and Masashi Sugiyama. Federated learning from only unlabeled data with llas-conditionaal-sariie clients. In Proceedings of International Conference on Learning Representations, 2022.
Ekdeep Singh Lubana, Chi Ian Tang, Fahim Kawsar, Robert P Dick, and Akhil Mathur. Orchestra: Unsu- pervised federated learning via globally consistent clustering. Xiv preprint arXiv:2205. 11506, 2022.
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 2021.
Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint Xrx:::200 12408, 2022.
Yishay Mansour, Mchryar Mohri, Jac Ro, and Ananda Theertha Suresh. Three approaches for personaliza- tion with applications to federated learning. arXie preprint arXiv:2002.10619, 2020.
Yuyi Mao, Changsheng You, Jun Zhang, Kaibin Huang, and Khaled B Letaief. A survey on mobile edge computing: The communication perspective. IEEE communications surveys & tutorials, 19(4):2322-23) 2358, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Commmication-effiiiendt learning of deep networks from decentralized data. Proceedings of Artificial Intelligence and Statistics, pp. 1273-1283 2017.
H Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas. Federated learning of deep networks using model averaging. arXiv preprint arXiv:1602.00033 2016.
Runxuan Miao and Erdem Koyuneu. Federated momentum contrastive clustering. ar preprint arXiv:2206.05003 2022.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang. Fedproc: Prototypical contrastive federated learning on non-iid data. aiv preprint arXiu:2109.0 12273, 2021.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), pp. 1-15, 2018.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint aixi 087888 2018.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulié, and Edoardo Maria Ponti. Modular deep learning, arXiv preprint arXiv:290.11152, 2023.
Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8))3710-33)) 37703372,, 2020.
Shai Shalev-Shwartz, Ohad Shamir, and Shaked Shammah. Failures of gradient-base deep learning. In Proceedings of International Conference on Machine Learning, pp. 3067-3075, 2017.
Haizhou Shi, Youcai Zhang, Zijin Shen, Siliang Tang, Yaqian Li, Yandong Guo, and Yueting Zhuang, Federated supersuue contrastive learning via ensemble similarity distillation. arXiv preprint arXiv:2109. 14611, 2021.
Jau Shin, Chihoon Hwang, Joongheon Kim, Jihong Park, Mehdi Bennis, and Seong- Kim. Xor mixup: Privacy-preservngg data augmentation for onc-shot federated learning. arXiv preprint arXiv:2006.05.4 05148, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint orXic:1409.15550 2014.
Virginia Smith, Chao-Ka Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017.
Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, and Seyit A Camtepe. Advancements of feder- ated learning towards privacy preservation: from federated learning to split learning. Federated Learning Systems: Towards Neat- Generrteeooo AI, pp. 79-109, 2021.
Chandra Thapa, Pathum Chamikara Mahawaga Arachchige, Seyit Camtepe, and Lichao Sun. Splitfed: When federated learning meets split learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8485-8493, 2022.
Valeria Turina, Zongshum Zhang, Flavio Esposito, and Ibrahim Matta. Combining split and federated architectures for efficiency and privacy in deep learning. In Proceedings of the 16th International Conference on emerying Nelworking EXperiments and Technologies, pp. 562-563, 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint
2016.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated lcarning with matched averaging. In Proceedings of International Conference on Learning Representations, 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33: 7611-7623, 2020b.
Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake. Docs learning from docentralizod non-lID unlabeled data benefit from self supervision? In The Eleventh International Conference on Learning Representations, 2023a. URL http:////.ppppeeeeeeeeee//ee/eer
18

---

Published in Transactions on Machine Learning Research (11/2024)
Lixu Wang, Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. In Proceedings of the AAAT Conference Arlificial Intelligence, volume 35, pp. 10165-10172 2021a.
Xiaofei Wang, Yiwen Han, Chenyang Wang, Qiyang Zhao, Xu Chen, and Min Chen. In-edge ai: Intelli- gentixing mobile edge computing, caching and communication by federated learning. Teee Network, 33(5); 156-165, 2019.
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: alternative to end-to-end training. arXiv preprint arXiv:210..1 10832, 2021b.
Zhousheng Wang, Geng Yang, Hua Dai, and Chumming Rong. scaled vision pre-training, IEEE Transactions on Information Forensics and Security, 18:1539 1553, 2023b. doi: 10.1109/TIIS TTF.TIII......2...22222 2023.3244499.
split learning for large-
Yuxin Wu and Kaiming Hc. Group normalization. In Proceedings of the European Conference Computer Vision (ECCV), pp. 3 19, 2018.
Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. arXiv preprint arXiv:2306. ftfi:2306.11867,, 2023.
Miao Yang, Ximin Wang, Hongbin Zhu, Haifeng Wang, and Hua Qian. Federated learning with class imbalance reduction. In Proceedings of the Europeon Signal Processing Conference (EUSIPCO), pp. 2174- 2178, 2021.
Felix Yu, Ankit Singh Rawat, Adilya Menon, and Sanjiv Kumar. Federated learning with only positive labels. In Proceedings of International Conference Mochine Learning, pp. 10946-10956, 2020a.
Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen. Heterogeneous federated learning. arXiv preprint arXiv:2008.0611 06707, 2020b.
Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, and Xiang Chen, Fed2: Feature-aligned federated learning. In Proceedings of the Conference on Knowledge Discovery and Data Mining, pp. 2066-2074, 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33:5824-5836, 2020c.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pp. 7252-7261 PMLR, 2019.
Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang, Feng Yan, and Yang Liu. (BatchCrypt): Efficient homomorphie encryption for (Cross-Silo) federated learning. In Proceedings of USENIX Anrvaal Technical Conference (USENIX ATC 20), pp. 493 506, 2020a.
Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08922 2020b.
Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Improving semi-supervised foderated learning by reducing the gradient diversity of models. In Proceedings of IEEE International Conference on Big Data (Big Data), pp. 1214-1225, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXio preprint arXiv: 00582, 2018
Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi. Collaborative unsupervised visual representation learning from decentralized data. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1912-4921, 2021.
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervised learning. arXiv preprint ar xrXi::2204.0440 04385, 2022.
19

---

Published in Transactions on Machine Learning Research (11/2024)
A Related Works
A.1 Supervised FL with Heterogeneous and CIB Datasets
There are plenty of works in SFL aiming to address the issue of data heterogeneity among the clients. The first approach is to leverage a shared public or synthesized dataset on clients and/or the server to find a solution for the local models that under-represent the patterns at the clients Zhao et al. (2018): Hao et al. (2021). The second approach is based on regularization. FodProx Li al. (2020) regularizes the Euclidean distance between the local and global models. MOON Li et. al. (2021) uses contrastive loss to maximize the agreement of the representations learned by the local and global models. SCAFFOLD Karimireddy et al. (2020) reduces the bias in the gradients by introducing some control variates. FedDyn Acar et al. (2021) dynamically changes the local objectives at each FL round to ensure that the local optimum is consistent with the global optimum. The third approach is based on reducing the bias of the aggregated parameters at the server. The authors in Hsu et al. (2019), leveraging the momentum update on the server, tried to alleviate the oscillations resulting from averaging the biased gradients. The authors in Waan et al. (2020a) proposed match the local updates while aggregating reduce the effect of heterogeneous data. In particular, to achieve fair (if not perfect) performance of the model on the local datasets, rescarchers have recently studied personalized FL, through various approaches, e-g., by treating each client as a lask in multi-task learning framework [Smith et al. (2017) or by dividing clients into different clusters based on their tasks and performing based aggregation as in Sattler et al. (2020): Ghosh et al. (2020).
In addition to the issue of data heterogeneity, the number of data samples for each class in a client's 's dataset is not necessarily the same. This means that each client's data might. be often CIB rather than CB. Over the past few years, some efforts have been devoted to addressing the issue of CIB data in the clients by regularization techniques as in Wang al. (2021a), data level approaches such as data augmentation in Duan et al. ), or data distribution estimation in Yang et al.] (2021). However, most of the existing FL, methods, including all the algorithms mentioned above, assume that clients have fully labelled data. Figure shows different scenarios of data distribution of clients in FL.
A.2 Contrastive Learning and Self-Supervised FL
Rescarchers have studied contrastive and non-contrastive approaches for SSFL Zhuang et al. 2022 Zhang et al. 2020bb; Miao & Koyuncu (2022); Shi et al. (2021): ; Yu el al. (2020a). The contrastive loss, used in Oord al. (2018), defined between two augmented views (i.j) of the same data sample in a mini-batdi of the size of 2K, follows:
1
exp
L
Σ JJEB
log
(7)
2K
sim(ha.bee)
[xx
where hú and hy are the hidden representations of two positive examples [ is an indicator function that is equal to if A i, sim (h,, h2) is the cosine similarity between two vectors the hidden representations, T temperature scalar, and B is a randomly sampled mini-batch consisting of augmented pairs of images. For representation learning without contrastive loss, the authors in Lu al. (2022) proposed use prior information about the client's dataset, instead of labels, to apply supervised learning methods to unlabeled data. Orchestra Lubana el al. (2022) is proposed as SSFL scheme to partition the client 's data into distinguishable clusters. Recently, the authors in Zhuang et al. (2022) proposed a divergence-aware aggregation for FL, based on different self-supervised learning methods. Unfortunately, their scheme does not perform well when the number of clients large (c.g., more than 10). Also, they did not consider the case in which the clients' datasets are CIB.
By "examples", we mean augmented data samples. Positive examples differently augmented versions of the same image, and negative examples the augmented versions of other images.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Sammle par Clavs
40
40
40
40
40
-0
20
Smmol
20
2
Class
Class
Class
ClMss
Class
Class
Claos
Class
(a)
(b)
Same le per Class
Ciient
Gliet2
Clent's
FServer
Client]
Cleenz
Clients
SStverr
sanve et
Class
(c)
(d)
Clientl
Client2
Client3
rServer
Samples
20
20
20
20
Data at Clients
Representations the rServer
Figure Different scenarios for the (raw) data the clients and the contrastive rServer representation mini-batches (simply, reprosentations) at the rServer. (a) clients have homogencous and CB data; and the representations are CB at the rServer, (b) clients have homogeneous and CIB data; and the representations are CIB at the rServer, (c) clients have heterogeneous and CIB data; and the representations are CB at the rServer, (d) clients have heterogeneous and CIB data; and the representations are CIB at the rServer, and (e) clients have heterogencous and CB data; and the representations are CB at the rServer.
Table 9: The architecture of the client module and its decoder on ResNet-18. For the convolutional layer (Conv2D) and transposed convolution layer (Conn2DTranspose) we present the list of the parameters in the soquence of input and output dimensions, kernel size, stride, and padding. For the max pooling layer (MaxPool2D), we show the list of the kernel and stride.
Layer
Details
Conv2D(3, 64, 3, 1)
TN, ReLU
Conv2D(64, 64, 3,
TN, ReLU, MaxPool2D(2,, 2)
Conv2DTrransoose(A4. 64, 3, 1,
TN, ReLU
Conv2D'Transposel 3, 3, L, 1) TN, ReLU Conv2D 3, 1,
B
Experimental Details
In this section, we illustrate the details of the model archilectures and some additional sinulation scenarios.
B.1 Heterogeneous and CIB Data
In our proposed MFCL, the data representations sent from the clients are collected at the rServer, and the contrastive iServer mini-batches constructed with the received representation vectors at the rServer.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Table 10: Same as Table for ResNet-50.
Layer
Details
Conv2D(3, 64, 1, 1)
2
TN, ReLU
3
Conv2D(64, 64, 3, 1)
4
TN, ReLU C64, 128, 3, 1, 1)
5
6
TN, ReLU
7
Conv2D(128, 128, 3, 1, 1)
8
'N, ReLU, MaxPool2D(2,
9
Conv2DTrransoose(222 128, 3, 1, 1)
10
TN, ReLU
11
Cruv2DTranspose(22 (128, 64, 3, 1, 1) TN, ReLU
12
13
Con-2DTranspose(AA. 64, 3, 1, 1) TN, ReLU
14
15
Cran22)rra (64, 3, 3, 1, 1) TN,, ReLU Conv2D(3, 3, 3, 1, 1)
17
Therefore, need to define the concept of CB CIB representations at the rServer. For clarification, provide Figure 6 which consists of data distribution in the clients and the distribution of the representations at the rServer. In Figure 6al the clients have homogeneous and CB data in cach client; In Figure 6b the clients have homogeneous and CIB data in each client; In Figures 6c and 6d the clients have heterogeneous and CIB data in each client; In Figure (6e| the clients have heterogeneous and CB data each client. To sum up, the datasct is CB in all clients, the rServer mini-batches are always CB, even some clients are not successful in sending their representations vectors (Figures 6a and Otherwise, the rServer mini-batches be CB CIB: the rServer mini-batches CIB in Figures 6b and 6d whereas they CB in Figure
6c
B.2
Details of the Client and rServer Modules
B.2.1 Data Distribution
Following baselines Lubana et al. 2002 Luo et al. (2021]: hs {E(0.01, 0.1, 0.5}. For CIFAR-10, 5-0.11 implies each client has two very unbalanced classes and 5=0.1 implies each client has 4 or unbalanced classes, and so on. The smaller g, the more unbalanced the classes which is most realistic for FL.
Makhija et al. (2022) chose ME{10, 100} and
(Figure 7
8
4000
4000
4000
3000
3000
3000
Clinnt
2000
2000
1000
2-
1000
1000
0
0
0.0
(a) CIFAR-10, E=0.01
(b) CIFAR-10, E=0.1
(c) CIFAR-10, E=0.5
Figure Client data distribution with CIFAR-10 considering various values for ç.
22

---

Published in Transactions on Machine Learning Research (11/2024)
Table 11: Projection layer architecture (adopted both ResNet- and ResNet-50).
Layer
Details
Dense
BN, ReLU
3
Dense
BN, ReLU
Dense
BN
Table 12: List of hyperparameters.
Dala
CIFAR-10/100
Ting-ena
Model
ResNet-18
ResNet-50
Client module (CM)
Finst 2 layers
First 4 layers
CM batch size
64
CM optimizer
Adam
Adam
CM learning rate
0.05
0.001
FL rounds
15
15
rServer module (rSM) epochs
150
200
rSM optimizer
LARS
LARS
rSM learning rate
cosine, Ing = 1.0 cosine, Ing = 1.0
Table 13: List augmentation techniques.
Dataset
CTFAR-10/100 Tiny-ImageNet
Augmentation
Brightness
Contrast
Colorjitter
Grayscale
Saturation
Hue
Crop and resize
Gaussian blur
Flip
B.2.2 Modules' structures and hyperparameters
In our experiments, the client's module includes the initial layers of a deep network, extending up to a specified cut-off layer. The decoder for the client module designed such that each client module, conjunction with its decoder forms a convolutional autoencoder. Considering ResNet-18. the client module (i.e., the first two layers of the network) and its decoder are trained federally, with binary cross-entropy loss, across M = 10 clients for CIFAR-10 and M 100 for CIFAR-100 For ImageNn, we include the first layers of ResNet-50 the client's module and train deeper autocncoder federally across M = 100 clients. The details of the client module and decoder are provided in Tables and 10 for ResNet-18 and ResNet-50, respectively.
We use non-linear projection layers (Table 11 on top of the rServer module and also one lincar Dense layer is used on top of the projection layers for linear evaluation. list of other hyperparameters and augmentation strategies listed in Tables 12 and 13 respectively.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Table 14: Accuracy comparison of MFCL and baselines on CIFAR-10 with 10 on ResNet-18, CIFAR- 100 with M 100 on ResNet-18, and Iny-leeeeNet with M 100 on ResNet-50. The best performance is highlighted in boldface. For SSFL methods, the top-1 accuracy with linear evaluation is reported. For MFCL, BN is used at the rServer.
CIFAR-IO
CIFAR-100
Many-ImaeeNet
0.01
0.1
0.5
200
0.01
0.1
0.5
200
0.01
0.1
0.5
200
Method
Avd
62.3
79.2
84.2
92.1
36.7
48.2
55.3
60.5
29.4
36.6
45.3
49.5
FedProx
70.8
83.0
87.6
93.2
36.9
48.7
59.2
67.0
30.1
36.9
47.0
50.1
MOON
65.4
77.7
86.3
94.1
37.2
49.1
60.6
67.6
30.7
36.7
47.4
50.6
FSL
54.1
66.7
74.3
77.0
30.5
29.4
34.3
37.6
19.8
25.4
28.2
31.0
PodSimCLRGS
72.1
79.6
81.9
32.4
35.0
39.6
42.3
21.7
24.3
30.9
33.8
Orchestra
79.6
82.1
82.1
81.7
37.4
40.3
41.2
40.9
26.2
28.8
30.1
29.7
FedEMA
78.8
81.9
83.5
84.1
37.2
40.3
40.9
37.6
27.0
29.1
31.2
38.9
FedU
74.7
77.6
79.1
83.2
32.1
36.7
37.2
39.8
24.6
26.3
29.4
30.1
FealARC
83.3
84.6
85.2
86.7
42.5
54.6
61.0
67.3
32.9
37.5
MFCL(BN)
15.3
32.5
53.8
86.4
14.1
28.3
38.2
51.3
12.9
22.1
27.8
29.5
(FCI(GN)
79.3
79.8
80.7
82.6
41.5
43.2
45.2
47.1
29.6
30.4
33.5
38.3
MFCL(TN)
84.1
84.5
84.6
85.3
44.4
45.5
46.5
47.8
33.1
33.9
36.8
42.2
Table 15: a, B, and in MFCL(TN) on VGG-16-BN (M = 10).
Method ACC
a(bits) ((FL.Ps/Byte)
f=
FedSimCLR 65.7 81.3
21 x1x10
2 16x 13 39 10
MFCI((TN) 82.8
83.5
25x
53x109 11x107
B.2.3 MFCL performance on other backbones
We provided more simulation results more data scenarios in Table 14 Also, to thoroughly assess the per- formance and generalizability of MFCL, conducted additional experiments using VGG-16-BN as another backbone architecture, which is the VGG 16-layer model with batch normalization [Sionnyan & Zisserman (2014). This allowed us to evaluate how MFCL performs across different network structures, ensuring that its evfensiveness is not limited to the ResNet family. Similar to ResNet-18 on CIFAR-10, we select the first two layers as the encoder module and design the decoder as the mirror structure of the encoder part. We train the encoder module and its decoder foderally across the clients and the next layers are trained at the Server. The simulation results presented in Table 15
Since the client modules of the ResNet-18 and VGG-16-BN backbones are identical, the communication. computation, and memory footprints of MFCL(TN) are the same for both backboncs on the client side. However, VGG-16-BN has 138 million parameters, which is why the communcattion, computation, and memory footprint of E2E. FedSimCLR with a VGG-16-RN backbone is higher than those with a ResNet-18 backbone.
B.2.4 Effect of the batch size on the side of clients
As outlined in Algorithm [ the MFCL framework has two distinet phases: the first phase involves federated training the client module and its decoder using the gServer, while the second phase focuses on training the representation modules at the rServer through contrastive learning. One of the benefits of MFCL is that the batch size in the client module, K, does not need to be large. The performance accuracy of MFCL with various K is provided in Table 16
24

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm Modular Federated Contrastive Learning (MFCL)
1: Input: Clients' data {DmM
Jm=1-
2: Initialize: gServer with initial model 0b, rServer with initial model Xo-
3: Phase Training Client Modules and Decoders Federally at gServer
for = .T do
gServer sends 0f to subset of selected clients {me)
6: for each client m } ) in parallel do
7:
Client downloads 0f-)
8:
Client creates two augmented versions X j 1,2 of x;" E D
Client constructs contrastive input mini-batches 1mTh 8 1, R ................................ xK22
10: Client performs the local update on the client module and the decoder.
11: Client sends updated parameters 0 to gServer.
12: end for
13:
gServer aggregates updated parameters: 0fed 2 D
Σ EmEn,
D
D
14: end for
15: Phase 2: Training Representation Modules at rServer
16: for each client m in parallel do
17: Client generates reprosentations = = f(rle "g).j= = 1,2 from the client module.
18: Client sends the representations rerver.
19: end for
20: rServer constructs batches 8 from the received representations.
21: rServer updates its parameters X by minimizing contrastive loss.
22: Output: The trained rServer module is broadcast to clients to ensure global model across clients.
Table 16: Effect of the client module's batch size, K, on the performance accuracy of MFCL(TN). ResNet-18 is trained on CIFAR-10 with 10 clients and { 0.01.
K
8
16
32
64
Method MFCL(TN)
83.9
84.0
84.1
84.1
B.2.5 Effect of the client dropout in training phase
When training phase is finished, the clients send their data representations to the rServer. In real- world scenarios, however, not all the clients who participated in training phase 1 can join training phase 2. Specifically, in training phase 2, the rServer may not successfully receive the representation vectors from some clients who participated in training phase 1. This could be due to the representation vectors being corrupted or lost during transmissions, or some clients being unable to send their data representations. In this case, our assumption that the contrastive rServer mini-batches are (closely) CB might not be valid anymore. We investigated how the performance of our proposed MFCL, with BN, GN, and TN is affected by client dropout, considering the scenarios where 8(%) of clients cannot transmit their represcntation vectors to the rServer during training phase? 2.
In Figure Sa. trained MCFL with BN, GN, and TN the CIFAR-10 dataset with & 200 when M 10 clients participated in training phase 1, assuming the representation vectors from 5(%) clients are not received 0.01. From Figures [a) and Sb] one can see that both TN and GN exhibit similar behaviour when some clients are dropped during training phase For & 200, the gap between TN and BN decreases as more clients are dropped in training phase 2. Meanwhile, for & = 0.01, our proposed TN continues be the best choice even more clients are dropped in training phase? 2.
by the rServer in training phase 2. We repeated the same experiments in Figure Sb with
25

---

Published in Transactions on Machine Learning Research (11/2024)
0.90
0.9
BN
GN
0.8
0.85
PN
0.7
Accuracy
Accuracy
0.6
0.80
0.5
GN
PN
0.4
0.75
0.3
0.2
0.70
10
20
30
0
20
30
Client Drop Rate (%6 Training Phase 2
Client Drop Rate (%) Training Phase 2
(a)
(b)
Figure 8: The classification accuracy of the proposed MFCL with BN, GN, and TN, when the client dropout
rate is 5% in training phase 2. The number of clients participating in training phase is M 10. (a) & 200 is considered, which is the scenario of homogeneous and CB data. (b) 0.1 considered, which is severe case of data heterogeneous clients with CIB data.
Table 17: Effect the number of channels per group on the performance accuracy of MFCL(TN) and MFCL(GN). ResNet-18 is trained on CIFAR-10 with 10 clients and g = 0.1.
A
32
Method
MFCL[GN)
79.2
79.4
79.5
79.8
79.8
79.6
MFCL(TN)
83.1
83.7
84.5
84.5
84.2
83.9
B.3 More Simulation Results for TN
B.3.1 Effect of the number of channels group in TN
We evaluate the effect of the number of channels per group, A, in TN in Table 17 Note that in the extreme of 1 channel per group, GN is equivalent to Instance Normalization Ulyanov et al. (2016) but TN still normalizes the two examples together based the statistics of both channels (i.e., channel per group for each example). One can compare the accuracy performance of MFCL((N) and MFCT(GN) when the number of channels per group is the same in both methods (i.e., A = 4 channels per group in MFCL(GN) A = 2 channels per group for MFCL(TN), which mcans 4 channols in total). Even using as few as channels per group for each example (i.e., channels lotal), MFCL(TN) has substantially better accuracy than MFCL(GN).
B.3.2 Implementation
We implemented TN with a few lines of code in TensorFlow in Table 18 mean and variance, aligned with the respective axes for TN.
We simply need to calculate the
B.4 Visualization of learned clusters by MFCL
In this subsoction, the features of the learned clusters are visualized to provide deeper understanding of the advantages of the proposed normalization method, TN. Each client is assumed to have classes of the CIFAR-10 dataset, with M = 10. Our objective is to evaluate the quality of the learned clusters with MFCL, at the output of the projection layers when BN, GN, or TN is utilized in the client modules and decoders.
Figure Da presents the t-SNE visualization of raw input data at a client, while Figures 9b 9d display the
t-SNE visualizations of the clusters learned from the data representations of all clients at the rServer. Figure
26

---

Published in Transactions on Machine Learning Research (11/2024)
Table 18: Twin normalization TensorFlow.
def TwinNorm(x, gamma, beta, CP, epa-le-5):
# wi features with shape [N beta, gamma: shift and scale, with shape [1,C,1,1]
# CP: number of channels group
N, C, H, W shape
G=C/CP
tf reshape IN, G, c // G, H, W])
meanCN varGN tf. nn moments (x, [2, 3, 4], keep dims=True)
meanP=tf math. add ( varP=tf. math add //2]/>/aarrNN////22222
meanTN-tf.n concat [meanP, meanP], axis=0)
varTN-tf. concat ([varP, varP], axis=0)
(x meanTN) tf. wtf..qrttvvvvtt (varTN + eps)
x tf. re..eehhpeee (x, [N, C, H, W])
gamma beta
9b illustrates the learned clusters by MFCL when BN used in the clients' module/decoder The proposed MFCL. identifies distinguishable clusters for truck, ship, and frog, with some misclassifications within the ship cluster. Specifically, MFCL(BN) fails accurately separate airplanes from ships. Also, there is small cluster of automobiles, with the remainder of the automobile samples being very close to the truck cluster.
Figure 9c demonstrates the learned clusters by MFCL when GN is used in the client module, MFCL(GN) can learn distinguishable clusters for truck, ship, frog, automobile, horse, dog, and airplane, with improved separation between the airplane and ship clusters.
Figure 9d shows the learned clusters by MFCL when TN is used in the clients' module/decoder. MFCL(TN) successfully learns 8 distinguishable clusters for truck, ship, frog, automobile, airplane, deer, and horse.
27

---

Published in Transactions on Machine Learning Research (11/2024)
0.0
airplain
automebile
0.2
0.4
0.6
0.6
airplain ddg automobile frog
0.8
bird horse
ship
1.0
deer
truck
0.6
0.0
0.2
0.4
0.8
1.0
(a)
(b)
airplain dog
airplain dog automobile frog bird horse
automobile
frog
bird
horse
0.2
cal
ship
0.2
cat ship
dear
truck
deer truck
0.4
0.4
0.8
0.8
1.0 0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
(c)
(d)
Figure t-SNE visualization of data client and the clusters learned from the representation vectors at the rServer in the proposed MFCL. CIFAR-10 is trained with M 10, assuming each client has only 2 classes. (a) (-SNE visualization of raw data in single client. In the other three figures, t-SNE visualizations of the clusters learned after the projection layer are shown when the normalization technique used in the client podulee/ecodee is (b) BN, (c) GN, and (d) TN.
28


Reviews:
[{"review_id": "lSOOdLOzNk", "reviewer": "Reviewer_5GpL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written and the narrative is coherent. \n- They tackle two important problems in Federated SSL at once, (i) compute/comms cost and (ii) data heterogeneity.\n- Differences against some of the well-known competitor baseline are properly sketched and elaborated which helps understanding the core advantages of the proposed approach. \n\n**Weaknesses**\n- Comparison against a few other relevant and *most recent* baselines are missing (most references are up until 2022). Here are a few other ones to consider:\n    - [for Federated SSL] Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision? (ICLR 2023)\n    - [for Federated SSL] A Mutual Information Perspective on Federated Contrastive Learning (ICLR 2024)\n- On a related note, there are other FL baselines that are communication/compute efficient. They do not communicate all the model parameters or representations of the entire local dataset. Only prototypes of the local data are used to facilitate collaboration. Examples include:\n    - Personalized Federated Learning with Feature Alignment and Classifier Collaboration (ICLR 2023)\n    - Conditional Moment Alignment for Improved Generalization in Federated Learning (NeurIPS FL workshop 2022) \n\n- Peer normalization (PN) seems to be a combination of Batch and Group normalizations. This has to be further elaborated in the paper. What are the pros and cons of adopting such a construct? Is this only beneficial in Federated SSL, and why is it called \"peer\" in the sense that it hardly related to the peer clients, and more so to the SSL training?\n \n- It is unclear how the two model parts/components come together at the end of training (single global or multiple personalized ones)? The fact that these two components are trained separately might lead to communication efficiency, but what is the impact on the performance? Is there some sort of trade off here? These aspects need to be further clarified in the paper.  \n\n- Minor one, on page 4, $\\theta_m^t$ denotes the encoder and decoder parameter. However, the input and output shape of each component is different. Therefore, given the parameters of the encoder is $\\theta_m^t$, the decoder parameters should be the transpose $[\\theta_m^t]^T$. Nonetheless, it is not clear if the parameters are shared or if they are replaced at each round to make them equal?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a modular FL approach to address the compute/communication cost of self-supervised contrastive FL frameworks, mostly imposed on the clients involved. Here model parameters are divides into two chunks and trained independently through two separate servers. The paper also proposes a form of normalization tailored towards their approach to addressing data heterogeneity in a contrastive setting. The outcome is compared against a suite of self-supervised and fully supervised FL methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All requested changes are already listed in the weaknesses section.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mlDFCiG67T", "reviewer": "Reviewer_CJAM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is clearly written, making it easy to understand and follow.\n- The problem addressed is practical and important, focusing on real-world challenges in federated learning.\n- The paper includes a dedicated section that effectively compares MFCL with traditional Federated Split Learning, offering valuable insights into their differences.\n\n\nWeaknesses:\n\n- Privacy Concerns: MFCL involves transmitting data representations, which raises potential privacy issues, as these representations could be used to reconstruct sensitive user data on the server. Although the authors suggest methods to mitigate this risk, the paper lacks a thorough analysis or visual demonstrations to assess how likely such reconstructions are.\n\n- Technical Contribution: The peer normalization method proposed in the paper seems to be a relatively simple modification of batch and group normalization techniques. Given this, it may not be considered a substantial technical contribution in terms of novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a federated learning algorithm, Modular Federated Contrastive Learning (MFCL), designed to reduce communication, computation, and memory overhead compared to traditional federated learning methods. MFCL adopts a modular approach, splitting the global model into two distinct modules that are trained independently across two severs and multiple clients. The authors conduct comprehensive experiments, benchmarking MFCL against various baseline methods across the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets under multiple experimental conditions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I believe this work requires significant revisions before it can be considered for publication. Below are some suggestions for improvement:\n\n- Thoroughly address the potential privacy leakage issue: A systematic analysis or visualization demonstrating the likelihood of raw data reconstruction from transmitted representations would strengthen the paper's claims. This would provide clearer insight into the extent of the privacy risks associated with MFCL.\n\n- Ensure claims are well-supported by evidence: Avoid making conclusions without sufficient backing. For example, in the introduction, the paper states, “Self-supervised loss functions… require deep and wide networks. Therefore, extracting representations from the client’s local data using shallow or pruned narrow networks with self-supervised loss functions is impractical.” The reasoning in the first sentence does not adequately support the conclusion in the second. More evidence or clarification is needed to justify this claim.\n\n- Improve the clarity and completeness of the writing: For instance, in the introduction, it would be helpful to include a brief explanation of the proposed Peer Normalization technique, giving readers a preview of the method’s technical details.\n\n- Include error bars in the experimental results: The absence of error bars in the results table is a significant omission. Running multiple experiments and including error bars would help account for randomness and provide more robust and reliable conclusions. This is essential for demonstrating the reproducibility of the findings.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u7yMDubXz3", "reviewer": "Reviewer_nPVs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is generally well-structured and easy to follow\n2. The problems this paper addresses are important problems in federated learning.\n3. The proposed method is interesting and seems to be sound: 1) split the model; 2) use two servers for different operations; 3) peer norm.\n4. It is good that the authors explicitly compare with federated split learning. \n5. Extensive experiments are provided to evaluate different aspects of the proposed method.\n\nWeakness:\n\n1. The performance of the proposed method seems to be sub-optimal under scenarios that the data heterogeneity is not that severe. \n2. Peer norm seems to be the combination of BN and GN. Would it have similar drawbacks of BN in FL?\n3. Figure 2 is not easy to understand. \n4. Seems that the models evaluated are both ResNet. It would be good to evaluate one alternative backbone.\n5. Questions: \n    1. What is the goal of this whole learning process? Single global model or personalized models? As the two servers are have no communication, how does it combine and construct a single global model\n    2. Why it is named Peer Norm? The name gives a sense that the normalization is between clients (peers) under the context of distributed training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to address the data heterogeneity under resource constraints in federated learning. The authors propose a new method called Modular Federated Contrastive Learning, together with a new normalization method — Peer Normalization. Experiments demonstrate that MFCL outperforms existing methods under severe heterogeneous scenarios.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please improve Figure 2 and addresses some the concerns above.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cf6CBuMsi8", "reviewer": "Reviewer_YJvQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\tThe paper is clearly written and well-structured.\n\tThe experimental results are impressive, significantly outperforming the baselines.\n\tThe paper provides a solid background and situates the contribution well within existing research.\n\nWeaknesses:\n\tConsider adding an algorithm table to outline the steps of the MFCL process. This would enhance the reader’s understanding by providing a clear, step-by-step guide to your methodology.\n\tConsider adding a paragraph that compares PN with BN and GN. Discussing the similarities and differences will help highlight the incremental improvements and specific advantages of PN in handling heterogeneous data.\n\tThe motivation for training only the initial layers of the model in MFCL isn't clearly explained. Please provide more insight into why this approach was chosen and its advantages over training the entire model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Modular Federated Contrastive Learning (MFCL) to address resource constraints and data heterogeneity in federated learning. MFCL trains only the initial layers of the model federatedly, reducing communication overhead. It also introduces Peer Normalization (PN) to handle class-imbalanced data more effectively than traditional normalization methods. Experimental results demonstrate that MFCL with PN achieves superior accuracy and efficiency, particularly in heterogeneous data scenarios, compared to conventional methods. The approach also enhances data privacy by separating server functionalities and minimizing data transmission, making it suitable for real-world applications on resource-limited devices.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please see the weakness part.", "broader_impact_concerns": "Not involved.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies a modular framework for federated self-supervised learning. The proposal, called Modular Federated Contrastive Learning (MFCL), trains the first few layers through a server in a federated fashion, and then trains other layers at a local server with no communication. The authors additionally propose Peer Normalization (PN) to tackle data heterogeneity in this context. There were several questions/comments about privacy leakage, the presentation of the results, as well as effectiveness of peer normalization by the reviewers that the authors have successfully addressed through additional evidence. As such, the paper is recommended to be accepted. Congratulations to the authors!
Decision: Accept (poster)