A Mutual Information Perspective on Federated Contrastive Learning
Authors: Christos Louizos
Matthias Reisser
Denis Korzhenkov
Venue: ICLR 2024 spotlight (2024)
Keywords: federated learning
contrastive learning
self-supervised
semi-supervised
mutual information
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We extend SimCLR to the unsupervised / semi-supervised federated learning setting through a mutual information lens and study how it interacts with different sources of non-i.i.d. data..

Abstract: We investigate contrastive learning in the federated setting through the lens of Sim- CLR and multi-view mutual information maximization. In doing so, we uncover a connection between contrastive representation learning and user verification; by adding a user verification loss to each client’s local SimCLR loss we recover a lower bound to the global multi-view mutual information. To accommodate for the case of when some labelled data are available at the clients, we extend our SimCLR variant to the federated semi-supervised setting. We see that a supervised SimCLR objective can be obtained with two changes: a) the contrastive loss is computed between datapoints that share the same label and b) we require an additional auxiliary head that predicts the correct labels from either of the two views. Along with the proposed SimCLR extensions, we also study how different sources of non-i.i.d.-ness can impact the performance of federated unsupervised learning through global mutual information maximization; we find that a global objective is beneficial for some sources of non-i.i.d.-ness but can be detrimental for others. We empirically evaluate our proposed extensions in various tasks to validate our claims and furthermore demonstrate that our proposed modifications generalize to other pretraining methods.

Published as a conference paper at ICLR 2024
A MUTUAL INFORMATION PERSPECTIVE ON FEDERATED CONTRASTIVE LEARNING
Christos Louizos, Matthias Reisser, Denis Korzhenkov Qualcomm Al Research' (cloui zos, mreisser, sser, dkorzhen) @qt qua comm com
ABSTRACT
We investigate contrastive learning in the federated setting through the lens of Sim- CLR and multi-view mutual information maximization. In doing so, uncover a connection between contrastive representation learning and user verification; by adding user verification loss to each client's local SimCLR loss recover lower bound to the global multi-view mutual information. To accommodate for the case of when some labelled data are available at the clients, we extend our SimCLR variant to the federated semi-supervised setting. We see that a supervised SimCLR objective can be obtained with two changes: a) the contrastive loss is computed between datapoints that share the same label and b) we require an addi- tional auxiliary head that predicts the correct labels from either of the two views. Along with the proposed SimCLR extensions, we also study how different sources of non-i.i.d.-ner can impact the performance of federated unsupervised learning through global mutual information maximization; is beneficial for some sources of non-i.i.d de-ness but can be detrimental for others. We empirically evaluate proposed extensions in various tasks to validate our claims and furthermore demonstrate that our proposed modifications generalize to other pretraining methods.
find that global objective
INTRODUCTION
For many machine-learning applications "at the edge", data is observed without labels. Consider for example pictures smartphones, medical data measurements smart watches video-feeds from vehicles. Leveraging the information in those data streams traditionally requires labelling e.g. asking users to confirm the identity of contacts in photo libraries, uploading road recordings to a central labelling entity- or the data might remain unused. Fundamentally. labelling data from the edge either happens at the edge or one accepts the communication overhead, privacy costs and infrastructure effort to transfer the data to a central entity and label it there. Labelling at the edge on the other hand either requires enough hardware resources to run more powerful teacher model or it requires costly end-user engagement with inherent label noise and potential lack of expertise for labelling. Ideally, we can leverage unlabelled data directly at the edge by applying unsupervised learning, without the need for labels nor needing to transfer data to a central location.
In this work, we consider the case of federated unsupervised and semi-supervised learning through the lens of contrastive learning and multi-view mutual information (MI) maximization. The main challenges in this context are twofold: estimating the MI can be difficult because often requires intractable marginal distributions (Poole et al.] 2019). Additionally, the federated environment introduces extra complications, as the global MI objective does not readily decompose into a sum of local (client-wise) loss functions, thereby making it difficult to employ FedAvg (MeMahan et al.) 2017). the go-to algorithm in federated learning.
To combat these challenges, we introduce specific lower bounds to the global MI that decompose appropriately into local objectives, allowing for straightforward federated optimization. In doing so, arrive a principled extension of SimCIR (Chen et al. 2020) the federated (semi-) unsupervised setting, while uncovering interesting properties. While each user can run vanilla SimCLR locally,
"Qualcomm Al Research initiative of Qualcomm Technologies, Inc. and/or its subsidiaries.

---

Published as a conference paper at ICLR 2024
Z/2
Z1
Figure I: Graphical model of the assumed generative process under the various sources of non-i.i.d.-
ness: label-skew, covariate shift and joint shift.
establish lower bound for the global MI, it is necessary to add "user-verification (UV) loss (Hosseini et al. (021) for each view. When also dealing with labelled data, the local SimCLR loss on each client needs contrast datapoints in the batch that belong to the same class, thus acting as a form of hard-negative mining. Additionally, besides the UV loss, a label loss is also required for each view. Along with the proposed extensions, we also consider how different sourees of non-i.i.d.-ness.. can impact the performance of federated unsupervised learning through global MI maximization. We show that such an objective beneficial for specific sources non-i.i.d.-nss but it can be detrimental for others. Finally, while our theoretical analysis and model design was based on SimCLR, we demonstrate that they are generally applicable to other pretraining methods as well,
such spectral contrastive learning (HaoChen et al. 2021) and SimSiam (Chen & He 2021
2 FEDERATED MULTI-VIEN MUTUAL INFORMATION MAXIMIZATION
Mutual information (MI) has been paramount tool for unsupervised representation learning; Sim- CLR (Chen et al. 2020), one of the most popular self-supervised learning methods, can be cast as learning encoder model that maximizes the MI between two views of the same image (Wu) ct al, 2020). Applying SimCLR to the federated setting however not straightforward, primarily because the global dataset is not accessible during optimization. In FL, each client only has subset of the available dataset, and this subset not necessarily representative of the global dataset due to differences in the titte-genrratiee process between clients. Various methods have been proposed to mitigate this effect via global dictionaries of representations (Zhang et al., (020) or feature alignment regularizers (Wang et al.,, 2002). In this work, we adopt different view and extend SimCLR to the federated setting through the lens of global multi-view MI maximization.
2.1 FEDERATED SIMCLR
Assume that we have access to an encoder po(x)x) with parameters We would like to train this encoder, such that we maximize the MI between the representations of two views of the input x E IR,=, namely, 2 Z2 E PR, in the federated setting. Let s E N denote the client ID and p(s) a distribution over clients.
In federated learning (FL), the non-i.i.d.-nes. can manifest in various ways: a) label skew, where each client has different distribution over labels p(y) s) but the same p( p(x(y), the most common non-iid- ness assumed in the FL literature, b) covariate shift, where each client has a different distribution over features for specific class p(xly, s), e.g. due to different mobile sensors, but the same p(y) and c) joint shift, where both, the distribution of x, y vary as a function of s. This affects the assumed data-gen gettrrtnnn process of SimCLR representations accordingly, which we illustrate in Figure I
Let I(x; y) denote the MI between x, y and I(x; y/2) be the MI between æ. y conditioned on a third variable Based on the aforementioned generative process and assuming that all labels are unknown, we start the derivation of federated SimCLR from the chain rule of MI:
To 1,118,22) [o 22) + To (/2)) To(21; 8) To(21;2||s)
Ip (2)
= Lo [o(2)222s)) + [o(2)))
((12)
(2)
Global multi-view MI
Local milti-view MI
Client ID MI
Excess client MI
We see that the multi-view MI in the federated setting decomposes into three terms; we want to maximize the average, over the clients, local MI between the representations of the two views Z1, Zg, along with the MI between the representation 2 and the client ID s while simultancously minimizing the additional information 2 carries about conditioned Z2. Such MI decompositions have
2

---

Published as a conference paper at ICLR 2024
also been considered at SSordoi et al. (0021) for improving MI estimation in different context. Unfortunately, in our case these terms require access to potentially intractable or hard to obtain distributions,
will resort easy compute and evaluate variational bounds.
For the first term, i.e., the client conditional MI between the two views, we provide proposition ! which uses the standard InfoNCE bound (Poole al. 2019], leading an objective that decomposes into a sum of local terms, one for each client, thus allowing for federated optimization with FedAvg.
Proposition 1. Let s E N denote the user ID, X E D= the input and Zı, Z2 E RDs the latent representations of the two views of x given by the encoder with parameters Given a critic function R: P. x RD + R, we have that
exp(f(zIR, 22x)) K D exp=1xxp((zzz 228))
10(21;2215) 2 E Ep(s)pa(e: 22)6)1:
log
All of the proofs can be found in the appendix. This corresponds to straightforward application of SimCLR to the federated setting where each client performs SimCLR training locally, i.e., clients contrast against their local dataset instead of the global dataset. We will refer to this objective as Local SimCLR.
In order to optimize the global MI instead of the local MI, we need to address the two remaining terms of equation 2 The first term, 1g (Z1: s), requires information from the entire federation, i.e., pe (Z)). which is intractable. However, with lemma 2.T we show that by introducing "client classification" task, we can form a simple and tractable lower bound to this term.
Lemma 2.1. Let E N denote the client ID, x E Dx the input and 21 E RD= the latent representation of view of x given by the encoder with parameters O. Let 6 denote the parameters of a client classifier (s(z)) that predicts the client ID from this specific representation and let H(s) be the entropy of the client distribution p(s). We have that
[p(2 8) 2 Ep(s)n((21(s) [logra(s/21]] + H(s)
With this bound we avoid the need for the intractable marginal po pz(z)) and highlight an interesting connection between self- euffsuupreieee learning in FL and user-verification models (Yu et al. 2020 Hosseini et al., 202T). For the last term of equation [ we need an upper bound maintain an overall lower bound to To (2 22). Upper bounds to the MI can be problematic as they require explicit densities (Poole et al. 2019). Fortunately, in our specific case, we show in lemma 2.2 that with an additional client classification task for the second view, obtain simple and tractable upper bound.
Lemma 2.2. Let E N denote the user ID, x RD the input and Z1, Z2 E RD the latent the views of x given by the encoder with parameters 0. Let á denote the parameters
representations the of a client classifier ( (s)z2) that predicts the client ID from the representations. We have that
Lo(215 s s/z2) S -Ep(s)po
[log ra [s|)))]
(5)
By combining our results, we arrive at the following lower bound for the global MI that decomposes into a sum of local objectives involving the parameters 0, Φ. We dub it as Federated SimCLR.
1 K
exp(f(Z1k: (2z1k.
To (211))))= 2 (20))p0(2.
log
f= Σ exp( (f(zz,
K
+ llogr (s/Zin) log ( (s|zez) +H(s).
(6)
In this way, Federated SimCLR allows for a straightforward optimization of 0, ó with standard FL optimization methods, such as Reddi et al. (2020), and inherits their convergence guarantees. Furthermore, it is intuitive; each client performs locally SimCLR, while simultaneously training a shared classifier that predicts their user ID from both views. The additional computational overhead of this classifier is relatively minor compared to the encoder itself, making it appropriate for resource constrained devices.
3

---

Published as a conference paper at ICLR 2024
Optimizing the user-verifiaation loss For the client ID loss use single linear layer followed by softmax with three important modifications, as the local optimization of the client ID loss is prone to bad optima due to having "labels" from only "a single class" (that of the client optimizing it) (Yu iet al. 2020); a) the linear layer does not have a bias, as that would make the local optimization of the UV loss trivial and would not meaningfully affect the encoder, b) both the inputs to the linear layer as well as the linear layer weights are constrained to have unit norm and, c) each client locally optimizes only their associated vector weight in the linear classifier while all of the others are kept fixed. In this way each client needs to find their "own cluster center" to optimize the UV loss locally. These centers need to be sufficiently far from the cluster centers of the other clients that a client receives from the server and keeps fixed throughout local optimization.
Effects of non-i.i.d. on the performance on downstream tasks Given access to both the global and local MI objectives, we now want to understand how the type of non-i.i.d.nnes.. determines whether a specific objective is the better choice. To answer this question, we first show at proposition 2 that in the case of label skew, the client classification objective a lower bound to the MI between the representations Z1, Zg and the unavailable label y.
Proposition 2. Consider the label skew data-genert matta-genrrting process for federated SimCLR from Figure i with 8 N denoting the user ID with H(s) being the entropy of p(s). E RD2 the input, Z1, E D= the latent representations of the two views of x given by the encoder with parameters f. Let y be the label and let (s)zz) be a model with parameters e that predicts the user ID from the latent representation 23. In this case, we have that
[o(2i;y) + Io (2) 2 Ep(shpo(a [log (s(2)) + log (s|22)) + 2H()).
Therefore, when the source of non-i.i.d.-nes is heavily dependent on the actual downstream task, the additional client classification objective stemming from the global MI bound is beneficial as it is a good proxy for the thing we care about. In the case of covariate shift, we know that the source of non-i.i.d.nness independent of the label, i.e., I(y: 8) O, so the additional client classification term can actually become detrimental; the representation will encode information irrelevant for the downstream task and, depending on the capacity of the network and underlying trade-offs, can lead to worse task performance. In this case, optimizing the local MI is expected to work better, as the client specific information (i.e., the irrelevant information) not encouraged in the representations.
2.2 FEDERATED SEMI-S SUME-SSPERVISED SIMCLR
In practice, labeled data for a specific task are sometimes available. These could for example constitute a curated dataset at the server or a small labelled subset of data on each client. In this case, it will generally be beneficial for the downstream task if the objective takes these labels into account. To this end, we can use the following deeendeetn expression for the client conditional MI
TØ [2 (8) (1l)) + Te(2), 22/y, 8) [z 3/8, 22)-
(8)
Therefore, once we obtain a label-specific lower bound for this quantity, it will be straightforward translate label-specific lower bound for the global MI by adding back the ver-verification losses for the two views. For the following we will assume that we have an underlying classification task, hence label y N.
For the MI between the two views 2 Z2 conditioned on the label y and client s, we can make use of proposition by treating s, y as the conditioning set. In this case, we again use the InfoNCE loss, with the exception that we now contrast between datapoints that also belong to the same class,
1
K
exp(1 f (ZIk, Zzre)) log
Ie(zi E (m
(9)
K
K Ef=1 exp( (Zz5. Z2x))
For the other two terms that involve the label y we can proceed in similar manner to the client ID For the MI between 2 and y conditioned on s, as y is also discrete, we can make use of lemma 2.] by treating y Therefore, introduce classifier re(2)zz)) and obtain the following lower bound
/)))))s) 2 (p(s)p((r.)111)) [logr<(y(z1) H(y)s),
(10)
4

---

Published as a conference paper at ICLR 2024
Contrastive
loss
Contrastive loss
projector
projector
loss
projector
projector
Label projector
projector
projector
ClientID
projector
Label
71
7-2
ClientiD projector
21
ClientID loss
encoder
encoder
encoder
encoder
Client ID
Z
Z2
11
augrmnt x sugment
encoder
encoder
augmeen
X Buginant
QUgEMEnt X augment
Figure 2: Overview of the SimCLR architectures considered. Local SimCLR (left): each client optimizes contrastive loss their own data, thus the federation implicitly optimizes a lower bound to [[2 zy]s). Federated SimCLR (center): along with the contrastive loss on their own data, each client also optimizes client classifier, thus the federation implicitly optimizes lower bound to I(2,: Supervised federated SimCLR (right): a debelddepnndent variant of federated SimCLR that encourages clustering according the label while also optimizing lower bound ((21;
where H(yls) denotes the entropy of the label marginal at the client, p(y]s). For the MI between Z and y conditioned on Z₂ and we make use of lemma 2.2 and get the following upper bound
Le Le(21; ylz2, s) < - -Ep(s.)) (ea.s. (we lv..) [logro(y) [22]]
(1I)
Putting everything together, we arrive the following label-dep lower bound for local SimCLR
exp(f(Z1x: Z2k))
10(2);22|s) 2 E (a..)))
log
K
1 K
Ej=
224))
+ logra(ulzze) log ( H(y)s)
(12)
which decomposes into intuitive terms; we are performing InfoNCE between the views of the datapoints that belong to the same class and client, while simultancously trying predict the class from the representations of both views. To transition from label-dependent bound for the local SimCLR to label-dependen bound of the federated SimCLR, suffices to add the client classifiers
K
exp( (Zık) Zzx))
Ta (2(3)22) 2 E (2..)))
log
log (>(ri))
K
EE= exp (Zij, Z2x))
+ logg (s(2zz)) log r
( (g) 21k) log r r 2 + H(yls)
+ H(s).
(13)
Figure visualizes all of the SimCLR architectures considered in this work.
The case of unlabelled data The primary motivation of the previous discussion is to tackle the semi-supervised case, i.e., the case when some clients do not have access to all labels. A simple way to handle the unlabelled data is to fall back the bound of proposition] I for the conditional MI when we do not have access to labels. In this way, each client can do form "more difficult" contrastive learning for their labelled data, where they contrast against datapoints which are more semantically similar (i.e., they share the same class), while simultancously trying to predict the correct class whereas for their unlabelled data, they perform standard contrastive learning.
5

---

Published as a conference paper at ICLR 2024
Label-dependent vs label-independent bound Even though both our label-dependent and label- independent bounds are lower bounds of the MI between the representations of the two views, the former should be preferred if labels are available. This is because the label independent one can be satisfied without necessarily clustering the representations semantically, whereas the label dependent one directly encourages clustering according to the label through the additional classification losses, so it is expected to perform better for downstream tasks.
RELATED WORK
Unsupervised learning in the federated context has gained significant attention in recent years. On the contrastive learning side, ZZang et al.] (2020) introduces FedCA, a SimCLR variant for federated setting. The main idea is that the representations between the clients can become misaligned due to the non-i.i.d. nature of FL. The authors then introduce a global dictionary of representations which is shared between all participants and is used to align the representation spaces. One of the main drawbacks of this method is that it requires the transmission of data representations of clients, which leads to reduced privacy. Compared to a global dictionary module, our federated SimCLR aligns the representations of the clients through the additional UV loss component, requiring the communication of just some additional model parameters and not representations. [Dong & Voiculescu (202T) introduces FedMoCo, an extension of MoCo (He et al. 2020) to the federated setting. Similar to FedCA, FedMoCo shares additional client metadata, Le.. moments of the local feature distributions, from the clients to the server, thus leading to reduced privacy. Li et al. (2023a) also extends MoCo to the federated setting however, instead of using a FedAvg type of protocol, the authors employ split learning (Poirot et al. 2019) protocol, which leads to reduced compute requirements at the edge but also requires communicating raw representations of the local data to the server. Finally, the closest to work is the work of Wang et al. (0022) where the authors also explore the effects of non-i.i.d. -iess-ness when training a model with SimCLR in the federated setting. The authors further propose an extension that uses multiple models and encourages feature alignment with additional loss function. In contrast to FeatARC where the feature alignment loss is added ad-hoc to SimCLR, we can see that from our MI perspective SimCLR, feature alignment loss naturally manifests via an additional user-verification loss to SimCLR when optimizing lower bound to the global MI.
On the non-contrastive learning side, Makhiia et al. (2022) introduces Hetero-SSFL. an extension of BYOL (Grill et al. 2020) and SimSiam (Chen & He 2021 to the federated setting where each client can have their own encoder model but, in order to align the local models, an additional public dataset is required. Zhuang et al. (2022) introduces FedEMA, where a hyperparameter of BYOL is adapted in way that takes into account the divergence of the local and global models. In contrast to these methods which require several tricks for improved performance, i.e., moving average updates, custom type of aggregations and stop gradient operations, our federated SimCLR method works by just optimizing straightforward loss function with the defacto standard, FedAvg. On a different note, Lu jet al. (202)) proposes to train a model with pseudo-labels for the unlabelled data and then recover the model for the desired labels via a post- peosstsrocssing step. Finally Lubana et al. (2022) proposes an unsupervised learning framework through simultancous local and global clustering, which requires communicating client data representations, i.e., the cluster centroids, to the server.
On the federated semi-supervised learning side, most works rely on generating pseudo-labets for the unlabelled examples. Peong et al. (2020) proposes FedMatch, an adaptation of FixMatch (Sohn ce al. 2020) to the federated setting by adding one more consistency loss that encourages the models learned on each client output similar predictions for the local data. The authors also propose pseudo-labelling strategy that takes into account the agreement of client models and a parameter decomposition strategy that allocates separate parameters be optimized on unlabelled and labelled data. In contrast, our semi- sunivsupervised objectives are simpler, do not rely on pseudo-labels (which introduce additional syper-parameters for filtering low-confidence predictions) and do not require communicating client specific models among the federation. [Liagg et al. (2022) proposes a student- teacher type scheme for training on unlabelled data, where consistency regularization applied. The teacher model is exponential moving average of the student and novel aggregation mechanism is introduced. Our proposed methods for semi-supervised learning could potentially also benefit from better aggregation mechanisms, but leave such an exploration for future work. Finally, Kim et al.) (2022) introduces ProtoFSSL, which incorporates knowledge from other clients in the local training via sharing prototypes between the clients. While such prototypes do improve performance, they also

---

Published as a conference paper at ICLR 2024
reveal more information about the local data of each client, thus reducing privacy. In contrast, our federated semi supi-suppr framework does not rely on sharing prototypes between the clients.
EXPERIMENTS
Our experimental evaluation consist of unsupervised and semi-supervised experiments, where for the latter each client has labels for 10% of their data. To quantify the quality of the learned representations, we adapt the classical evaluation pipeline of training linear probe (LP) be in line with common assumptions of self-supervised learning. In the unsupervised case, we report the LP accuracy on the union of clients' labelled version of their data, as this corresponds to the traditional evaluation pipeline. For the semi--upervised case, we train LP on top of the representations of the clients' labelled training data (which is subset of the full training set) and then report its test accuracy. At every evaluation for plotting of learning curves, we initialize the LP from the final parameters of the previous evaluation. Furthermore, as we mention at section 2.1 the nature of non-i.i.d. data in FL can manifest in various ways: label skew, covariate shift and joint shift, i.e., a combination of the two. We therefore evaluate, besides label skew (the predominant type of non-i.i.d. ness assumed in the FL literature), covariate shift by creating a rotated version of CIFAR 10 and CIFARIO 100 well as a joint shift case where both sources of non-i..d,, -ness are present. For CIFAR 10 we consider 100 clients whereas for CIFAR 100 consider 500 clients. For the encoder we use ResNet18 architecture adapted for the CIFAR datasets where, following [Hsieh et al.. (2020), we replace batch normalization (loffe& & Szegedy 2015) with group normalization (Wu & He 2018).
federated
-nss
In order to demonstrate the general usefulness of our theoretical results and model design stemming from our MI perspective, we include two more methods in our evaluation besides SimCLR. The first one is spectral contrastive learning (HaoChen et al. 202I) (dubbed as Spectral CL) as another instance of constrastive learning and the other is SimStam (Chen He [021I. non-contrastive method. For both of these methods, we consider both a "local" " variant where each of the losses optimized locally and Reddi et al. (2020) is applied to the parameters as well as, based on the intuition from our federated SimCLR, "global" variant where the same UV loss component of federated SimCLR is added to the baselines. As we show in proposition 2 such an auxiliary task is beneficial in the case of label skew in general. Furthermore we also extend these baselines to the cemi-supervised setting. Based on the insights from our label-dependent MI bounds for SimCLR, we consider abel-deeendent variants of SimSiam and Spectral CL where, when labels are available, the unsupervised losses are evaluated between elements that share the same class and a classification loss for the two views is added to the overall loss function.
Unsupervised setting The results in the unsupervised setting can be seen in Table T In the case of label skew, adding our user-verification loss to each of the local losses leads to (sometimes dramatic) improvements in all cases. This is to be expected, as in this case the mutual information between the labels and the client ID, I(y: s), is quite high, the UV loss acts good proxy for the downstream task. For SimCLR we observe a ~ 6% improvement on CIFAR 10/100 and on Spectral CL we observe 11% and 8% respectively. SimSiam type of methods generally underperformed compared to SimCLR and Spectral CL., and we believe this is due to representation collapse, especially given that in our setting we employ group normalization instead of nommaaizzmiiiiiinn. On covariate shift, we now see that the situation is flipped; as in this case 1(y; s) O, local SimCLR / Spectral CL are doing better compared to their global counterparts that include the UV loss. Both local SimCLR and Spectral CL perform better by ~ 1 - 2% and ~ 2 4% on CIFAR 10 and CIFAR 100 respectively. with local SimCLR providing the better overall performance. Finally, on the joint shift case, the label skew is strong enough to allow for improvements with the additional UV loss components in most cases; for SimCI. there is an improvement of - 5% and for Spectral CL. there is ~ 8% improvement for CIFAR 10 but a drop of ~ 8% for CIFAR 100. We attribute the latter to the overall instability of Spectral CL in our CIFAR 100 experiments, explained by the large standard error.
Overall, we observe that the results are consistent with our expectations; when the source of non-i.i.d.- ness in the federated setting is strongly correlated with the downstream task, optimizing a "global" objective, such as [(z), 22), is beneficial, downstream task. This intuition also generalizes to one of our baselines as, e.g., even Spectral CL benefits from the addition of the UV loss in such settings. In the absence of such correlation, the
the additional UV term serves for good proxy for the
7

---

Published as a conference paper at ICLR 2024
Table Test performance (%) on the unsupervised setting along with standard error over seeds. Clients' data is assumed to be fully annotated for LP fine-tuning in the unsupervised case.
CIFAR
CIFAR 100
Method
Label skew
Covariate shift
Joint shift
Label skew
Covariate shift
Joint shift
Local SimCLR
79 4
74.3
71.020.4
42.2+t0.
41 41.2
38. 14.3
Federated SimCLR
85. 85.010.2
73.8
74.8
48.5
39.510.2
43. 43.110.2
Spectral CL
76.5
73.510.4
68.2
33.316..
33.6.2.3
29. 69.6.1.2
Spectral CL UV
87.8
71.7=0..
76.640.6
41.0=a.4
29.3+44.8
21.5+6.2
SimSiam
40. 40.0
39.9
39. 39.640.3
16.940.3
16.640.4
16.940.4
SimSiam + UV
35.4+0.4
35.4.0.2
34.5.0.3
16.540.2
16.54a.3
16.340.
Supervised
89.640.1
78.3
76.3
59.240.2
47.940.2
43.940.3
Table 2: Test set performance (%) on the semi-supervised setting with 10% labelled data on each client along with standard error over 5 seeds. We use the corresponding labelled subset for the LP.
CIFAR 10
CIFAR 100
Method
Label skew
Covariate shift
Joint shift
Label Skew
Covariate shift
Joint shift
Local SimCLR
24.5
49. 19.1+1.
45.8+1.
30.3_0.2
15.1
13. 13.1
Federated SimCLR
78.0g0.2
50.3
49.9
34.5+0.8
14.8g0.3
14.6.0.a
Spectral
71.2
48.020.7
45.4+1.5
30.120.2
14.1
12.3
Spectral CL. UV
79.6
49.7+1.0
49.8
34.0+0.2
13.7
13.640.4
SimSiam SimSiam UV
75.3
46.820.7
40.540.9
30.7_0.2
13.440.3
12.840.3
80.4
50.0
44.311.0
34.3
13.610.3
14.0.0.4
Supervised
75.1
48.1
42.7
29.6
12.6g0.2
12.2
simple local SimCLR Spectral CL variants are doing better since they do not encode information in the representations that is irrelevant for the downstream task.
Semi-supervised setting Our semi-supervised results with 10% labelled data in Table show interesting observations. Overall, we improve performance with semi-superviser training relative to purely supervised training on the labelled subset of the data. On CIFAR 10, we notice that our semi-supervised models with the UV loss do better than the local variants on all sources of non-i.i.d.- ness, even the case covariate shift. Despite the limited quantity of labels available, we believe that the encoders possessed sufficient capacity to both retain and separate the label-specific and cubel-independent (e.g., rotation) information. Consequently, the downstream LP could accurately use the label-specific portion of the representations for its predictions. SimSiam does much better in this setting, as the supervised objective prevented representation collapse, achieving the best performance on label skew when add the UV loss, whereas Federated SimCLR does best the joint shift.
4.1 ABLATION STUDIES
In this section perform additional experiments in order to investigate the behaviour of local and federated SimCLR under different settings. We adopt our CIFAR 10 setting with 100 clients and strong (a 0.1) joint shift, unless mentioned otherwise.
Amount of non-i.i.d.-ness For the first set of experiments we investigate how the amount of non-i.i.d.-ness affects the local and federated SimCLR performance with E We adopt the joint shift setting and perform experiments with different strengths for each source of non-i.i.d.nness. The results can seen Figure Ba where we have an interesting observation; federated SimCLR does better the higher the amount of label skew non-i.i.d. of local SimCLR on i.i.d. data. This can be explained from our proposition 2 As the amount of label skew increases, the client ID carries more information about y. thus Ig (z1. y/s) becomes lower and the lower bound tighter. On the flipside, when there is strong covariate shift and not enough label-skew, observe that local SimCLR has consistently better performance.
in fact even surpassing the performance
8

---

Published as a conference paper at ICLR 2024
10r
Local SimCUR
Federated SimCUR
78.0
75
70
712
Fodorated SimCUR
Teat accursey
201-
435
757
74.
30"
741
Fully supervised Local SimCLR
72.6
55
Federated SmCLL
10
16
16
10
10
10a
10
10
10
30
10
16
36
16
0.2
0.4
3.6 0.5
SCOM
ellabel ccoM
opochs
(a)
(b)
Figure 3: CIFAR 10 ablation studies. (a) Performance of local and federated SimCLR as a function of the non-i..d.-ness strength for covariate shift and label skew. (b) Performance of local and federated SimCLR for different amount of local epochs E in the case of strong (a 0.1) covariate shift and label skew. (c) Performance of local and federated SimCLR in the semi-supervised setting as a function of the amount of available labelled data.
Amount of local updates The auxiliary UV objective in federated SimCLR can be problematic for a large amount of local updates, as there is only single available class at each client. Therefore, federated SimCLR requires relatively frequent synchronization. We show in Figure 3b how the amount of local epochs affect local and federated SimCLR when keeping fixed computation budget; more local epochs imply less communication rounds and vice versa. We can see that federated SimCLR achieves the best performance of the two with 1 local step, however, its performance drops with more local updates and eventually becomes worse or comparable to local SimCLR.
Amount of labelled data for the semi-supervised setting Finally, we also measure the impact of the amount of available labelled data in the semi-supervised setting for local and federated SimCLR. We measure this by keeping a fixed and labelled holdout set which we use to train a LP on top of the representations given by the two algorithms. We also train fully supervised (i.e., on 100% labelled training data) baseline with the same augmentations as the SimCLR variants. We can see in Figure that the test accuracy of the LP improves with more labelled data for both algorithms, as expected. Federated SimCIR demonstrates improved performance compared to local SimCLR on all cases considered, with the biggest advantages seen when the amount of available labelled data during training is low. Furthermore, federated SimCLR R reaches performance comparable to the fully supervised baseline with 2 50% labelled training data.
5 DISCUSSION
In this work analyze contrastive learning and SimCLR in the federated setting. By adopting a multi-view MI view, we arrive at several interesting observations and extensions. We show that naive application of local SimCLR training each client coupled with parameter averaging the server, corresponds to maximizing a lower bound to the client conditional MI between the two views. We then identify that, in order to close the gap. for global MI an auxiliary verificiiinntonn task is necessary. Finally, through the same MI lens, we extend both local and federated SimCLR to the semi-supervised setting in order to handle the case of partially available data. Despite the fact that these modifications were developed through the MI view for SimCLR, we show that they are generally useful for pretraining in the federated setting, yielding improvements for both spectral contrastive learning and SimSiam.
As non-i.i.d. data are an inherent challenge in Fl., we further discuss how affects contrastive learning, both theoretically and empirically. In the of label skew, the most predominant type of non-i.i.d. -.iss.-eess in the FL literature, we show that maximizing the global MI through federated the auxiliary user classification task is good proxy for the unavailable
SimCLR is appropriate, label. On the flipside, in the case of covariate shift, local SimCLR leads to better models due to not being forced to encode irrelevant, for the downstream task, information in the representations.
For future work, we will explore improved variants of the UV loss that can tolerate more local optimization, well better bounds for the MI in the federated setting.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Nanqing Dong and Irina Voiculescu. Federated contrastive learning for decentralized unlabeled medical images. Medical Image Computing and Computer Assisted citieenttionMMMCCA 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part III 24, pp. 378-387, Springer, 2021.
-aan-Battien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21253 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-500 50100,5011111000 2021.
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXi::2104.05704 2021.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Hossein Hosseini, Hyunsin Park, Sungrack Yun, Christos Louizos, Joseph Soriaga, and Max Welling, Federated learning of user verification models without sharing embeddings. International Conference on Machine Learning, pp. 4328 4336. PMLR, 2021.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pp. 4387-4398. PMLR, 2020.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Mcasuring the effects of non-identical data distribution for federated visual classification. arXiv preprint
2019.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Wonyong Jeong, Jaehong Yoon, Hunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. arXiv preprint htt::::0006120999 2020.
Woojung Kim, Kcondo Park, Kihyuk Sohn, Raphael Shu, and Hyung-Sin Kim. Federated semi- supervised learning with prototypical networks. arXiv preprint 1231, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: arXiv:1412.6000 1412.6980, 2014.
Jingtao Li, Lingjuan Lyu, Daisuke Iso, Chaitali Chakrabarti, and Michael Spranger. MocoSFL: enabling cross-client collaborative self-supervised learning. In The Eleventh International Confer- ence on Learning Representations, 2023a. URL wttpt:ttt////c. hattps://openzeeiews net femeetfoounng Forummiiii
20GJXyMNoPZ
Ming Li, Qingli Li, and Yan Wang. Class balanced adaptive pseudo labeling for federated semi- supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (6292-16301 2023b.
10

---

Published as a conference paper at ICLR 2024
Xiaoxiao Liang, Yiqun Lin, Huazhu Fu, Lei Zhu, and Xiaomeng Li. Rsefed: Random sampling consensus federated semi- supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10154-10163, 2022.
Haowen Lin, Jian Lou, Li Xiong, and Cyrus Shahabi. Semifed: Semi-supervised federated learning with consistency and pseudo-labeling. arXiv preprint arXiv:2108.09412 2021.
Nan Lu, Zhao Wang, Xiaoxiao Li, Gang Niu, Qi Dou, and Masashi Sugiyama. Federated learning from only unlabeled data with class-conditiona--saainng clients. arXiv preprint arXivv2204.03304, 2022.
Ekdeep Singh Lubana, Chi lan Tang, Fahim Kawsar, Robert P Dick, and Akhil Mathur. Orchestra: Un- supervised federated learning via globally consistent clustering. arXiv preprint 15Xiv 2022.
Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. arXiv preprint arXiv:2205. 12493, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiiieenn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1288 PMLR, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arX.vv:887.03375. 2018.
Maarten G Poirot, Praneeth Vepakomma, Ken Chang, Jayashree Kalpathy-Cramer Rajiv Gupta, and Ramesh Raskar. Split learning for collaborative deep learning in healthcare. arXiv preprint arXiv:1912.12115. 2019.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180.
PMLR, 2019.
Sashank Reddi, Zachary Charles, Manzil Zaheer. Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.0022 00295, 2020.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Alessandro Sordoni, Nouha Dziri, Hannes Schulz, Geoff Gordon, Philip Bachman, and Remi Tachet Des Combes. Decomposed mutual information estimation for contrastive representation learning. In International Conference on Machine Learning, pp. 9859-9869. PMLR, 2021.
Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake. Does learning from decentralized non-iid unlabeled data benefit from self supervision? In The Eleventh International Conference on Learning Representations, 2022.
Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual in- formation in contrastive learning for visual representations. arXiv preprint 13311 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Felix Yu, Ankit Singh Rawat, Aditya Menon, and Sanjiv Kumar. Federated learning with only positive labels. In International Conference on Machine Learning, pp. 10946-10956. PMLR, 2020.
Fengda Zhang, Kun Kuang. Zhaoyang You, Tao Shen, Jun Xiao, Yi Zhang, Chao Wu, Yuet- ing Zhuang, and Xiaolin Li. Federated unsupervised representation learning. arXiv preprint arXiv:2010.08983 2020.
11

---

Published as a conference paper at ICLR 2024
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-aware federated self-supervisee learning. arXiv preprint arXiv:2204.0433 04335, 2022.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETUP
Data partitioning and non-ii.d.nnss For the label-skew setting, we use the Dirichlet splits for CIFAR 10. 100 discussed at Reddi et al. (2020) with 0.1 in both cases. Notice that we adopt the convention of Hsu et al. (019)) where a is multiplied by the prior probability of the label in the dataset, so, for example, the case of CIFAR 10 the final concentration parameter is 0.01.
For the covariate shift setting we consider the case of rotation non-i.i.d-ness More specifically, we first perform an i..d., with respect to the labels, split of the data into 100 and 500 clients for CIFAR 10 and CIFAR 100 respectively. Afterwards, we bin the [0, 2Tr] range into 10 rotation bins and then assign each client bins according to a Dirichlet distribution with a 0.1. In this case, each client receives one or two bins of rotations. After that bin assignment, each client randomly rotates each image of their local dataset once with an angle for each image sampled i.i.d. from the bins selected at that client. For the evaluation we consider non-rotated images.
For the joint shift setting we mix the two cases above by first performing a non-i.id., Dirichlet split, i.e., 0.1, according to the labels and then apply the non-i..d. rotation strategy described above.
Architecture details For all methods we use the same encoder model, ResNet18 architecture adapted for CIFAR 10/100 by replacing the kernel of the first convolutional layer with a 3 x 64 x 3 x 3 kernel and removing the max moooingg and last fully connected layer. Furthermore, to better accommodate for the non-i.i.d. issues the federated learning scenario (Hsich al.. 2020) we replace batch normalization (loffe & Szegedy 2015) with group normalization (Wu & He 2018). For the client ID projector, we use a simple MLP on top of the encoder output with single ReLU hidden layer of 2048 units and 128 output units. For the auxiliary classifier in the case of semi-supervised learning we use a simple linear layer on top of the encoder output.
For our SimCLR and spectral constrastive learning variants, the representations of the encoder are passed through an MLP projector with a single hidden layer of 2048 units and 128 dimensional outputs. The contrastive loss between the two views is measured at the output of the projector.
For our SimSiam baseline we measure the cosine similarity objective on the output of a projector that follows the SimCLR design with the exception that we also add group normalization layer before the hidden layer, as SimSiam was unstable without it (especially at the unsupervised experiments). For the predictor we another single hidden layer MLP with 2048 ReLU units and group normalization.
For the data augmentations, in order to create the two views, we follow the standard recipe of random cropping into 32 32 images, followed by random horizontal flip, random, with probability 0.8, color distortion with brightness, contrast, saturation factors of 0.4 and a hue factor of 0.1. The final augmentation is random, with probability 0.2, RGB-to -ogaysscale transformation.
Optimization details For local optimization we use standard stochastic gradient descent with learning rate 0.1 for both CIFAR and CIFAR 100 for, unless mentioned otherwise, single local epoch and batch size of 128, After the local optimization each client communicates to the server the delta between the finetuned parameters and the model communicated from the server to the clients. The server averages these deltas, interprets them as "gradients", , and uses them in conjunction with the Adam Kingma & Ba (2014) optimizer order update the global model. This is a strategy originally proposed in Reddi al. (2020). For the server-side Adam we are using the default hyperparameters.
a specific round has been completed,
B ADDITIONAL EXPERIMENTS
In this section we consider more baselines for both our unsupervised and semi-supervised setups in the federated setting.
B. UNSUPERVISED SETTING
Additional baseline We consider more baseline for self- self-supervised learning in the federated setting, FeatARC /Wang et al., 2022). specifically the "Align Only" variant. We omit the clustering approach as it makes additional assumptions compared to unsupervised learning setup. The
13

---

Published as a conference paper at ICLR 2024
authors report results with loss-coefficient of À 1.0, which lead to loss divergence in so we report À = 0.5, which was stable except for the covariate shift setting. We see in table 3 that adding FeatARC aligmment regularization does not result in improved accuracy, contrary FeatARC paper results would lead us to expect. We hypothesise that this due to the differences in setup. Whereas FeatARC considers cross-silo setting with large number of local update steps, our setting focuses on the cross-device setting with one local epoch per client communication round. We leave further analysis of FeatARC applicability this cross-device setting future work.
what the
Table 3: Test set performance on the unsupervised setting of CIFAR 10. Clients' data is assumed to be fully annotated for LP fine-tuning in the unsupervised case.
Method
Label skew
Joint shift Covariate shift
Local SimCLR
79.4
14.3
71.0
Local SimCLR + FeatARC
70.4+0.2
34.4+-
57.642.7
Federated SimCLR
85.0=0.2
73.8+0.2
74.8+0.5
Spectral CL Spectral CL + UV
76.5+1.1
73 73.540.4
68. 68.2+0.6
87.8 00.3
71.7
76.6 76.6
SimSiam
40.0
39.9
49.640.3
SimSiam UV
35.4
35.440.2
34.540.3
Supervised
89.6
78.340.1
76.3+1.1
Tiny Imagenet dataset To demonstrate the scalability of our theoretical results and model design stemming from our MI perspective, we also consider the more challenging task of self-supervised pretraining TinyImagenet. It consists of 100k training examples and 10k test examples, each beloging to one of 200 classes. We apply our federated CIFAR 10 setting to this dataset as well, i.e., we partition the training dataset to 100 clients with either the covariate shift or joint shift non-i.i.d strategies. We sample 10 clients round in order to optimize the models and each client performs one local epoch of updates. The encoder mddel we use
Compact Convolutional Transformer Hassani et al. (202T) in the "CCT-4/3x2" variant, i.e. with 4 transformer encoder layers and a 2-layer convolutional feature extractor with a 3x3 kernel size. The results with the different methods can be seen at table 4
Table 4: Test set performance (%) on the unsupervised setting of LinyImagenet with 100 clients after 50k rounds. Clients' data is assumed be fully annotated for LP fine-tuning in the unsupervised case.
Method
Label skew
Covariate shift
Joint shift
Local SimCLR
33.3
30.3
29.6
Federated SimCLR
38.0
30.0
31.6
Spectral CL Spectral CL + UV
34.0
28.4
27.9
39.7
29.5
32.4
SimSiam
10.6
4.7
0.5
SimSiam UV
0.5
0.5
0.5
Supervised
44
36.6
33.0
Overall, see that the results are consistent with our intuitions and story in the case of contrastive methods; the biggest gains from the additional UV loss are in the case label skew and joint shift. SimSiam generally underperformed in this setting, which is also consistent with our observations in the case of unsupervised learning on CIFAR 10/100, probably due to representation collapse, given that in setting we group normalization instead of batch normalization.
14

---

Published as a conference paper at ICLR 2024
B.2 SEMI-SUPERVISED SETTING
Additional pseudo-labeling baselines We provide more results on our partially labeled (with 10% labeled data on each client) sumervipe setting by also considering baselines that perform pseudo- labelling as a means for semi-supprrvsed learning. The two methods we consider are SemiFed (Lin jet al. 202T) and CBAFed (Li et al.] 2023b). For both of these settings we have the following modifications that bring them in line with our semi-supervised setup.
For SemiFed we do not make use of an ensemble of client models in order to impute the missing labels but rather assign a pseudo-label to the datapoint based on the received server model on each client. In this way, our proposed methods and SemiFed have similar communication costs and privacy, exchanging models directly trained on local data between clients reduces the overall privacy. For CBAFed, we do not use residual weight connection, in order to have a consistent optimization strategy for all our methods, but do use the class balanced adaptive threshold strategy. We follow the setup described in Appendix F.5 of (Li et al. 2023b) to train a model with partially labeled clients.
From what we can it table 5 and table 6 our conclusion about the usefulness of the UV loss (c.f. proposition 2 applies to this setting as well. While SemiFed underperforms when trained without the UV loss, it manages improve upon the fully supervised baseline and be comparable to the other methods when we add it back. On CIFAR 10, adding the UV loss yields a significant 16%7% improvement in the case of label skew and on CIFAR 100, while it gets a more modest 6% improvement, it manages to outperform all other methods. CBAFed performs worse than self- supervised methods albeit also benefits from adding the UV loss all the conducted experiments.
Table 5: Test set performance (%) on the semi-supervvsed setting of CIFAR 10 with 10% labelled data on each client along with standard error over 5 seeds for all experiments except of CBAFed which have one seed only. We use the corresponding labelled subset for the LP.
Method
Label skew
Covariate shift
Joint shift
Local SimCLR
74.5
49.1
45.8 49.9
Federated SimCLR
78.0+0.2
50.3
Spectral CL
74.2+0.3
48.040.
45.4+1.5
Spectral CL + UV
79.6+0.3
49.0
49.8-1.1
SimSiam
75.3+0.4
46.8+0.7
40.5
SimSiam UV
80.4
50.0
44.3 1.0
SemiFed
60.0
18.641.8
37 37.2.1.99
SemiFed + UV
76.7
24.0.2.2
45. 12.0
CBAFed
66.3
45.9
34.8
CBAFed + UV
74.1
48.2
36.2
Supervised
75.1
48.1
42.7+1.7
Tiny Imagenet dataset To demonstrate the scalability of our semi-supervised model design stem- ming from our MI perspective, we also consider the more challenging TinyImagenet task in the case of label skew non-i.i.d-nnes with Dirichlet splitting and an a 0.1 multiplied by the prior probability of each class. The setup is similar to our semi-supervised federated CIFAR 10 setting, with 100 clients and 10% labelled data per client. We sample clients per round in order to optimize the models and each client performs one local epoch of updates. We use the same CCT architecture as the unsupervised TinyImagenet experiment. The results with the different methods can be seen in table 7
We observe similar patterns to our unsupervised TinyImagenet setting, with the biggest gains for the contrastive methods from the UV loss being in the case where some label skew is present. SimSiam did experience representation collapse at the case of label skew, however, by adding to it the UV loss, this successfully mitigated and improved significantly the performance.
15

---

Published as a conference paper at ICLR 2024
Table Test set performance (%) on the semi-supervised setting of CIFAR 100 with 10% labelled data on each client along with standard error over 5 seeds. We use the corresponding labelled subset for the LP.
Method
Label Skew
Covariate shift
Joint shift
Local SimCLR
30.3
15.1
13.1
Federated SimCLR
34.5
14.8+0.3
14.6.0.3
Spectral CL Spectral CL + UV
30.1
14.140.4
12.3
34.0+0.2
13.740.
13.6=0.4
SimSiam
30.7
13.4+0.3
12.8+0.3
SimSiam + UV
34.310.1
13.610.3
14.0.10.4
SemiFed
29.7
13.3
12.3
SemiFed + UV
35.7
13.410.6
13.1
Supervised
29.610.3
12.610.2
12.2
Table 7: Test set performance (%) on the semi-supervised setting of Tinylmagenet with 100 clients after 50k rounds. We use the corresponding labelled subset for the linear probe.
Method
Label skew
Covariate shift
Joint shift
Local SimCLR
18.5
8.1
6.7
Federated SimCLR
19.5
8.4
7.4
Spectral CL Spectral CL + UV
17.8
8.3
6.9
18.9
8.1
7.5
SimSiam
0.5
8.1
6.9
SimSiam UV
20.0
8.5
6.9
Supervised
17.9
8.4
7.7
C ALGORITHMS
Algorithm The server side algorithm for our federated SimCLR Spectral CL SimSiam with optional verr-verifiittion and semi-supervision.
Initialize 0 and o with 81, 6;
for round in I.... do Sample S clients from the population
Initialize V = 0, V =
for s in S do
0, 0,
CLIENT(S,, 0z, Φ.) V 0.-0.
[S]
V +
d.-o,
1ST
end for
gi+11
ADAM(V VA)
end for
16

---

Published as a conference paper at ICLR 2024
Algorithm 2 The client side algorithm for our federated SimCLR / Spectral CL / SimSiam with optional user-verification and semi-supervision. Lu corresponds unsupervised loss component of SimCLR / Spectral CL/ SimSiam. 3 is coefficient that determines the weight of the UV loss, with default value of 1.
Get 8, & from the server Ony ts 6,, Φ for epoch in l,..... E do for batch b B do
> Unlabelled and labelled datapoints of the batch 6
Lut. (æ), 31) - b
Get the two views through augmentations
=f] = AUG æx)), AUG xz])
Representations of the views from the encoder f with parameters 8,
[z = [ = + s(==,, 0.), [[ [[==a,=
& Unsupervised loss with, depending on B. additional UV loss
C. = Lu (2 z o.) + BBu (s, 2ut 2
& Supervised loss on the labelled data
for label i E {0...., :.....-1 |
1 } do
& Unsupervised loss between datapoints of the same class = = [yn = i], o [yn = i]; d.)
end for
& Standard supervised loss
L.+ = (yt, 2/ 27; 6&)
b Local gradient updates the loss
0, 0. + SGD(VG, Co...v.....! L.)
end for end for return 0, +,
D MISSING PROOFS
Proposition 1. Let S E denote the user ID, X E Rr the input and Zz, Zg E RD, the latent representations of the two views of x given by the encoder with parameters 6. Given critic function f: D: x RDD + RR we have that
K
exp(s(zık: 226)) Σ
[o(2;; 22(s) 2 E Ep(s)pa (n 221s)
log
(14)
K
exp(s
K
17

---

Published as a conference paper at ICLR 2024
Proof. The proof follows Poole al. (2019). We can show that
8 log Pe 222
Ia :o(zi:z210 k2|s) = pp (#11 (M w2l.1/////pssp....... (s)
(15)
pa (2(,222 ())0(21,115)
log (21,1: 82
Plle)po(2 2210)
(16)
Po (11,111 (s)
Po (21,111 22, 8 ( / log
(17)
1: 22, Po (21,1:K|8)
(Z1,11:K /22,
log
Po (0(,1,1:
log Po Z1,11K1 22, 8) (((,111K 22,
122.0)
(18)
log (Z1,11k 82)
Ee(s) (m m2|s)
Po (0(
(p a)
[DKI. (po (21,1:K 12, (l(((11,1 (22,,
(19)
log (Z1,1: 22,
2 Epte)paleen
(20)
pe (0(,1,1:K(5)
and then by parametrizing qız1,1. 12, s) in terms of critic function f.
((21, po (s) ex Z1.1:K))
(21)
[epo(e,,uis]]expp(( (exp( (22, 21,1:K))
we have that
log exp(f(Z2, 21.1:k)) E (x1 [exp( (Z2, 21.1:
lo(21i22(8) 2 Ep(s)potan (2.l)))
(22)
Since the denominator depends on the aggregate score exp(f(zg, 21,1:K)) over pe (s), which similarly intractable, we can introduce one more lower bound that will allow us work with minibatches of data Poole et al. (2019). Due to the positivity of the exponent, we have that for any >>
E (po(en.)))))) [exp(f(z2, 21.1K)]] + log 1. (23)
log E Eo ( (xle) (22,(((2221,,1xx))) <
Using this bound with exp(1), we have that
(2(2:22)8)) 2 Ep(e)po (21:x log ( 21,1:K)]]
exp(- (xp(--))nttnpoooalllee [x
(24)
We can now set f(z2, 21.1:) as Poole et al. (019
f(22, 211kk) > + f 21,1) - log (zz,, 21,1:).
(25)
In this way, we end up with
log exp 21.1))
Ia 22|s) 2 1 (ppsspooeea
al22, 21.1:K)
eep( Z1,1))
E (p(s)po(ka (om
(26)
a Z2, 21,1:K
18

---

Published as a conference paper at ICLR 2024
We now average the bound over K replicates and reindex 21
K
[1og exp(S 21,1))
T0
2 +
K
alz2, 21,1:K)
k=1
ex 21,1))111111
E pp(s)pp (xals)))o (x1,)l)))
al22, al22, 21.1:k)
(27)
exp( z1,1)) log
1
E (1
K
alz2, 21,1:K)
k=1
1
exp(f 21,))
(28)
K
ol 21,1:K)
K
21,)))
1 + Ept (p: 21
log
K
al22 21,1:K)
exp( exp(f(Z2,,11.)))
E (pp
(29)
K
21,11
and for the specific choice of a(z,, 11 i.c.
K EK (xp(((22,71,,)) we have that terms cancel,
exp(s 21,k))
E pls) (2)
Σk k==1 exp( (22, 21,x))
K
exp( 21.k))
p (pa pee (xxx0
1.
(30)
k= exp( / 21,)))
In this way, we end up with the well known InfoNCE loss Oord et al. (201)) where now we contrast
between datapoints that share the same class
K
exp(J 22x))
Le (e(zi:zzls) 2 E (p(s)pa(a mgls)ii
log
(31)
K
I K Zド=1 exp(f(z15. 22n))
k=
Lemma 2.1. Let E N denote the client ID, X E R.D. the input and zı RDs the latent representation of a view of x given by the encoder with parameters 0. Let & denote the parameters of client classifier (s(z)) that predicts the client ID from this specific representation and let H(s) be the entropy of the client distribution p(s). We have that
La(z1;s) 2 (p(s)p((r))))) [logr((s/z1] + H(s)
(32)
Proof.
log (8(8 21)
E (p())p((a)s)) log (a(s)2))
La (z]:8) = Epo(s.2))
(33)
p(s)po (z
p(s)
log r (1(21)
(p(s)po((a1l))
Ep(s) (DoL((((sl))))lrrr6
(34)
p(s)
rs(s|21)
2 (p(s))(((11s))
log p(s)
(p(s)o(()))))) [logro(s/zi) H(s).
(35)
Lemma 2.2. Let E N denote the user ID, x E RD the input and Z1, E RD the latent
representations of the views of x given by the encoder with parameters B. Let 6 denote the parameters of client classifier (2)22) that predicts the client ID from the representations. We huve that
T To(215 -Pptteppo
[log rs [22)
(36)
19

---

Published as a conference paper at ICLR 2024
Proof.
Lo(z); 8/21))/222))))))))) = Ho(s|z)) Ho(s/22, z1)
(37)
Hg(s)z)) H(s) [z - s) [logra ra [22)]
(38)
are the conditional entropies of & given Z2 where H(s) is the entropy of p(s), Hp(s) 22). Ho(s|z2, 2i) and 22, and the last inequality is due to the lower bound of lemma 2. We also used the fact that the entropy of a discrete distribution is non-negative.
Proposition 2. Consider the label skew data aata-generating process for federated SimCLR from Figure
i with E N denoting the user ID with H(s) being the entropy of p(s), x E RD= the input, Z1, Z2 E D= the latent representations of the two views of given by the encoder with parameters Let y be the label and let ra(s)z,) be model with parameters & that predicts the user ID from the latent representation Zi. In this case, we have that
La(z1:0) y) + Le(z2: v) (p(s)p((x ma|s) [log (s(z))) log (oggz()))22] 2H(s).
(39)
Proof. The claim is consequence of the data processing inequality. We start by noting that
3)) + (o(;;; s|y) (z s) = Lo(z):)) s) + Lo(2::yls)
(40)
and since this graphical model we have that TT zıly. so [s; 21 = 0, we end up with
To(21; v) To [21 8) + Tz(2):0|s)
(41)
Ta (2):8) 2 (2(s))o(z)1e) ( (8/21)] H(s),
(42)
where we use the positivity of mutual information and our lemma 2.1 In a similar manner we can also show that
Ie(z2:))
E (p())po(a)(s)) [log (ogra(s(zp)) [2)) H(s).
(43)
By adding up and eq.
42
43 we arrive at the claim.
20


Reviews:
[{"review_id": "OxGnJIlmTZ", "reviewer": "Reviewer_JEx9", "summary": "This work extends SimCLR for federated learning, emphasizing multi-view mutual information maximization. It connects contrastive representation learning with user verification and introduces a user verification loss to improve global multi-view mutual information. Additionally, the study extends SimCLR to the federated semi-supervised setting, achieving a supervised SimCLR objective with specific modifications. The research explores the impact of non-i.i.d. data on federated unsupervised learning and shows that the global objective has mixed effects depending on the source of non-i.i.d. data.", "questions": "- How does the proposed MI LB/relaxation work if we move slightly away from SimCLR and look at related objectives: InfoNCE, or even non-contrastive ones like Barlow Twins? Are federated versions of these similar to Federated SimCLR?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem of pretraining large models in a federated setting is quite important and has seen little progress so far.\n- The proposed LB on the global multi-view objective is principled and as the authors show amenable to federated training. \n- Experiments in the semi-supervised setting are a nice addition to the paper, and clearly shows that their objective can be built upon.", "weaknesses": "- The paper lacks convergence analysis of their optimization algorithm, which is quite common in FL papers. \n- Experiments on more challenging/heterogeneous benchmarks like ImageNet are missing. \n- Discussion on how their objective can be adapted to other centralized pretraining objectives is missing. (See questions)\n- (Minor/Nit) Proposition 2 need not be stated, it follows immediately from previous Lemmas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OKuLy3oRKr", "reviewer": "Reviewer_wQ6s", "summary": "The paper presents a federated variant of SimCLR for unsupervised representation learning. It motivates its approach by a mutual information argument: since in standard SimCLR the goal is to maximize the mutual information (MI) between the two generated views, this MI can be decomposed into a local variant that corresponds to local SimCLR and two excess terms that need to be bounded. The first relates the mutual information between the first view and its local client, which is lower bounded using a classifier that seeks to predict the client ID from the first view. The second term relates to the additional or excess mutual information of the second view on the client which can be upper bounded by a second classifier that predicts the client ID from the second view. In addition, the paper presents a semi-supervised variant of this approach.", "questions": "**Question:**\n- How does the semi-supervised variant of SimCLR relate to pseudo-labeling approaches, such as distributed distillation [2] and federated co-training [1]?\n\n**Detailed Comments:**\n\n- Please discuss the empirical results, in particular the fact that FedSimCLR is not outperforming the baselines, in more detail. The proposed method does not have to outperform the baselines, as long as the benefits and limitations of it in comparison with existing methods are properly discussed. To stress this point: it has become usual to require papers to have large tables where the proposed method has \"the best number\" in each row, but this just promotes scientifically questionable practices to improve the numbers. I am happy that this paper presents more interesting results, but they require, unfortunately, a more thorough discussion. One could even, for example, use mutual information as quality measure (where one would probably rely on the more tractable Wasserstein dependency measure [6], isntead of approximating MI).\n- Please state what exactly the supervised baseline in your experiments is (I assume FedAvg). Please compare to (one of the) FL variants for non-iid data, such as FedProx [4], FedBN [5], and SCAFFOLD [3], as baselines.\n- For the semi-supervised setting, please compare to pseudo-labeling approaches [1,2] as semi-supervised baselines.\n\n\n[1] Abourayya, Amr, et al. \"Protecting Sensitive Data through Federated Co-Training.\" arXiv preprint arXiv:2310.05696 (2023).\\\n[2] Bistritz, Ilai, Ariana Mann, and Nicholas Bambos. \"Distributed distillation for on-device learning.\" Advances in Neural Information Processing Systems 33 (2020): 22593-22604.\\\n[3] Karimireddy, Sai Praneeth, et al. \"Scaffold: Stochastic controlled averaging for federated learning.\" International conference on machine learning. PMLR, 2020.\\\n[4] Li, Tian, et al. \"Federated optimization in heterogeneous networks.\" Proceedings of Machine learning and systems 2 (2020): 429-450.\\\n[5] Li, Xiaoxiao, et al. \"FedBN: Federated Learning on Non-IID Features via Local Batch Normalization.\" International Conference on Learning Representations. 2021.\\\n[6] Ozair, Sherjil, et al. \"Wasserstein dependency measure for representation learning.\" Advances in Neural Information Processing Systems 32 (2019).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- unsupervised federated representation learning is an important and interesting use-case\n- the method theoretically motivated and sound", "weaknesses": "- some baselines for semi-supervised learning, and some proper supervised baselines are missing.\n- The empirical results show that the proposed federated SimCLR variant is only en par with spectral constrastive learning when using a user verification loss. This is not properly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OU5ouwAYr3", "reviewer": "Reviewer_3B5H", "summary": "The paper presents an extension of SimCLR, a contrastive learning framework, to the federated learning (FL) setting, with a focus on mutual information maximization (MI) across multiple views.\nThe authors establish a connection between contrastive representation learning and user verification and propose a method that incorporates a user verification loss into each client’s local SimCLR loss, resulting in a lower bound to the global multi-view mutual information. \nAdditionally, the paper extends the approach to the federated semi-supervised setting, introducing modifications to accommodate labelled data at the clients and proposing an auxiliary head for label prediction. The paper also investigates the impact of different sources of non-i.i.d. data distribution on federated unsupervised learning performance.", "questions": "Additionally, the proposed multi-view MI estimation might result in additional computation overhead.  This is particularly crucial in FL where computational resources are may be limited. Therefore, the paper could be significantly enhanced by a more thorough analysis and discussion of the trade-offs between performance and computational complexity.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The extension of SimCLR to the federated setting and the exploration of MI maximization in this context is particularly given the increasing interest in FL.\nThe paper provides a theoretical foundation for the proposed methods, including the connection between contrastive learning and user verification, and the derivation of a lower bound to the global multi-view MI.\nThe authors conduct both unsupervised and semi-supervised experiments, providing a thorough evaluation of their proposed method.", "weaknesses": "The theoretical derivations, propositions, and lemmas mainly connect to and extend existing methods, which might be perceived as a lack of novelty. For example, the idea of decomposed MI in (1) and (2) has been presented in Sordoni et al. (2021). The proofs of propositions and lemmas are quite standard which mostly follow the existing approaches in literature. It would be more beneficial if the authors can clarify the unique aspects and advantages of their approach, and clearly differentiate it from existing methods.\nFurthermore, the authors only provide analysis for a two-view setting, which might not be completely satisfied with the proposed multi-view MI.\nThe paper does not present algorithms for federated training, which is crucial for practical implementation. Moreover, as a federated learning algorithm, there should be a thorough analysis of the convergence guarantees, which seems to be lacking.\nThe experimental setup presented in the paper demonstrates a certain degree of comprehensiveness; however, it appears to be somewhat limited in terms of diversity. The FL baselines utilized in the study are mainly adaptations from centralized methods, which may not fully represent the state-of-the-art in unsupervised representation learning within the FL context. \nLooking at the results outlined in Tables 1 and 2, it becomes evident that in a majority of the scenarios, the performance of the proposed method is either on par with or falls short of other unsupervised baselines. This observation raises questions about the clear and tangible benefits of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)