FedProp: Cross-client Label Propagation for Federated Semi-supervised Learning
Authors: Jonathan Scott
Michelle Yeo
Christoph H Lampert
Venue: Submitted to ICLR 2023 (2023)
Keywords: federated learning
semi-supervised learning
label propagation
cryptographically secure computation

Abstract: Federated learning (FL) allows multiple clients to jointly train a machine learning model in such a way that no client has to share their data with any other participating party. In the supervised setting, where all client data is fully labeled, FL has been widely adopted for learning tasks that require data privacy. However, it is an ongoing research question how to best perform federated learning in a semi-supervised setting, where the clients possess data that is only partially labeled or even completely unlabeled. In this work, we propose a new method, FedProp, that follows a manifold-based approach to semi-supervised learning (SSL). It estimates the data manifold jointly from the data of multiple clients and computes pseudo-labels using cross-client label propagation. To avoid that clients have to share their data with anyone, FedProp employs two cryptographically secure yet highly efficient protocols: multi-party Hamming distance computation and secure aggregation. Experiments on three standard benchmarks show that FedProp achieves higher classification accuracy than previous federated SSL methods. Furthermore, as a pseudo-label-based technique, FedProp is complementary to other federated SSL approaches, in particular consistency-based ones. We demonstrate experimentally that further accuracy gains are possible by combining both.

Under review as a conference paper at ICLR 2023
FEDPROP: CROSS-CLIENT LABEL PROPAGATION FOR FEDERATED SEMI-SUPERVISED LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Federated learning (FL) allows multiple clients to jointly train a machine learning model in such a way that no client has to share their data with any other partic- ipating party. In the supervised setting, where all client data is fully labeled, FL has been widely adopted for learning tasks that require data privacy. However, it is ongoing research question how to best perform federated learning in semi- supervised setting, where the clients possess data that is only partially labeled or even completely unlabeled. In this work, we propose a new method, FedProp, that follows a manifold- based approach to semi-supervised learning (SSL). It es- timates the data manifold jointly from the data of multiple clients and computes pseudo-labels using cross-client label propagation. To avoid that clients have to share their data with anyone, FedProp employs two cryptographicaly secure yet highly efficient protocols: secure Hamming distance computation and secure sum- mation. Experiments on three standard benchmarks show that FedProp achieves higher classification accuracy than previous federated SSL methods. Furthermore, as a pseudo-label-base based technique, FedProp is complementary to other federated SSL approaches, in particular consistency-bass ones. We demonstrate experi- mentally that further accuracy gains are possible by combining both.
INTRODUCTION
Federated Learning (FL) is a machine learning paradigm in which multiple clients, each holding their own data, cooperate to jointly train a model. Training is coordinated by central server, which, however, must not have direct access to client data. Typically this not due to the server being viewed as hostile party but rather to comply with external privacy and legal constraints that require client data to remain stored on-device. FL has been receiving abundant interest in recent years as allows models to be trained on valuable data that would otherwise be inaccessible. To date, the vast majority of research within FL has been focused on the supervised setting, in which client data is fully labeled. However, in many real- worldw settings, this is not the case. For instance, in devsce FL. smartphone users are not likely to be interested in annotating more than a handful of the photos on their devices in a cross-silo setting the labeling of medical imaging data may be both costly and time consuming. As such, in recent years there has been growing interest in learning from partly labeled data in a federated setting.
In this work we propose FedProp, a method for semi-supervised learning (SSL) in the federated setting that follows manifold-based approach to pseudo-labeling client data. During each training round FedProp leverages the data of multiple clients to obtain an estimate of the data manifold which it then uses compute pseudollabels for clients' unlabeled data via label propagation. Using these pseudo-labels clients then train in a supervised manner for the remainder of the round. The motivation for this approach comes from the fact that the more data that available, the more densely we have sampled the manifold and therefore the better our estimates and pseudo-labels will be. Thus, it is crucial importance to be able to combine information from multiple clients, rather than treating each client's data in isolation. The key challenge lies in how to perform such cross- client eseudo-labeling the steps of which would normally require the data of participating parties to be shared in order to estimate the manifold and compute label propagation.
Our main contribution lies in the CrossClientLP subroutine of FedProp, which we propose to address this challenge. It uses locality-sensitie hashing and secure Hamming distance computa-

---

Under review as a conference paper at ICLR 2023
tion to efficiently estimate the cross-client data manifold. It then distributes the label propagation computation across clients and aggregates the results using secure sunmation. CrossCli innt preserves privacy, in the sense that it does not require clients to share their data with anyone else. At the same time, it adds only limited communication and computation overhead relative to popular federated learning methods such FedeentedAAeeraging-
Our experiments show that redProp outperforms all existing methods for federated semi-suuerviii learning, as well as a range of natural baselines in the standard CIFAR-10 setup. Going beyond prior work, we also evaluate redProp on more challenging datasets, namely CIFAR-100 and Mini- ImageNet, where we also observe substantial improvements in accuracy. Moreover, as a method for pseudo-labeling unlabeled data, FedProp is orthogonal to other approaches for federated SSL, in particular those based on consistency regularization. We demonstrate this empirically by combining FedProp with such an approach and observing that this often leads to further accuracy gains.
2 RELATED WORK
Semi-supervised Learning Semi-supervised learning (SSL) classical and well studied prob- lem in machine learning where the goal is to leverage both labeled and unlabeled training examples to improve performance on some task, see (Chapelle al. 2006) for full overview. In recent years there has been a great deal of interest in applying deep learning techniques to SSL. Broadly speaking such semi-super deep learning approaches can be categorized into two groups. The first group consists of methods that add an unsupervised loss term to the objective function. In particular many of these methods introduce some form of consistency regularization (Sajjadi et al., (016), which encourages the model to produce similar outputs for similar inputs, examples include (Tarvainen & Valpola 2017 Berthelot et al.) 2019; Xie et al. 2020). The second group consists of methods that exploit unlabeled data by computing pseudo-labbbe for unlabeled points and then training on these in a supervised fashion, for instance (Lee 2013 Shi et al. 2018 Iscen et al. 2019 Rizve et al. 2021
Combinations of both approaches also possible (Iscen al. 2019 Sohn et al. 2020).
Federated Learning Federated learning (FL) (McMahan et al. 2017) was originally proposed for learning on private fully labeled data split across multiple clients. For survey on recent devel- opments in the field see (Kairouz et al. 2021). A number of recent works propose federated learning in the absence of fully labeled data. When only unlabeled data is available. methods for cluster anal- ysis, dimensionality reduction have been proposed (Dennis et al.) [202]] Grammenos et al.) 2020). Federated superviedd learning (Zhuang et al. 2022] Makhija et al., (022) can also be performed where the goal is to learn representations of unlabeled data that can later be fine-tuned to other tasks. However, all of these settings are different from the task of semi-supervised learning, in which the goal is to directly learn better classifiers from labeled and unlabeled client data.
approach. peong et al. (202 propose inter-client consistency and parameter decomposition to separately learn from labeled and unlabeled data. Long et al. (2020) apply consistency locally through client based teacher models. Zhang et al. (2021) and Diao et al. (2022) focus on an alternative setting in which the server has access to labeled data. In this setting Zhang et al. (2021) combine local consistency with grouping of client updates to reduce gradient diversity while Diao et al. (2022) combine consistency, through strong data augmentation, with pseudo-labeling unlabeled client data. Other methods focus exclu- sively on pseudo-labeling: [Albaseer et al. (2020) and Lin et al. (202T) both use network predictions to assign pseudo labels, while Presotto et al. (2022) develop specialized method for human activ- ity recognition which uses label propagation locally on each client to pseudo label incoming data. Unlike in classical SSL, none of the above methods fully make use of the knowledge gained from estimating the data manifold because they exploit interactions between data points at most locally within each client. In contrast, FedProp uses securely computed cross-client interactions, thereby obtaining a better estimate of the data manifold.
For semi-supervised FL., several works follow a consistency-
PRELIMINARIES AND BACKGROUND
We assume a federated classification setting with m clients coordinated by a central server. Each (1)
client possesses partly labeled data (xf) 2 x()) with x() (x=(1) (.. , ) 12 =
2

---

Under review as a conference paper at ICLR 2023
(a) Label propagation. Left: dataset with 2 labeled data points per class (colored) and many unlabeled ones (black). Right: soft labels for the unlabeled data points are estimated by propagating label information along 10-nearest-neit neighbo graph.
(b) Locality sensitive hashing. The expected number of random hyper- planes that separate two data points (here: 2 out of 10) is proportional to the angle between the points.
Figure 1: Illustration of label propagation and locality sensitive hashing
Algorithm 1: Labe aabelPropagation
Input: data matrix V, matrix of partial labels Y
1 A + matrix of pairwise cosine similarities
Aij
(o I[ ||
Ay
if Ag E topR+1 ((Aie) and + j.
2 B sparsify to k-NN graph by Bi
otherwise
3 W < B BT
4 W < D D- for D diag(W1,,)
5 7 < (Id,, aY) 'Y
6 2 < normalize via Zij = Zi/((E Zic)
7 compute labels via 0; arg max [C] Zio
8 W + compute weights via w; = 1 - H / log C' with H(p) = = -Ejejo Pj logps.
Output: pseudo-labels ÿ. weights
(v))) ()) ..... Mren and X = (x (1 no0) ). Here (60) and n²() denote the number of labels
(
and the total number of data points owned by client j. Let D denote the data dimension, so that (i)
a E PRP. and the number classes, so that y E [C] == (l...., ......,G ,J,. We further denote by So : D - R a parametric classifier of the data, with parameters which in our case is deep neural network. We decompose fo into feature extractor fo RDD + Rd and a classifier head fo Rdd + R so that fo = fw fo with 0 = (vb Φ). In standard fashion, fo will be trained by stochastic gradient optimization of loss function L(X, w,w;), = D (x;))) y:), where w; are per-sample weights and é is the cross entropy loss.
Our aim to learn the classifier So by making use of all client data, both labeled and unlabeled, while respecting the key principle of federated learning setting, namely that clients should not be required to share their data with anyone else. To this end, we build on two established techniques: label propagation and secality-sensitiie hashing, which we now introduce.
3.1 LABEL PROPAGATION (LP)
Given partly labeled data, LP is a classic technique for assigning (pseudo-)labels to the unlabeled data. We describe variant of the graph-based diffusion approach of et al. (2004). The key idea to propagate label information from labeled to unlabeled points taking into account the geometry of the underlying data manifold, see Figure Ta for an illustration. The manifold is approximated by a neighborhood graph of the data, in which points are connected based on how similar they are.
Algorithm I describes the procedure in pseudocode. As input, it takes set of data vectors, V V. Vu {.I.... Vi, U1+1.... en and partial p labels (VI...., M/ from C' classes, which we encode in one-hot matrix form: Y xxx with Yic (g:=e) for and Y,, = 0 otherwise. The algorithm starts by computing matrix A Ruxn of pairwise similarities between all data
3

---

Under review as a conference paper at ICLR 2023
Algorithm 2: FedProp
ym Input: labeled data (XF,) Mr. j==1 unlabeled data ( )=
// stored on-device at clients
1 f + initialize randomly
2 8 FederatedOpt imi immpttioo v
I/ initial training on labeled examples
3 for round t [T] do
server 4 e randomly selects Tm clients
5 Server broadcasts 0 to each client in P
6
for client j P in parallel do
v(i fx((() with X = x(6 U ux( yú), Server) // compute pseudo-labels train with pseudo-labels
x(i) embed labeled and unlabeled data
7
8
gi),wuh CrossclientLP (G) + cl ClientUppatee(X... ,06),, wl) E2
9
10
Client j sends B60 to the server
11
0 + ServerUpdate ((()) SEP))
Output: model parameters 0
points (line A is sparsified by keeping only each point's k nearest neighbors, symmetrized, and then normalized (lines 4. resulting in a transition matrix, W. Label propagation now amounts to repeatedly spreading label information across the resulting graph by multiplying with aW until convergence, where a E 1) is a hyper-parameter reflecting how much the algorithm should weight the original labels. 'he outcome of this can be computed in closed form (line 5 The resulting matrix, Z E TR=xC_ can be interpreted as unnormalized class scores for each data point. Normalizing Z across rows yields, for each point, a probability distribution across all classes (line 6). From this, one assigns a mexiiuu--likelhhen label to each point (line 7 as well as a weight reflecting the confidence in the assignment based on the entropy of class probabilities (line
3.2 LOCALITY-SENSIIN HASHING (LSH)
LSH (Indyk & Motwani 1998) is procedure for hashing real-valued vectors into binary vectors while preserving their similarity. Let v IR be vector. To encode v into a binary vector b of length L, LSH randomly samples L hyperplanes in Rd. For each hyperplane it checks whether e lies above or below and sets the ith bit in b as 1 or O accordingly. Formally, b; = 1 (m,a)200 where u, E RR the normal vector of the ith hyperplane. A key property of LSH is that it approximately preserves cosine-simin cosine-similarity. Concretely, for vectors 11, 02 with L.H encodings b1, b2, one has (1) [ R
cos(T H (b1, b2) where H is the Hamming distance (number of bits that differ) between two binary vectors. The reason that the probability of b and b2 differing at any bit the probability that the ith sampled hyperplane lies between ij and which is equal to Z(U1, 22)/ T,, see Figure lb By the law of large numbers, the more hyperplanes one samples, the better the approximation quality.
FEDPROP: LABEL PROPAGATION FOR FEDERATED SSL
In the following section we describe our approach the problem of federated semi-supervised learn- ing, FedProp, shown in pseudocode in Algorithm 2 follows a general FL template of alternating lo- cal and global model updates. As such, is compatible with existing FL optimization schemes such as FederatedAveragingg (McMahan et al., 2017). FedProx (Li et al. 2020). or SCAFFOLD (Karim- liredyy et al. 2020). The choice of scheme, which we refer to FederatedOpt imi imatiiionn, determines the exact form of C1 i lentUddate and ServerUpdate.
The first stage of FedProp (lines 2 to initialize the model with federated training on only the labeled examples. The second stage takes place over T rounds. To start each round the server samples some fraction T of the clients. Each sampled client gets the current model parameters from the server and embeds its labeled and unlabeled data with the feature extractor fo- Clients and server then collaboratively execute a cross-client label propagation step on these embeddings (line [), which we discuss in detail in Scction[4.1 As output of this step each client gets pseudo-label. labell and weights for their unlabeled data, which they then use to run E epochs local supervised training. Finally, clients send the updated local models to the server which aggregates them (lines 11.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 3: CrrossClientin
Input: set of clients P, client data (VO), Y()) jEP
// data stored on-device at clients
1-XS: Setup: clients exchange private and public keys, agree on random seeds
2-CS: for client P in parallel do
3-CS: II t random projections
II same II for each client
4-CS:
B6) + LSH(V
5-XS: H SecureHlamingg( (BB))
I/ server gets Hamming matrix
6-S:: t compute cosine similarity matrix from H
// line
7 of Alg.
7-SS: B < sparsify A to k-NN graph
// line
2 of
7
8-SS: W
// line
of Alg.
7
9-5S: W + D -D-
// line
of Alg.
10-SS: S - -Idd, - -W)-
#/ line of Alg. (srveer part)
11-CS: for client P in parallel do
12-CS:
s(i) - labeled-cols,
// client gets columns corresponding to labeled data
20) s
13-CS:
// line
of Alg.
(per-client part)
14-XS:
Z6) + SecureSum/ (rows, (2(5)
1l line
5 of Alg. (cross-client part)
15-CS:
20) + normalize(Z())
// line 6 of Alg.
16-CS:
g()) w() < pseudo-labels and weights
// lines of Alg.
Output: pseudo-labels and weights (i/1), (()))
I/ available only to respective clients
4.1
CROSS-CLIERS LABEL PROPAGATION
To compute pseudo-labels, would like to execute label propagation hood graph built from the embedded data of all currently selected clients. The challenge lies in how to carry out this computation. By default, Algorithm would require the clients to share the feature embeddings of their data in order to compute the similarity matrix A (Iine and to share their labels when computing the class score matrix Z (line 55). thereby violating the constraints of the federated setting. A naive solution to overcoming this problem would be execute Algorithm| [ i its entirety using multi-party computation (MPC) Yao 1982; Cramer et al. 2015), a set of cryptographic tools that-in principle- allow participants jointly compute any function in way that leaks no infor- mation to any of them. However, for complicated functions of vell-valudd inputs, such as matrix inversion, MPC has high computational and communication overhead and would not be practical.
cross-client neighbor-
Our main contribution lies in the CrossClient routine, which allows secure yet efficient com- putation of pseudo-labed based on cross-cliem label propagation. The key idea is to take hybrid approach and divide the steps of the LP algorithm into three groups: steps that clients execute locally using only their own data (client steps, CS), steps that the server executes on aggregated data (server steps, 5S), and steps that require computing quantities based on cross-client data (cross steps, XS).
Adopting the viewpoint in which the server is non-hostile, we trust the server execute its steps on non-revealing aggregate data and return correct results of computations! Client steps are also unproblematic from privacy viewpoint, so what remains is how execute the cross steps efficiently yet without leaking client data. To do so, we transform the problem into form that is equivalent but allows for the application of tailored cryptographic tools, that are highly efficient yet fully secure.
Algorithm shows pseudo-code for CrossClientio For simplicity of exposition, we describe as if communication between clients were possible, which a reasonable assumption in a cross- silo setting. If this is not possible, as is likely the case in cross-device FL., then communication orchestrated by the server, see e.g. [Bonawitz et al. (2017). First, the clients use secure key exchange procedure to agree on shared random seed that remains unknown to the server (line []. This is common step in federated learning when cryptographic methods are meant to be employed, e.g. the description in Bonawitz et al. (2017). The clients use the agreed-on random seed to generate a common set L random projections (line 3). Each client then uses LSH to hash its
In particular do not consider malicious servers in the cryptographic sense, which would be allowed to employ attacks such as model poisoning or generating fake clients in order to break the protocol.
5

---

Under review as a conference paper at ICLR 2023
feature vectors into binary representation (line +. this is the most efficient input format for cryptographic computation. The key insight here is that all clients should use the same random projections. That way, it is possible to recover the cosine similarity between any two feature vectors, even across clients, from the Hamming distance between their binary representations. To securely compute all Hamming distances, each pair of clients run suitable cryptographic routine, such as SHADE (Bringer et al. 2013). From this, the server receives a matrix, H E N4xx, for n = EjeP n0)) containing the Hamming distances between all pairs of embedded feature vectors across all currently selected clients (line [5 Crucially, no clients need to share their data to make this possible and the security guarantees of the cryptographic protocols mean that participating party (server or client) can learn anything from intermediate stages of the computation. With H the server can now execute number of steps by itself (lines T0)::: it converts H to an estimate of the cosine similarity matrix A using the relation from Section 3.2 it computes the normalized and symmetrized nearest-neighbors matrix, and then inverts that to obtain the propagation matrix, S E Rux*.
The next step of label propagation would be to compute Z = SY, where the matrix Y E RR"xC' contains the label information of all participating clients. To do this without the clients having to share their labels, we express the computation as Z = ΣeEP s(il where 5ti E R2xn() is the sub- matrix of S consisting of only the columns that correspond to the data of client j. In fact we can refine this further by observing that by definition the rows of yot) that correspond to the unlabeled data of client j are identically 0 and hence do not contribute to the multiplication. We therefore let y 100' be the rows of y corresponding to labeled points and s E R be the corresponding columns (ti) and we still have that Z = E T. This observation has allowed us to distribute the computation of Z among the clients while minimizing the size of the matrices that must be transmitted. Therefore, each client P receives 5(6) from the server (line 12 and locally computes Z(i) 5 E B (line 13). which reflects the influence of j's labels on all other data points. By now we have computed 2, but the result is additively split across the clients since Z ΣSEP Z Each client & requires only the rows of Z that correspond their
own data points which we denote by Z(k) E "=,"'xc, This is computed by client using secure summation routine over these rows of the matrices held by all other clients in P. Note that secure summation commonly used in FI. for model averaging (Bonawitz et al. 2017). Finally, each client
k locally computes their pseudo-labels labels and weights from zkk) (lines 15
16
4.2 ANALYSIS
In this section, we analyze the correctness, privacy, efficiency and robustness of Algorithm 3
Correctness Algorithm 3 implements the same computation as Algorithm D except it estimates cosine similarity via the Hamming distance of the LSH binary vectors. Thus, the output of Algo- Jappmoximmo that of Algorithm The approximation quality depends on L, the LSH vector length. In practice, we observe no difference in behavior already for small values, e.g. L = 4096.
Privacy The main insight is that Algorithm adheres to the federated learning principle that clients do not have to share their data or labels with any other party. This is ensured by the fact that all cross-client operations are computed using cryptographically secure methods. The only information seen by the server about client data is contained in the matrix of Hamming distances H, which approximates the matrix W of cosine similarities between client feature vectors. While certainly influenced by the client data, we consider W (and therefore H) rather benign object for a non-hostile server to have access to: first, the similarity is not computed between input data itself, but its feature representation according to the current model. Second, even the feature vectors could not be reconstructed from W because cosine similarity depends only on angles, so any rescaling and rotation of the feature vectors would result in the same W matrix. Clients do not see W but they some of the columns of S. These reflect how their labeled data can influence all other datapoints according to the data manifold estimated from the participating clients' feature vectors. However, this influence is unnormalized and hence the influence relative to other clients cannot be known.
Efficiency Computational
Ordinarily, LP imposes only small overhead compared to the computational cost of training the deep network on the clients. That is because the matrices makes use of scale with the number of datapoints, rather than the (much larger) number of network

---

Under review as a conference paper at ICLR 2023
parameters. The same holds for CrossClientle which- wwtt the exception of the additional LSH steps- pprffrrms the same computational steps as ordinary LP. Furthermore, the most expensive steps of normalization and matrix inversion are carried out by the server. LSH itself requires only one additional matrix multiplication per client which adds negligible overhead on top of the computation of client features. By design all cryptographic primitives
chosen allow maximal efficiency. routines, such as SHADE (Bringer et al.) 2013), rely on oblivious transfer (OT) (Ishai et al. 2003 Roy 2022) as the only cryptographic primitive. Highly optimized libraries exist for this, such as libOll (Peter Rindal), which allows for hundreds of thousands of OT operations per second. Secure um can be implemented efficiently using, e.g.,
For the SecureHanming ng step, special-purpos
secrel sharing (Shamir, 1979) or garbled circuits (Yao 1986). A similar secure summation step is already used in FL. to avoid information leakage during model averaging (Bonawitz et al. 2017). A number of libraries exist that are tailored to this task, for instance SAFELeam (Fereidooni et al., 202I), which allows for aggregation of tens of megabytes per second. The computational overhead added by both of these methods is typically small compared to the training of the network itself.
incurs additional communication costs at two steps of Algorithm 3 Let p = [P] be the number of participating clients and EjEp n2() the total number of data points they possess. For computing the Hamming matrix, each client & has to communicate (in encrypted form) (p - In bits to other clients, and the server receives n (*)) values. To propagate the labels via the distributed matrix multiplication, each client first receives from the server a matrix of size n ((k) and then transmits a matrix of size n(d) C' to every other clientj j E }. For example, in our CIFAR- 10 experiments, in total this adds up to approximately 2.5MB of data per client, whereas transferring the model to the client and back requires 24MB of payload per client. Thus, in this setting, FedProp adds an overhead of about 10% over pure FederatedAveraging. When using larger networks the cost of sending model updates becomes even more dominant, e.g. in our MinilmageNet experiments the overhead drops to approximately 4%.
Efficiency - Communcation
CrossCliente
Robustness In cross-device FL clients may be unreliable and prone to disconnecting during train- ing. Therefore, it is important that FL algorithms can still execute and make progress even in the event of client dropouts. This is indeed the case for FedProp: a client dropping out before the Secure Hammm ng step (Algorithm 3 line 5) is equivalent to it not having been sampled by the server in the first place: its data will not be used for this round of label propagation and the sub- sequent model update. Because the Hamming computation is executed pairwise, a client dropping out during this step has no effect on the computation of other clients. The result will be missing entries in the Hamming matrix, HI, which the server can remove, thereby leading the same out- come as if the client had dropped out earlier. If clients drop out after H has been computed, but before the Secure um step (Algorithm{) line 14]. they will have contributed to the estimate of the data manifold, but they will not contribute label information to the propagation step. This has the same effect as a client only had unlabeled data. The exact effect of dropouts during SecureSum depends on the characteristics of its underlying cryptographic implementation. For example, when using (k, n) -thresholld secret sharing, the computation can be completed as long as at least k out of clients participate until the end (Shamir 1979). Any later dropout will only result in that client not contributing to the following model update step, but it will not affect the other clients' computations.
5 EXPERIMENTS
In this section, we evaluate the accuracy of FedProp against other methods for cemi-supervised FL as well as report on ablation studies. As our emphasis here is on accuracy, not real-world efficiency, we use a simulated setting of federated learning. rather than distributing the clients across multiple devices. Therefore, we also use plaintext placeholders for the cryptographic steps that have identical output. Source code for our experiments will be made publicly available.
5.1 EXPERIMENTAL SETUP
Datasets We evaluate FedProp three standard datasets for multi-class classification: CIFAR- 10 (Krizhessky/ 2009). which has 10 classes and is used in previous federated SSL works, as well as the more difficult CIFAR-100 (Krizhevsky 2009) and Mini- Mini-ImageNet (Vinyals et al., 2016) which both have 100 classes. To the best of our knowledge ours is the first work in this federated SSL setting to evaluate these more challenging datasets. All three datasets consist of 60,000 images
7

---

Under review as a conference paper at ICLR 2023
which we split into training sets of size N == 50,000 and test sets of size 10,000. From the training set, Nr examples are labeled and the remaining N - -Ny are unlabeled. For CIFAR-10 10 we evaluate with Nr. 1,000 and 5,000. For CIFAR- 100 and Mini Mini-ImageNet we take Nr. = 5,000 and 10,000.
Federated Setup We simulate a FL scenario by splitting the training data (labeled and unlabeled) between clients. my of these have partly labeled data, while the others have only unlabeled data. Each client is assigned a total of N/m data points of which Nr,m,, are labeled the client is one of the m which possess labels. We simulate statistical heterogeneity among the clients by controlling the number of classes each client has access to. In the i.i.d. setting all clients have uniform class distributions and receive an equal number of labels of each class. In the non-i.i.d setting we assign a class distribution to each client and clients receive labels according to their own distribution.
Networks Following prior work, we use 13-layer CNNs (Tarvainen & Valpola, 2017) for CIFAR- 10 and 100 and a ResNet-18 18 (He et al., 2016) for Mini-ImageNe Feature extractors are all layers except the last fully connected one, thus embeddings have dimension 128 and 512, respectively.
Hyper-parameetrs We use Federat RederatedAver edAveraging as the FederatedOps imi zat ion scheme. We choose hyper-parameters for all methods based on training progress (LSH dimension, k-NN parameter) or accuracy on held-out validation set consisting of 10% of the training data (batchsize, learning rate). Detailed values are provided in the appendix.
Baselines We compare redProp to a broad range of other methods, both from the existing feder- ated literature as well as our own baselines. For fairness all methods use the same network archi- tectures, and hyyer-ppaameete
are chosen individually to maximize each method's performance. As representatives of consistency regularization we report results for three methods: FedMatch (Jeong et al. 2021), FedSiam (Long et al.] 2020). and FedArg+ PedAAvgMMT, MMT, where the last is own adaptation of mean teacher (MT) (Tarvainen & Valpola 2017) to the FL. setting: In each training round the server broadcasts both a global (student) model and teacher model to the clients. Clients locally train the student model and update the teacher model as the empirical moving average of the student. Clients then return both models the server which separately averages the local students and teachers. To reflect pscudo-label (PL) based approaches, we include three methods: Feddvg++P[nnewoork re- peatedly uses network predictions to pseudo-label sampled client data. FedSem+, which is based on FedSem (Albaseer et al., (020), computes all pseudo- only once during training, but it addi- tionally uses entropy-based sample weights, as we found this to consistently improve performance. CodAAv++I[(lcaall
based on FedHlu (Presotto et al., 2022). uses per-client label propagation,
otherwise it is identical to Algorithm B Additionally, we report results for training in a supervised manner only the available labeled examples, FedArg (labeled only). This provides a lower bound for all SSL methods which have access to both the labeled and unlabeled data held by the clients.
Combinations As label based approach, FedProp is orthogonal techniques that utilize unlabeled data by modifying the loss function, such as the consistency regularization in FedAvg+ Movgg+MM To demonstrate this, we also report results when combining both approaches as FedProp +MT.
5.2 EXPERIMENTAL RESULTS
We report the results of our experiments in Tables I and 2 as the average accuracy and standard deviation over three random splits of the data for each setting. Table [ provides a comparison of FedProp to other approaches and baselines in the standard setting of CIFAR-10 with 100 clients, which has been previously used evaluate FL and SSL methods. In each case, we report results when 1,000 or 5,000 of the data points are labeled. Either all or half of the clients have labels, which are either i.i.d. or non-i.i.d. across clients. In all cases but one, FedProp achieves the best results among all methods, in some cases by a large margin. The most competitive alternative is the FedAvg+ +MT baseline, As a method based on munsistency-regularizatn with our paeudo-label based approach and we observe that this combination (FedProp+MT] achieves even better results across the board. Table Table[]reports reports on the harder situation with many more classes, for which the previous methods have not been evaluated. Again, FedProp achieves better results than the baselines here, and for CIFAR-100 can be further improved by combining it FedProp+ + MT.
it can be readily combined
Besides a ranking of methods, Tables 2 also provide some insights: First, pseudo-labels based on LP tend to help more than those based on network outputs. Presumably, this is because they do not just reinforce information that is already in the classifier but potentially also correct prediction error
8

---

Under review as a conference paper at ICLR 2023
Table Classifical accuracy [in %] on CIFAR-I0 (average and standard deviation across three runs)
CIFAR-10, L..dd (m T00)
m.
ML
50
Method
N₂ = 1000
NŁ = 5000
N₂ = 1000
N2 = 5000
FedAvg (labeled only)
55.46 + 0,43
76.13 0.46
56.97 - 0.59
80.36 0.07
Fedevv-)L.nnetwork)
60112
79.45 0.31
59.14
81.01 0.20
FedAvg-+P((lccalLP)
2.22
85.11
65.29 1.52
84.41 0.25
FedMatch
0.56
72.22 1 0.14
57.10
77,80 0.32
FedSem+
59.98 0.49
79.49 0.15
59.67
69.67:0... 0.47
80.94 0.25
FedSiam
67.02 0.98
82.06 0.56
62.98 1.61
78.45 0.34
FedAvg+MT
62.37 1.69
84.92 0.64
70.14 1.87
85.34 + 0.18
FedProp (ours)
70.91 + 0.71
86.65 0.16
70.81 1 1.65
86.29 1 0.34
FcdProp+MT (ours)
72.58 0.36
88. 17 + 0.18
73.63 1.99
87 54 + 0.14
CIFAR-10, non-i.idd (m 100)
m.
mL 50
Method
N2 = 1000
N2 = 5000
N2 1000
N2 = 5000
FedAvg (labeled only)
50.94 + 0.14
75.34 1.38
53.26 + 0.69
79.65 0.12
FedJvvg-PL(network))
60.60 + 0.60
80.07 0.53
59.82 + 1.05
81.14 0.23
FedAvg-PL[(locall
50.94+0.14
76.61 + 1.50
53.26 0.69
79.65 0.12
FedMatch
50.71 1.57
71.99 = 0.70
0.70
0.86
66.37 0.41 0.41
FedSem+
60.93
79.70 0.78
59.74
81.30 0.09
FedSiam
67.85 0.26
82.23 0.46
62.29 1.84
78.84 0.72
FodAvg+MT
64.54 +
84.07 0.60
67.40 + 0.53
85.58 + 0.35
FedProp (ours)
73.76 1 0.71
85.53 0.56
70.01 1 1.29
85.42 1 0.43
FedProp+MT (ours)
70.56 1 0.75
88.47 0.28
68.35 1.07
87.71 1 0.49
Table 2:
Classifical accuracy [in %] on CIFAR- 100 and Mini-Imagene (average and standard devi- ation across three runs)
CIFAR-100, Lid.
Mini-Imagenee Li...
m71 = my = 50
m my = 100
m = m 50
my = 100
Method
Nr = 5000
NL = 10000
Nı = 5000
Nz= 10000
FedAvg (labeled only)
43.80 0.19
53,91 0.25
23.39 0.52
31.72 - 0.54
FedAvg--L.(netoork)
43.80 0.19
4..19
23.98 = 0.36
31.86 +: 0.57
FeeAAv=+PL(local....
43.82 0.59
54.38 + 0.36
25.53 = 0.22
33.09 + 0.62
FedAvg+MT
49.09 + 0.38
56.05 + 0.23
25.98 = 0.70
33.20 + 0.68
FedProp (ours)
50.19 + 0.60
57.00
+ 0.08
26.93 4 0.41
35.78 0.56
FedProp+MT (ours)
50.60 = 0.28
58.66 0.28
25.62 = 0.75
33.24 0.71
in regions close the classifier decision boundaries. Second, the fact that FedProp always has clear advantage over Fedovg++P[(occllL)) supports the hypothesis that it is beneficial to obtain an estimate the underlying data manifold by combining data from multiple clients. In the i.i.d. case this is particularly noticeable when the number of examples per class is small. In the non-i.i.d. case, where the local label distribution does not match the overall one, we even observed local LP to often reduce the accuracy, which caused the model selection procedure completely deactivate it. This contrast to FedProp, which consistently improves the accuracy even in this case.
6 CONCLUSIONS
In this work we introduced the FedProp method for federated semi-supervised learning. Building on efficient cryptographic primitives. it allows the computation of label propagation along an estimate of the data manifold to which the data of all participating clients contribute, without the clients having actually share their data with anyone else. Rather, the server received only matrix of approximate similarity values between embedded data points. An interesting aspect for future work would be how to add additional security to this step, such as making it differentially private, which could allow an extension of the current approach to the malicious server setting. Our experiments established that cross-cliens label propagation can substantially increase the classification accuracy compared training only on labeled examples, pseudo-labels based on network outputs or using per-client label propagation. In future work. it would be interesting to study not just FedProp's accuracy, but also its efficiency real-world federated learning tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abdullatif Albaseer, Bekir Sait Ciftler, Mohamed M. Abdallah, and Ala I. Al-Fuqaha. Exploiting unlabeled data in smart cities using federated edge learning. In International Wireless Communi- cations and Mobile Computing Conference (IWCMC), 2020. URL https: /oi.. /ttp:///doo.g/1...
11C9/IWCNC48107. 2020. 9148475
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A holistic approach to semi-supervised learnin In Conference Neural In- formation Processing Systems (NeurIPS), 2019, URL URU/attps:://roocdiiiooo http:///pproodnnnn....nee.aa.a neurips. abaa -bbsseeracc--bb Abstract.
html
Kallista A. Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy-preserving machine learning. In ACM Conference on Computer and Communications Security (SIGSAC), 2017. URL hwtps: /tt.///oi..e wwe.lorrg11... 0t/g//0.1145/11333 11411111311155551 3133982
Julien Bringer, Hervé Chabanne, and Alain Patey. SHADE: Secure HAmming DistancE com- putation from oblivious transfer. In International Conference on Financial Cryptography and Data Security (FC), 2013.
com/ chapter/10.110077
978 3- 642 41320-9-111
Olivier Chapelle, Berhhard Schölkopf, and Alexander Zien (eds.). Semi-Supervised Learning The MIT Press, 2006. URL https://doi.. http:///..rrrr...../t//////ras//
001 0001
Ronald Cramer, Ivan Bjerre Damgård, and Jesper Buus Nielsen. Secure multiparty computation. Cambridge University Press, 2015. http:///tp/////...............r.
org.aa/iioooiioioaaoo.ooooooooor and coddcooingg/ oew.uuteweiitiiioo.oooooooooooo.
Don Kurian Dennis, Tian Li, and Virginia Smith. Heterogeneity for the win: One-shot federated clustering. In International Conference on Machine Learing (ICML), 2021. URL http://
proceedings www.mrr....... www.srppree//ee sezees/vv39/ htteesssvv13//eeniiii11.. dennislla.nii
Enmao Diao, Jie Ding, and Vahid Tarokh. SemiFL: Communication efficient semi-supervised fed- erated learning with unlabeled clients. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
01432
Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Helen Möllering, Thien Duc Nguyen. Phillip Rieger, Ahmad-Reas -ezadd-Reza Sadeghi, Thomas Schneider, Hossein Yalame, and Shaza Zeitouni. Safelearn: Secure aggregation for private federated learning. In IEEE Secu- rity and Privacy Workshops (SP), 2021. URL htt.l:tpp/// //o. wwt. 119
2021. 00017
Andreas Grammenos, Rodrigo Mendoza -mith, Jon Crowcroft, and Cecilia Mascolo. Feder- ated principal component analysis. In Conference Neural Information Processing Systems (NeurIPS), 2020. URL 47a658229eb2366 21w.:6aa099911122221-86------000
cc/ httepppppe2222//aaaeee html
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. URL MtLL:tttt:://peaaceee Deep_ Res Reesidual Lea____earrinnnn_eeee_e CVPR htt____C______Peee____ html
thecvf. com/ content__vpppeeeeeeeeeeeeeeeeeee 2016/html/He
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Symposium on Theory of Computing (STOC), 1998. URL htttts:///dd..
acm. oramoorg/oo//be 110 /1t.1...1221111..2222
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. URL
theevf.
10

---

Under review as a conference paper at ICLR 2023
Iscen Label Propaqation for Deep Semi-Supervised Learning_CVE 2019
Yuval Ishai, Joe Kilian, Kobbi Nissim, and Erez Petrank. Extending oblivious transfers efñ- ciently. In Advances in Cryptology (CRYPTO), 2003. URL /ttp::::///....or1111..10
978-3-540-433
Wonyong Jeong, Jaehong Yoon, Eunho Yang. and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. In International Conference on Learning Representations (ICLR), 2021. URL https: /t/////peeeeeiee/e net. fooummiid ce6CFXBh30h.
https:///penrevier...
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista A. Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L.. D' DlOvveii,, Hubert Fichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Gar- rett, Adria Gascón, Badih Ghazi, Phillip B. Gibbons. Marco Gruteser, Zaïd Harchaoui, Chaoyang He. Lie He. Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Hang Qi. Daniel Ramage, Ramesh Raskar, Mariana Raykova, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Xiong, Zheng Xu, Qiang Yang, Felix Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 2021. /0. 20..0011112222000000
URL
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: stochastic controlled averaging for federated learning. In International Conference on Machine Learing (ICML), 2020. http:httpp//poeeeeeee
wttpss/v1l/karrrr.../ http1/////aaimmeed222eeaaa htmmm] html
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical re- port, University of Toronto, 2009. URL https::/wwwt cs wow.oooooo..oooooor.........
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML. Workshop Challenges in Representation Learning (WREPL.), 2013.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. In Machine Learning and Systems (MLSys), 2020.
htttpp::/prrocee.nng www.gnmmmmm.......m............. sys. org bttbb/oo/////// /11/.116.. pd
Haowen Lin, Jian Lou, Li Xiong, and Cyrus Shahabi. SemiFed: Semi-supervised federated learning with consistency and pseudo-labeling (.Xi:22108.094122 2021. URL htlphttts///axxi.ooot
abs/2108 2108 09412
Zewei Long, Liwei Che, Yaqing Wang, Muchao Ye, Junyu Luo, Jinze Wu, Houping Xiao, and Fen- glong Ma. FedSiam: Towards adaptive federated semi-supervised learning. arXiv:2012.03292,
2020. /Rtpttss://axxii
http://axiivv.ooo wwwp:/iwwvrrgaa//////aoa wwwpa//.......eoo 03292
Disha Makhija, Nhat Ho, and Joydeep Ghosh. Federated self-supervised learning for heterogeneous clients. 1cXiv:2205.12493 2022. URL htttttts///axiii
htlgg/bb/222000 12493
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agtiera Ar-
cas. Communication-effiien learning of deep networks from decentralized data. In Inter- national Conference on Artificial Intelligence and Statistics (AISTATS), 2017. URL http:
//proceedings.. ml pwessppess....mm. v poress//34/mcmmmaannnnn. httm///maaaaaaaaaaaa
Lawrence Roy Peter Rindal. libOTe:
efficient, portable, and easy Oblivious Transfer wtt.c//moooooooo Library. http:///t//.tta..o.oo/oooooo/o//
Riccardo Presotto, Gabriele Civitarese, and Claudio Bettini. Semi-supervised and personal- ized federated activity recognition based on active learning and label propagation. Personal and Ubiquitous Computing, 26: 128111112 2022. URL hattps://dae http://///all.oo1111000000
s0 1D0779-022-01682

---

Under review as a conference paper at ICLR 2023
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S. Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-awar pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations (ICLR), 2021. htthtts:: //openreviexs eww wwt./ d
Lawrence Roy. SoftSpokenOT: Communicationn-compuuaaiioo tradeoffs in OT extension. Cryptol- ogy ePrint Archive 2022/192, 2022. /tLppttt:///ppritttt waw...t..iaa.oooooo wwt.aaorr..gg/2/////33// httpas////pparana.aa.cooooo////2
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor-
mations and perturbations for deep semi-supervised learning. In Conference on Neural Informa-
tion Processing Systems (NeurIPS), 2016. URL /ttp::ss///rroccedingg...oo//// neurips ca/
.html
Adi Shamir. How to share secret. Communications of the ACM, 22(11)6112-61, 1979. URL
acm.
359176
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng Ma, Xiaoyu Tao, and Nanning Zheng. Transductive semi-supervised deep learning using min-max features. In European Conference on Computer Vision (ECCV), 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekin Do- gus Cubuk, Alcxey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi-supervised learn- ing with consistency and confidence. In Conference Neural Information Processing Systems (NeurIPS), 2020. URL httpppsss//rroocenngg....sttt neurips http://w.....aaoccc/aaaa////////
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Conference on Neural Informa- tion Processing Systems (NeurlPS), 2017. URL URL/Btttee//prrocedd.ngg htwp:/eeeddnnnn..........e. neurips cc/
Daper/2017/haree
68053aff222. http:h/hhh66633332222222222233a3 aaaaaaaaaaacccaaooooaaaaaaaaatat Absttrat.ttrrtt.a..... hhtml
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Conference on Neural Information Processing Systems (NeurIPS), 2016. URL phttps://ppte /tttpp:://proccediiggs neuripeeepppp.. http://pss..pppppe///////a/aaaaa
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. Unsupervised data aug- mentation for consistency training. In Conference on Neural Information Processing Systems (NeurIPS), 2020. URL httpsss/pprooced..gg...eee.
cc/ppaperr/222//aash .hml
Andrew Chi-Chih Yao. Protocols for secure computations. Foundations of Computer Science (FOCS), 1982. (ttptttp:://bbb.o////0111///////
Andrew Chi-Chih Yao. How generate and and secrets. In Foundations of Computer Science (FOCS), 1986. UR/ppttps://ooo..or/000111100350
.55
Zhengming Zhang, Yaoqing Yang, Zhewei Yao. Yujun Yan, Joseph E. Gonzalez, Kannan Ramchan- dran, and Michael W. Mahoney. Improving superviper federated learning by reducing the gradient diversity of models. In International Conference on Big Data (Big Data), 2021. URL.
htttps://doi.coo http://doo.ogg///// org/10.......... /199/BigData2255352 2021. 9671693
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In Conference on Neural Information Processing Systems (NeurIPS), 2004. URL
wtwp:r/oceeedi.........e..eeeeee httpp//sc...pppp////////1/1///// pdd
8 87682805257e61 baw
Weiming Zhuang, Yonggang Wen, and Shuai Zhang. Divergence-uware federated self-supervised learning. In International Conference on Learning Representations (ICLR), 2022. d=ovE1
//openreview. net. fort/formm 1 28N1Ne
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENTAL DETAILS
A. HYPER-PARE PARAMETER SETTINGS
Here we detail specific parameter settings our experiments. For clarity we separate the parameters into those relating to federated learning, those relating to network training and those belonging to the Crossclis LP routine.
Federated learning parameters We set the number of clients to m = 100, except for our exper- iments on CIFAR- 100 and Mini-Imagenet with n,, = 5000. In these cases we set m
as not possible to create an i.i.d. split of the data over 100 clients since the number of classes (C=100)
too large. For CIFAR-10 10 we set the number clients which possess labels to my = 100 and mL 50. On CIFAR-100 and Mini- Imagenett set mL
For the FederatedOpt imii zation method we choose FederatedAveraginng ClientUpdate step therefore corresponds to E epochs of stochastic gradient descent (SGD) of a loss function. We set the number local epochs to E = 5 and the loss function is (per sample weighted) cross-entropy loss. The ServerUpdate step corresponds to averaging the model updates:
The
ServerUgdate((0) for E P)
Σ B0) JEP
P
The number training rounds is set to = 1500 and the number of clients sampled by the server 100 and 5 0.1 when m 50. Note
training round is set to 5, so 0.05 when that when my m we ensure that the server samples 7mg clients from the labeled portion (and T(m- - m,) from the unlabeled) to ensure that there are some labels present in the graph.
Network training parameters We use standard data augmentation following Tarvainen Valpola 10 and CIFAR-100 this is performed by 4x4 random translations followed by random horizontal flip. On Mini-ImageNet,, each image is randomly rotated by 10 degrees before random horizontal flip. We use weight decay for all network parameters which is set to 2 x 10-4 When carrying out SGD in the Client Update we use batches data B = B₁, U By where B5 batch of labeled data and By is batch of pseudo-labeled (previously unlabeled) data. We set [Bu] according to how many labeled samples the client has available, B&l = min (50, #labels). according to this batch size. On CIFAR-10, for [r.] < 50 we set the learning rate to 0.1 and for |Br| = 50 we set the learning rate to 0.3. On CIFAR-101 100 and MinilmageNet
(017). On CIFAR-10
We By |B.. Learning rate for SGD
always have 1 50 and we the learning rates 0.5 and 1.0 respectively. We decay the learning rate using cosine annealing so that the learning rate would be 0 after 2000 rounds.
CrossClientLP parameters We set the LSH dimension to L = 4096 as this gave near exact approximation of the cosine similarities while still being computationally fast (less than second per round). We set the sparsification parameter to k = 10, so that each point is connected to its 10 most similar neighbors in the graph, and the label propagation parameter to a 0.99.
13


Reviews:
[{"review_id": "xo3M2NZcQ_", "reviewer": "Reviewer_8bDJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies federated semi-supervised learning and introduces the FedProp method. FedProp allows the computation of label propagation along an estimate of the data manifold to which the data of all participating clients contribute and use efficient cryptographic primitives to avoid accessing the data from different clients. Experiments on three standard benchmarks show that FedProp achieves promising semi-supervised classification performance. The main contribution lies in using LP to generate pseudo labels for unlabeled samples and applying cryptographic primitives to avoid direct data access \n\n", "summary_of_the_review": "Detailed comments:\n\nThe main idea to leverage LP to generate pseudo labels is not new.  Also LP is shown to be not effectiveness enough to generate  pseudo labels, if the data distribution does not naturally form a manifold, e.g., when the data samples is limited.  \n\nWhy connecting data form different clients to perform label propagation? In the non-IID settings, the data distribution across different clients could be  significantly different. Is there a need to connect data from different clients to perform LP? What if each client just use its data to perform LP and obtain pseudo labels?  \n\nIs the proposed FedRep convergent? In what convergence rate? What is the risk bound?\n \nNo formal privacy guarantees.   For instance, “first, the similarity is not computed between input data itself, but its feature representation according to the current model“. Using indirect feature representations cannot protect the input privacy. Particularly, many existing works has shown that using shared model gradients can reconstruct the data, so does the learnt representations.  \n\nZhu et al. Deep leakage from gradients. In NIPS, 2018\n\n\nThere exist more advance SSL methods (e.g, mixmatch, FixMatch) which can augment data with pseudo labels. Applying mixmatch to the federated SSL setting should be not difficult? What is the federated SSL performance with these data augmentation performance? \n\nSemiFL (NeurIPS 2021) is FedRGD are mentioned, but not compared. Actually, SemiFL has shown to significantly outperform the compared methods, e.g., FedMatch, especially in the non-IID setting. \n\nWhat is the data distributed in CIFAR100 and Mini-Imagenet? IID or non-IID? \n \nWhat is the impact of different number of labeled samples? \n\nWhy connecting data form different clients to perform label propagation, especially in the non-IID setting,  is unclear to me. I suggest the authors should conduct experiments to validate this. \n\nThere exist several federated learning methods for semi-supervised classification, where the client data itself is already a graph. Please discuss them. \n\n\nZhang et al.,  Subgraph Federated Learning with Missing Neighbor Generation. NeurIPS 2021. \n \nXie et al., Federated graph classifica- tion over non-iid graphs. NeurIPS 2021 \nWang et al., Graphfl: A federated learning framework for semi-supervised node classification on graphs. ICDM 2022", "strength_and_weaknesses": "Strengths\n+ The studied problem is important\n+ Paper is easy to follow \n\nWeaknesses\n- Novelty is limited. \n- No theoretical results.  \n- Lack of  comparison with state-of-the-arts. \n- Evaluation results are insufficient \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "syKvEcXMNI", "reviewer": "Reviewer_jCXS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a solution for federated semi-supervised learning. the proposal lies averaging the manifold in the decision space over multiple clients. The sharing is done using two cryptographical protocols. The paper evaluates on three datasets", "summary_of_the_review": "The paper shows improvement over previous federated SSL methods. But I see this topic as a niche and this paper does too little to enlarge the auditorium. The innovation is limited to the topic.", "strength_and_weaknesses": "Pros: \n  - The idea to use manifold for pseudo-labeling has not been used before in federated semi-supervised leaning, although it has been used in classical SSL.\n- the paper has reported improvement in performance over similar federated SSL papers, but is still far from plain SSL performance\n\nCons:\n - the paper claims to used secure and cryptographic protocols for communication. The paper does not really explain why do we need those. I understand that there may be connection problems and communication to be disrupted and thus adding checksums and redundancy will help. I do not understand why using cryptographic protocols help ensuring intimacy of the data for each client since the framework do not assume external digital attacks. Furthermore, while the proposed method uses them, there is no evaluation of their efficiency. For instance how much damage to the communication can be absorbed without changes in performance? The fact that prior art used them, does not help claiming anything here.\n- Evaluation is strong from one perspective but disappointing from another. It is carried on standard benchmarks which is very nice because it allows very good comparisons with previous similar works. It is weak because it does not propose a scenario where it really make sense to have protected data such as face related problems or where to assume that data being separated over clients, there is also a bias (either in data, either in labels) due to independent gathering and annotation. While this may not be fair to this paper, yet this is a major conference, and simply improving inside the niche, after a while becomes less interesting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wie_LRKDZ4v", "reviewer": "Reviewer_G4t3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents  Federated Label Propagation (FedProp), an algorithm, to effectively learn semi-supervised models by distributing both labeled and unlabelled data across multiple clients.  Only model parameters are shared across clients through a server (following a client server architecture) but not the data, which is in fact one of the principles of federated learning. FedProp algorithm takes into account existing tools/techniques such as  Label propagation (LP) and Locality Sensitive Hashing (LSH) as part of its solution. \n \nIn summary, FedProp  first initializes the global solution by learning client-wise independent solutions from corresponding labeled data (that is assigned to a client) and finally running an aggregation step for parameters shared by clients. In addition to the aggregation, the server also acts as an intermediator to share parameters across clients through a broadcasting operation whenever necessary. At each iteration clients are responsible for independently assigning pseudo labels to its portion of unlabeled data and subsequently use those as new/additional labeled data to update its parameters before broadcasting to the server. The above process (pseudo labeling, parameter update and broadcasting and aggregation) iterates for a predefined number of iterations and is expected to converge (at the end). \n\nThe proposed technique has been tested on CIFAR-10, CIFAR-100 and Mini-Imagenet datasets and compared against existing techniques such as FedAvg, FeMatch, FedSem, and FedSiam. Reported results  are found to be comparable. \n", "summary_of_the_review": "I have gone through the paper more than once including the appendices. Overall, the idea is quite simple: combine existing tools and techniques in a smart way that does the job.  Distribute labeled and unlabelled data across clients and let clients learn and share their parameters to a server; the server then performs some aggregation and shares them back to clients. Iterate these back and forth steps until the solution converges. The work lacks major technical contributions although the reported results look promising.", "strength_and_weaknesses": "Strengths: Federated Label Propagation (FedProp) uses Label propagation (LP), Locality Sensitive Hashing (LSH), and some other established semi supervised tools/techniques and combines them in a structured fashion to guide semi supervised learning. The experiments and reported results look promising. Also, an analysis of the correctness, efficiency and robustness of their FedProp algorithm has been addressed in section 4.2. \n \nWeakness: The results reported in section 5 don’t include any statistical tests so it's difficult to verify whether the results make any significant difference to existing techniques. The proposed methodology also shows some limitations in terms of scalability (as per the results in section 5): The reported gain in performance reduces when tested for a larger (CIFAR-100 vs CIFAR-10) or a more complex datasets (Mini-Imagement -100).  \n\nAnother weakness/limitation is this work lacks major technical contributions; It uses existing tools and techniques and ties them as its solution.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VNlwbNINAv", "reviewer": "Reviewer_ZBdp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "-This paper studies the problem of federated learning with semi-supervised client (local) data. They propose a method that first pre-trains the client models with their labeled training data, then performs cross-client label propagation based on the embeddings of the labeled and unlabeled training data obtained from the pre-trained models, and after obtaining the pseudo labels and per-sample weights standard federated training is applied. They also provide some experimental results demonstrating the effectiveness of this method.", "summary_of_the_review": "-From the reviewer's point of view, this paper provides some interesting empirical results but more theoretical investigations are needed.", "strength_and_weaknesses": "Pros\n\n-The writing is clear and the paper is easy-to-follow. The authors attempt to solve a practically important question and show some favorable empirical results.\n\nCons\n\n-There are several issues that remain unsolved in the paper. The first concern is about novelty. To utilize the semi-supervised training data, the proposed method simply (in terms of methodology) employs the widely used label propagation in federated training. Although in practice, in order not to violate the federated training protocols (do not share data between clients) by using cross-client data information (embeddings), the authors introduce some cryptographic tools which are claimed to be fully secure, but they do not provide any privacy guarantees.\n\n-The second concern is about feasibility. Label propagation is not free. We may not accurately estimate the data manifold with only limited non-iid data and thus label propagation may not be reliable. The authors propose to solve this problem by doing so on embeddings of the training data from all clients. However, we still cannot verify the manifold assumption in this case and can only observe from the empirical performances. This raises another concern about the convergence of the proposed method (if the label propagation is not reliable). Existing experimental results are based on benchmark datasets and the semi-supervised data are artificially designed. It would be useful to test the proposed method with real-world non-iid semi-supervised client data and see if label propagation works well.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject