Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning
Authors: Wonyong Jeong
Jaehong Yoon
Eunho Yang
Sung Ju Hwang
Venue: ICLR 2021 Poster (2021)
Keywords: Federated Learning

Abstract: While existing federated learning approaches mostly require that clients have fully-labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data at each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to a new practical federated learning problem, namely Federated Semi-Supervised Learning (FSSL). In this work, we study two essential scenarios of FSSL based on the location of the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data (labels-at-client), and the second scenario considers a more challenging case, where the labeled data is only available at the server (labels-at-server). We then propose a novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with semi-supervised learning.

Published as a conference paper at ICLR 2021
FEDERATED SEMI-SUPERVISER LEARNING WITH INTER-CLIENT CONSISTENCY & DISJOINTT LEARNING
Wonyong Jeong's JJahong Yoon?, Eunho Yang and Sung Ju Hwang!.3
Graduate School of AI2, KAIST, Seoul, South Korea School of Computing", KAIST, Daejeon, South Korea
AITRICS Seoul, South Korea jeong, jaehong, yoon, eunhoy, s (chwang82]0kaise ac.
ABSTRACT
While existing federated learning approaches mostly require that clients have fully- labeled data to train on, in realistic settings, data obtained at the client-side often comes without any accompanying labels. Such deficiency of labels may result from either high labeling cost, or difficulty of annotation due to the requirement of expert knowledge. Thus the private data each client may be either partly labeled, or completely unlabeled with labeled data being available only at the server, which leads us to new practical federated learning problem, namely Federated Semi- Supervised Leurning (FSSL). In this work, study two essential scenarios of FSSL based on the location the labeled data. The first scenario considers a conventional case where clients have both labeled and unlabeled data labels-at-client and the second scenario considers a more challenging case, where the labeled data is only available at the server (laels-at-server)) We then propose novel method to tackle the problems, which we refer to as Federated Matching (FedMatch). FedMatch improves upon naive combinations of federated learning and semi-supervised learning approaches with a new inter-client consistency loss and decomposition of the parameters for disjoint learning on labeled and unlabeled data. Through extensive experimental validation of our method in the two different scenarios, we show that our method outperforms both local semi-supervised learning and baselines which naively combine federated learning with superviee learning. The code is available at http://gghuub.ooowwyooo/n///////
INTRODUCTION
Federated Learning (FL) (McMahan al., 2017: Zhao al., 2018; Li al., 2018; Chen al., 2019a;b), which multiple clients collaboratively learn global model via coordinated communica- tion, has been active topic of research over the past few years. The most distinctive difference of federated learning from distributed learning is that the data is only privately accessible at each local client, without inter-client data sharing. Such decentralized learning brings numerous advantages in addressing real-world issues such as data privacy, security, and access rights. For example, for on-device learning of mobile devices, the service provider may not directly access local data since they may contain privacy-sensitive information. In healthcare domains, the hospitals may want to improve their clinical diagnosis systems without sharing the patient records.
Existing federated learning approaches (McMahan et al., 2017; 7; Wang et al., 2020; Li er al., 2018) handle these problems by aggregating the locally learned model parameters. A common limitation that they only consider supervised learning settings, where the local private data is fully labeled. Yet, the assumption that all of the data examples may include sophisticate annotations is not realistic for real-world applications. Suppose that we perform on-device federated learning, the users may not want to spend their time and efforts in annotating the data, and the participation rate across the users may largely differ. Even in the case of enthusiastic users may not be able to fully label all the data in the device, which will leave the majority of the data as unlabeled (See Figure (a)). Moreover, in some scenarios, the users may not have sufficient expertise to correctly label the data. For instance, suppose that we have a workout app that automatically evaluates and corrects one's 's body posture. In this case, the end users may not be able to evaluate his/her own body posture all (See Figure 1

---

Published as a conference paper at ICLR 2021
Amateur-level Labeling
Global Model
Global Model
CONTATT
Labeled by Experts @ Server
BAR
Suay
Local Models
Labeled & Unlabeled @ Client
Local Models
Unlabeled Examples @ Client
(a) Labels-at-Client Scenario
(b) Labels-at-Server Scenario
Illustrations of Two Practical Scenarios in Federated Semi-Supervised Learning (a) Labels-at- Client scenario: both labeled and unlaheled data available local clients. (b) Labels-at-Server scenario: labeled instances are available only at server, while unlabeled data are available at local clients.
Figure 1:
(b)). Thus, in many realistic scenarios for federated learning, local data will be mostly unlabeled. This leads to practical problems of federated learning with deficiency of labels, namely Federated Semi-Supervised Learning (FSSL).
A naive solution to these scenarios is to simply perform Semi- Scmi-Supervised I Learning (SSL) using any off-the-shelf methods (e.g. FixMatch (Sohn et al., 2020), UDA (Xie et al., 2019)), while using federated learning algorithms to aggregate the learned weights. Yet, this does not fully exploit the knowledge of the multiple models trained on heterogeneous data distributions. To address this problem, present novel framework which refer as Federated Matching (FedMatch), which enforces the consistency between the predictions made across multiple models. Further, conventional not applicable for scenarios where labeled data is only available at the server (Figure (b)), which is a unique SSL setting for federated learning. Also, even when the labeled data is available the client (Figure (a)), learning from the unlabeled data may lead to forgetting of what the model learned from the labeled data. To tackle these issues, we decompose the model parameters into two, dense parameter for supervised and sparse parameter for unsupervised learning. This sparse additive parameter decomposition ensures that training on labeled and unlabeled data are effectively separable, thus minimizing interference between the two tasks. We further reduce the communication costs with both the decomposed parameters by sending only the difference of the parameters across the communication rounds. We validate FedMatch on both scenarios (Figure (a) and (b)) and show that our models significantly outperform baselines, including a naive combination of federated learning with semi-supervised learning, on the training data which are both non-i.i.d. and i.i.d. data. The main contributions of this work are follows:
semi-supervised learning approaches
We introduce a practical problem of federated learning with deficiency of supervision, namely Federated Semi-Supervised Learning (FSSL), and study two different scenarios, where the local data is partly labeled (Labels-at-Client) or completely unlabeled (abbels-tt-erver)
We propose a novel method, Federated Matching (FedMatch), which learns inter-client con- sistency between multiple clients, and decomposes model parameters to reduce both interference between supervised and unsupervised tasks, and communication cost.
We show that our method, FedMatch, significantly outperforms both local SSL and the naive combination of FL with SSL algorithms under the conventional dabels-at-clies and the novel labels-at-server scenario, across multiple clients with both non-i.i.d. and i.i.d. data.
PROBLEM DEFINITION
We begin with formal definition of Federated Learning (FL) and Semi-Supervised Learning (SSL). Then, we define Federated Semi-Supent Semi-Supervised Learning (FSSL) and introduce two essential scenarios.
2.1 PRELIMINARIES
Federated Learning Federated Learning (FL) aims collaboratively learn global model via coordinated communication with multiple clients. Let G be a global model and L (4, (4 X be set of local models for К clients. Let = (x, y: V. N be given dataset, where x; is an arbitrary training instance with corresponding one-hot label y, E ( C) for the C-way multi-class classification problem and N is the number of instances. is composed of K sub-datasets Día {x* y v privately collected at each client local model 4. At each communication round r, G first randomly selects A local models that are available for training L" C. and [C]] = A. The global model G then initializes L with global weights 0G and the active local models 1, E L" perform supervised
(1....,C)
2

---

Published as a conference paper at ICLR 2021
learning minimize loss (6'x)) on the corresponding sub-dataset D'a, G' then aggregates the learned weights 0 + N' Za 0 and broadcasts newly aggregated weights to local models that would be available at the next round 1, and repeat the learning procedure until the final round R.
N
Semi-Supervised Learning Semi-supervised learning (SSL) refers the problem of learning with partially labeled data, where the ratio of unlabeled data is usually much larger than that of the labeled data (e.g. 10). For SSL, D is further split into labeled and unlabeled data. Let S {xx, y; be set of S labeled data instances and U {u: 1 be a set of U unlabeled samples without corresponding label. Here, in general, |S| < [4. With these two datasets, and M, we now perform semi-supervised learning. Let ( (y) x) be neural network that is parameterized by weights and predicts softmax outputs with given input x. Our objective is to minimize loss function finall (0) = (0) + (9), where (0) is loss term for supervised learning on S and (0) loss term for unsupervised learning on L/
2.2 FEDERATED SEMI-SUPERVISED LEARNING
Now we further deseribe a practical problem of deficiency of labels in federated learning, which we refer to as Federated Semi-Supervised Learning (FSSL), in which the data obtained at the clients may or may not come with accompanying labels. Given a dataset D = (x., y.), is split into a labeled set S {x, y:) y and unlabeled set 21 {u, |/ in the standard remi-supervised learning. Under the federated learning framework, we have a global model G and a set of local models L. where
1 the unlabeled dataset U4 is privately spread over K clients hence U {u. Ji=1 For labeled set S, on the other hand, we consider two different scenarios depending on the availability of labeled data at clients, namely the Labels-at at- Client and the and learning procedures will be discussed later.
scenario, of which problem settings
3 FEDERATED MATCHING
We now describe our Federated Matching (FedMatch) algorithm to tackle the federated semi- supervised learning problem. We describe its core components in detail in the following subsections.
3.1 NTTERCLIIENT CONSISTENCY Loss
Consistency regularization (Xie et al., 2019; Sohn et al., 2020; Berthelot et al., 20:9b;a) is one of most popular approaches to learn from unlabeled examples in semi- supervisd learning setting. Conven- tional ponsistency-regularizattt
methods enforce the predictions from the augmented examples and original (or weakly augmented) instances to output the same class label, | (u) po (y F [1) where π (.) is a stochastic transformation function. Based on the assumption that class semantics are unaffected by small input perturbations, these methods basically ensures consistency of the prediction across the multiple perturbations of same input. For our federated semi-supervised learning method, we additionally propose a novel consistency loss that regularizes the models learned at multiple clients to output the same prediction. This novel consistency loss for FSSL, inter-client consistency loss, is defined as follows:
H
H Σ KL (ylu)) (ylu))
1
where poa (y x) are helper agents that are selected from the server based on model similarity to the client (which we describe later), that are not trained the client (* denotes that we freeze the parameters). The server selects and broadcasts H/ helper agents at each communication round. We also use data-level consistency regularization at each local client similarly to FixMatch (Sohn et al., 2020). Our final consistency regularization term ((-) can be written as follows:
" KL (ylu) |p (y/u))
(-) = CrossEEntropy($ Po (y) (u))
M
(2)
j=1
where Tu) performs RandAugment (Cubuk al., 2019) on unlabeld instance is our novel lseudo-labeling technique, which we refer to as the agreement- based pseudo label, defined as follows:
3

---

Published as a conference paper at ICLR 2021
Helper 2
Truck
Truck
Helper C
CE Pe
Truck
Bumln
LacalModel(
Cat
y
T())
Cat Truck
Unlabeled Instance
Inter-Client Consistency Loss
Agreement-based
Local Consistency Regularization
Pseudo-Labeling
Figure 2: Illustration of Inter- Inter-Client Consistency Loss. We illustrate each step of our inter-client consistency regularization process performed at local client. We provide the detailed explanations in Section 3.1.
H
ý = Max (p)) (y/w)) +
Σ (0 (y/u)))
==1
where 1-)) produces one-hot labels with given softmax values, and Max(-) outputs one-hot labels on the class that has the maximum agreements. We discard instances with low-confident predictions below confidence threshold r when generating pseudo-labels. We then perform standard cross-entropy minimization with the pseudo-label, y.
For helper agents, we select the H helper agents Paty. (y/u) for each client as the most relevant models from other clients. Specifically, we represent each model by its prediction m on the same arbitrary input a located server (we use random Gaussian noise), such that m'= Pai (m/a). The server tries to keep and update all model embeddings m 1:K from clients once each client updates its weights to server, and creates K-Dimensional Tree (KD Tree) on m in the current round r for nearest neighbor search to rapidly select the H helper agents for each client in the next rounds. We send helper agents for every rounds, and if a certain client has not yet updated its weights to server in the previous step, then server simply skips sending helpers to the client at the round.
3.2 PARAMETER DECOMPOSITION FOR DISJOINT LEARNING
In the standard semi-supervised learning approaches, learning on labeled and unlabeled data is simultancously done with shared set of parameters. However, since this inapplicable to the disjoint learning scenario (Figure 1 (b)) and may result in forgetting of knowledge of labeled data (see Figure (c)), we separate the supervised and unsupervised learning via the decomposition of model parameters into two sets of parameters, one for supervised learning and another for unsupervised learning. To this end, we decompose our model parameters 9 into two variables, o for supervised learning and v for unsupervised learning, such that 0 = + th. We perform standard supervised learning on o, while keeping va fixed during training, by minimizing the loss term as follows:
minimize (o) A CrCrssEntrropyyyy..mmm Po+y (ylx))
(4)
where x and y are from labeled set S. For learning on unlabeled data, we perform unsupervised learning conversely on vi while keeping fixed for the learning phase, by minimizing the consistency loss terms as follows:
minimize (bb) Aices P.. +.() + ALzllo" lb12 + λz, 1|
(5)
where all Às are hyper-parameters to control the learning ratio between the terms. We additionally add L2- and L Regulaarriitii on v such that & is sparse, while not drifting far from knowledge that a has learned. To sum up, our decomposition technique allows us:
Preservation Reliable Knowledge from Labeled Data: We empirically find that learning on both labeled and unlabeled data with a single parameters may result in the model to forget about what it learned from the labeled data (see Figure 6 (c)). Our method can effectively prevent the inter-task interference via utilizing disjoint parameters only for supervised learning.
Reduction of Communication Costs: Sparsity on the unsupervised parameter v allows to reduce communication cost. In addition, we further minimize the cost by subtracting the learned knowledge for each parameter, such that A e GhC and Ao = of of and transmit only the differences A and Ao sparse matrices for both client-to-sene and server-to-clien costs.
Disjoint Learning: In federated semi-supervised learning, labeled data can be located at either client or server, which requires the model's 's learning procedure to be flexible. Our decomposition technique allows the model for the supervised training to be done separately elsewhere.
4

---

Published as a conference paper at ICLR 2021
Algorithm Labels-at-Cliees Scenario
Asgregation
1: RunServer()
Σ
2: initialize o² and /0
B
GBD 090
3: for each round 1, .... do
4: C' +seleect random A clients from L)
5: for each client & L2 parallel
Global Model
High Similarity
Low Simil
6:
Wi-zr < GeeNVcareeNeiighhorss 7: ou 4% +- RunClienn(o" (..H)
spla
8:
EmbeeLooclMoodello, (2))
9: end for
Learn a's and yola with Helper 1, Helper 2
10:
1--1- (o% 11: 1++ 1
1 Σ.
12: end for
000
13: RunClient(r,, 1h 1/1.1s)
Labsled
+ P1.11 15: for each local epoch e from to Er. do
14: li, 6 1h Obi.
Local Model La
16: for minibatch E St, and E id, do
Figure 3: Illustrative Running Example Labels- at-Client Scenario We describe training and commu- nication procedure hetween local and global model under Labels-at-Client scenario corresponding the Algorithm 1. More details are described Section 4.
17:
Octp- Ootv" (o, w)) Eu (0.- tos Onaar
18: 0g-+** 00 Vl
end for
20: end for
4
SCENARIO
Problem Definition The Satels-at-Cliemn scenario posits that the end-users intermittenlly annotate a small portion of their local data (i.e., 5% of the entire data). leaving the rest data instances unlabeled as illustrated in Figure (a). This is common scenario for user-generated personal data, where the end-users can easily annotate the data but may not have time or motivation to label all the data (e.g. annotating faces in pictures for photo albums social networking). We assume that clients train on both labeled and unlabeled data, while the server only aggregates the updates from the clients and redistributes the aggregated parameters back to the clients. In this scenario, labeled data S is a set individual sub-datasets Stx = {xx* le S Ji=1- yielding sub-datasets for local models l.Kk The overall learning procedure of the global model is the same that of conventional federated learning (global model G' aggregates updates from the selected subset of clients and broadcasts them), except that active local models 4:A perform semi-supervised learning by minimizing the loss E fiiall (0k=) (0'=) (('=)) respectively on S and it...
FedMatch Algorithm for Labels-at-Clics Scenario Now we introduce our FedMatch algorithm for the labels-at-client scenario. As shown Figure which illustrates an example case of the labels-at-client scenario, active local models 41 at the current round r learn both ofi:a and eff. on both the labeled data Slia and unlabeled data 2//1:4 each local environment. After the completion of local training, the clients update both their learned knowledge I'i:A and wffi, to the server. The server then aggregates g/u.A and d respectively, after embedding local models based on model similarity as well as create KD-Trree to rapidly retrieve the top- H nearest neighbors bi for each client. At the next round, the server transmits the aggregated r++ and v/*+1, For helper agents, server retrieves И helper agents, 6hMin, to each client for every 10 rounds. More details of the training procedures for FedMatch, for the labels-at-client scenario, is described in Algorithm 1.
5 BREELS-T--EERVEE SCENARIO
Problem Definition We now describe another realistic setting, which the labels-at st-serer scenario. This scenario assumes that the supervised labels are only available at the server, while local clients work with unlabeled data as described in Figure (b). This is a common case of real-world applications where labeling requires expert knowledge (e.g. annotating medical images, evaluating body postures for exercises), but the data cannot be shared due to privacy concerns. In this scenario, SG is identical to S and is located at server. The overall learning procedure is the same that of conventional federated learning, except the global model G performs supervised learning on S by minimizing the loss (,(0 before broadcasting @ to local clients. Then, the active local clients 4:a at communication round r perform unsupervised learning which solely minimizes e,, (0'))) on the unlabeled data a
5

---

Published as a conference paper at ICLR 2021
Learn σ
Similarty
Algorithm Labels-at-Server Scenario
1: RunServer()
a&e COOP 000
-
2: initialize o² .1/'
3: for each round r= 1, do
Global Model
Laboled Data
4: for each server epoch from 1 to Eg do
5: for minibatch E So do 8o+y* Bo+v* - (00+0-)))
6:
7:
end for end for
8:
Learn bl with Helper 1, Helper 2
9:
L" (select random A clients from L)
10:
for each client 12 E L" in parallel do
11:
Vi-H <- CeiNearesiNeighhossss""" 12: Va f -RunClient(o**1 1h" (..))
13:
EmbeeLocalMMoello" r+1 v.
Local Model La
Uniabeled Dattn
14: end for
15: g 2 (1%T.)
Figure 4: Illustrative Running Example of Labels- at-Server Scenario We depict learning and transmit- ting procedure between client and the global server under Lahels-at-Server scenario corresponding the Algorithm 2. Note that, in labels-at- server scenario, labeled data is only available at the server, and thus global model at the server learns on labeled data, while local models at clients learn only unlabeled data.
16: end for
17: RunClient(o, 1h, s).H))
18: Ot o" w. Ob Obn. <- o² + 1h.
19: for each local epoch e from to Er. do for minibatch u E 2i,
21: f.-+2 t f0 (fo +w; 0ar.r u)
22: end for
23:
end for
Further details are explained in Section
FedMatch Algorithms for Labels-at-Server Scenario We now describe our FedMatch algorithm for the labels-at-server scenario. As depicted in Figure 4, which describes an illustrative running example for labels-at-server scenario, the global model G' learns a on labeled data S at the server and the active local clients 1ı:A the current round learn u on unlabeled data 2 at each local environment. After the completion of local training, clients update their learned knowledge wii to the server. The server then embeds local models based on model similarity and create a KD-Tree for rapid nearest neighbor search for the top-H most similar 1hh models for each client. At the next round, server transmits its learned "++ and the aggregated vf*+1. Server transmits top- H similar di to each client for every 10 communication rounds. Further training details of FedMatch for the aubels-at-server scenario described in Algorithm 2.
EXPERIMENTS
We now experimentally validate our method, FedMatch, on three tasks, such as Batch-IID, Batch- NonIID, and Streaming-NonlID, under both scenarios, Labels-at-Clics and Labels-at-Server.
6.1 EXPERIMENTAL SETUP
Tasks 1) Batch-IID: We use CIFAR-10 for this task and split 60, 000 instances into training (54, 000), valid (3, 000), and test (3, 000) sets. We extract 5 labeled instances per class (C=10) for each client (K=1000 as labeled set S, and the rest of instances (49, 000) used unlabeled data U. so that we can evenly split S and 2/ into S'a:200 and 24/1:000 such that local models 11:10 learn on corresponding labeled and unlabeled data during training. 2) Batch-NonIID (clas-imbbllanced) The setting of this task is mostly the same with the Batch-IID task, except we arbitrarily control the distribution of the number of instances per class for each client to simulate (clas--mbbllanced: In this task, data streams into each client
maas--mmalanceed
environments. 3) Streaming-Non from class-ime insalannbd distributions. We use Bussion-MMNE
dataset for this task, and split 70, 000 instances into training (63, 000), valid (3, 500), and test (3, 500) sets. From train set, we extract 5 labeled instances per class (C=5) for each client 1K= for a labeled set S. We discard labels for the rest of instances to construct an unlabeled set 1A (62, 000). Then, we split S and // into S11:100 and 100 based on a class-imbalancee distribution. For individual local unlabeled data Ut',, we again split all instances into U ,t 21.2.... T}. where T is the number of total streaming steps (we set T=10). We train each streaming step for 10 rounds. We describe above tasks under Labels-at-Clien scenario. For Labels-at-S -e-Svrr scenario, is simply located at server without any partition. Please Figure in Appendix, which visualize the concepts of dataset configuration.

---

Published as a conference paper at ICLR 2021
Table 1: Performance Comparison on Batch-IID & NonlID Tasks We use 100 clients (F==.055) for 200 rounds. We measure global model accuracy and averaged communication costs. Note that the SL (Supervised Learning) models learn on both S and U with full labels, and are utilized as the upper bounds for each experiment.
CIFAR-I0, Batch- HID Task with 100 Clients (K =100, F=0.05. H=2)
Sheelsat-Chiene Scenario
Labels-at- Scenario
Methods
Acc.( ))
S2C Cost
C2S Cost
Acc,( %)
S2C Cost
C2S Cost
FedAvg-SI. SL
58.60 + 0.42
100 %
100 %
52.45 0.023
100 %
100 %
FedProx-SL
59.30 0:31
100 %
100 %
49.11 11 +.28
100 %
100 %
FedlAvg-DDa -DA
46.35 0.29
"100%
100 %
24.81 4.73
Too %
TO %
FedProx-UDA
47.45 0:21
100 %
100 %
19.91 +.21
100 %
100 %
Fedavvg-Fx.Mach
47.01
0.43
100 %
100 %
11.95
00000
100 %
100 %
Fidtth
47.20 0.12
100 %
100 %
25.61
1:032
100 %
100 %
FedMatch (Ours)
52.13 + 0.34
79
46 %
44.95
0.49
45
22 %
CIFAR-I0, Batch-NonIID)
Task with 100 Clients (K 1K0100,
F=0.05,
H=2)
FedAvg-SL
55.15 a: 021
100 %
100 %
51.50
0.551
100 %
100 %
FedProx-SL
57.75 0.15
100 %
100 %
49.31 0.188
100 %
100 %
FedAvg-UID4
44.35 0.30
100 %
100
27.61 +071
100 %
TO0
FedProx-UIDA
46.31 206
100 %
100 %
26.01 0.778
100 %
100 %
FedAvg-FixMatte
46.20 0.552
100 %
100 %
09.45
0.044
100 %
100 %
Fixtach
45.55
100 %
100
09.21
0.024
100 %
100 %
Mach (Ours)
32.25
+ 0.81
85
49 %
44.17
0.19
42 %
20 %
Batch-lID Task (100 Clients)
60
Botch- NonliD (100 Clients)
60
Batch-UlD Task (100 Clients)
60
Botch-NonilD (100 Clients)
50
50
50
50
40
40
440
Accuessy ct
40
30
330
30
20
NedPooaTUA
TedFroa754
200
TedProsTUDA
TedProctSi
20
PesProcUUDA
PecProctSL
20
Festyg
10
Tedtratch (Dund))
10
TediMarch (Dund)
10
FecMatch (Dum)
FecProclF shaachh
10
Fedetach Joan!
0
50
180
150
200
0
50
100
150
200
50
100
150
200
5b
100
150
200
Communication Round
Communication Round
Communication Round
Communicaiin Round
(a) ---eeet Scenario
(b) Labels at- Server Scenario
Figure 5: Test Accuracy Curves Batch-IID & NonIID Tasks We visualize test accuracy curves of model performance corresponding to the Table 1. Note that the SL (Supervised I Learning) models learn on both and U with full labels, and are utilized as the upper bounds for cach experiment.
Baselines and Training Details Our baselines are: 1) Local-SL: local supervised learning (SL) with full labels (S + U) without sharing locally learned knowledge. 2) Local-UDA and 3) Local- FixMatch: local semi-supervised learning, including UDA and FixMatch, without sharing local knowledge. 4) FedAVG-SL and 5) FedProx-SL: supervised learning with full labels (S + 2/) while sharing local knowledge via FedAvg and FedProx frameworks. 6) Fed and 7) FedProx- UDA: naive combinations of FedAvg/Prox with UDA. 8) FedAvg-FixMatee
and 9) FedProx- FixMatch: naive combination of with FixMatch with FedAvg/Prox. For training. we use SGD with adaptive-learning rate decay introduced in (Serra et al., 2018) with the initial learning rate 1e-3. We use ResNet-9 networks as the backbone architecture for all baselines and our methods. We ensure that all hyper- parameersssssrss are set equally for all base models and ours to perform fair evaluation and comparison. Please see the Section A in the Appendix for further details. For all experiments, we report the mean and the standard deviation over 3 runs.
6.2 EXPERIMENTAL RESULTS
Results on Batch-IID & NonIID Tasks Table shows performance comparison of our models and naive Fed-SSL algorithms on Batch-IID and NonIID tasks under the two different scenarios. We observe that our model outperforms all naive Fed-SSL baselines for all tasks and scenarios. In particular, under labels-at-server scenario, which is more challenging than labels-at-cliem scenario, we observe that the naive combination models significantly suffer from the forgetting issue and their performances keeps deteriorating after certain communication round. This phenomenon is mainly caused by the base models failing to properly perform disjoint learning, in which case the learned knowledge from the labeled and unlabeled data causes inter-task interference. Contrarily, our methods show consistent and robust performance regardless where the labeled data exists, which shows that our decomposition techniques effectively handles the challenging disjoint learning scenario. In addition, when the class- wise distribution is imbalanced for each client (Non- task), we observe that the base models' performance slightly drops by 1-3%p, while our methods show consistent.
7

---

Published as a conference paper at ICLR 2021
Table 2:
Averaged Local Performance on Streaming-NonIID Task We use 10 clients 100 rounds. We
measure averaged local model accuracy and communication costs. Note that the SL models learn on both and U with full labels, and are utilized as the upper bounds for each experiment.
Dishion-MNIST Atreaming-Nonlll Task with 10 Clients (K=10, F=1.0, H=2)
Labels-at- Cleeitt Scenario
albele- Scenario
Methods
Acc. ))
S2C Cost
C2S Cost
Acc. (%)
S2C Cost
C2S Cost
Local-SL
87.19 + 0.36
N/A
N/A
N/A
N/A
N/A
Local-UDA
70.70 022
N/A
N/A
N/A
N/A
N/A
Latth
62.62 0.30
N/A
N/A
N/A
N/A
N/A
FedProx-SL.
82.06 + 0.26
100 %
100 %
77.43 +0442
100 %
100 %
TedlPrx-UDDA
73.71 + 0.37
100 %
100 %
83.34
4021
100 %
100 %
FedProx-Fix Fixch
62.40 0.43
100 %
100 %
73.71 1.032
100 %
100 %
FedMatch (Ours)
77.95
+ 0.14
37
48 %
84.15
+ 0.31
14
%
Inter-Client Consistency
Parameter Decomposition
Accurscy on Labeld Data
Buich (CPAN-10)
62-
Accurnry (84)
Accumny (8)
Accuracy (N)
75-
Tedilakm
Fediroct/DA
(ac IEPLEA
30
Fedifarch wis IDCL
30
25-
FedtMarch
Fackaahh
50
100
150
200
50
100
150
200
10
15
20
25
30
Communication Round
Communication Round
Communication Round
Number of per Class
(a) Inter-Clien Consistency
(b) Sigma Psi
(c) Inter-Task Interference
(d) Number of Labels
Figure 6: Ablation Study and Additional Analysis on FedMatch Algorithm We study effectiveness of cach components of our method, (a) inter-client consistency loss and (b) parameter decomposition. (c) We effectively tackle the inter-task interference. (d) Performance improvement of method when laheled data is increased.
This shows that our inter-client consistency effectively enhances consistency with the helper agents selected from server based on model similarity, which is good at, in particular, class-imbalanced tasks. We also visualize the test accuracy curve for our models and naive Fed-SSL in Fig. 5. Our method (Red line) trains faster and consistently outperforms the base models, and is most robustness against inter-task interference in both scenarios. For analysis on the averaged communication costs, please see Section 1 in the Appendix.
Results on Streaming-NonIID Task Table 2 shows averaged local model performance on Streaming-Nonlie tasks with 10 synchronized clients. For the labels-at-client scenario, our proposed method outperforms local-SSL. and naive Fed-SSL models with large margins, 4-15%p, except for the SL models. There is no huge difference of performance between local SSL and Fed-SSL models, and this implies that our method effectively utilizes inter-client knowledge in this streaming setting. In the labels-at-server scenario, interestingly, the performance of FedProx-SL decreases by around 5%p compared to the tubels-at-clies scenario, while Fed-SSL models obtain improved performance. We conjecture that this is because, for streaming situation, the model may not sufficiently train on the new data, while Fed-SSI. models overcomes by utilizing only the consistent pseudo-labels, Even on this task, FedMatch outperforms all baselines with significantly smaller communication cost on average (see Section B.I for detailed analysis of the averaged communication costs).
Effectiveness of Inter-Client Consistency To show the effectiveness of our inter-client consistency loss, we eliminate the loss, while learning on Batch-IID task with 100 clients (F=0.05). In Figure 6 (a), when we remove our inter-client consistency loss, we observe that the performance has slightly dropped (Pink line) from one with the loss term (Red line). This gap clearly tells us that our inter- client consistency loss improves model consistency across multiple models while keeping reliable knowledge. Interestingly, our model without inter-client consistency loss still outperforms base models. This additionally implies that our another proposed method, parameter decomposition for disjoint learning, also effectively enhances model performance.
Effectiveness of Parameter Decomposition Our model with parameter decomposition alone, without inter-client consistency loss, outperforms base models. For further analysis, we show the effect of each decomposed variables, a and v) in Figure 6 (b). Removing either of a and 1é results in substantial drop in the performance, with larger performance degeneration when dropping a, which captures much more essential knowledge from labeled data (Green line). Such decomposition is effective since there exists knowledge interference between supervised learning and unsupervised learning. We show this with experiment where we perform semi-supervised learning with 5 labeled instances per class and 1, 000 unlabeled instances for 100 rounds. We measure accuracy on the labeled set each training steps. As shown in Figure 6 (c), method effectively preserves
8

---

Published as a conference paper at ICLR 2021
learned knowledge from labeled set, while other base models suffer from knowledge interference. This effective separation of supervised and unsupervised learning tasks enhances the overall performance of our methods even without inter-client consistency loss as shown in Figure 6 (a) (shown in pink).
Number of Labels per Class We increase the number of labels per class for each client in a range of 1, 5, 10, and 20 on Batch-IID task (CIFAR-10). Our method shows consistent performance improvement as the number of labels increases. Interestingly, we observe that baseline models, SeuPPrx-UDD/FixMattee
show performance degradation even when the labeled data increases (5 + 10). These results show that our method effectively utilize knowledge from labeled and unlabeled data in federated semi-supervised learning settings, while other naive combinations of FSSL could fail to learn properly from labeled and unlabeled data in federated learning framework.
7 RELATED WORK
Federated Learning A variety of approaches for averaging local weights at server have been introduced in the past few years. FedAvg (McMahan et al., 2017) performs weighted-averrging on local weights according to the local train size. FedProx (Li et al., 2018) uniformly averages the local updates while clients perform proximal regularization against the global weights, while FedMA (Wang et al., 2020) matches the hidden elements with similar feature extraction signatures in layer-wise manner when averaging local weights. PFNM (Yurochkin et al., 2019) introduces aggregation policy which leverages Bayesian non- parametric methods. Beyond focusing on averaging local knowledge, there are various efforts to extend FL to the other areas, such as continual learning under federated learning frameworks (Yoon et al., 2020a) inspired by parameter decomposition techniques proposed by (Yoon et al., 2020b). Recently, interests of tackling scarcity of labeled data in FL are emerging and discussed in (Jin et al., 2020; Guha et al., 2019; Albaseer et al., 2020).
Semi-Supervised Learning While there exist numerous work on SSL,, we mainly discuss consis- tency regularization approaches. Consistency regularization (Sajjadi et al., 2016) assumes that the class semantics will not be affected by transformations of the input instances, and enforces the model output to be the same across different input perturbations. Some extensions to this technique perturb inputs adversarially (Miyato et al., 2018), through dropout (Srivastava et al., 2014), or through data augmentation (French et al., 2018). UDA (Xie et al., 2019) and ReMixMatch (Berthelot et al., 2019a) use two sets of augmentations, weak and strong, and enforce consistency between the weakly and strongly augmented examples. Recently, in addition to enforcing consistency between weak-strong augmented pairs, FixMatch (Sohn et al., 2020) performs pseudo-label refinement on model predictions via thresholding. Entropy minimization (Grandvalet & Bengio, 2004) which enforces the classifier to predict low-entropy on unlabeled data, is another popular technique for SSL. Pseudo-Label (Lee, 2013) constructs one-hot labels from highly confident predictions on unlabeled data and uses these as training targets inn standard cross- coos-entropy loss. MixMatch (Berthelot et al., 2019g 9c) performs sharpening on target distribution on unlabeled data, to further refine the generated pseudo-label.
CONCLUSION
In this work, we introduced two practical scenarios of Federated Supervisd Learning (FSSL) where each client learns with only partly labeled data (Labell-at-Clit scenario), or supervised labels are only available at the server, while clients work with completely unlabeled data (Labels-at-Servel at-Ser scenario). To tackle the problem, we propose a novel method, Federated Matching (FedMatch), which introduces the inter-client consistency loss that aims to maximize the agreement between the models trained at different clients, and the parameter decomposition for disjoint learning which decomposes the parameters into one for labeled data and the other for unlabeled data for preservation of reliable knowledge, reduction of communication costs, and disjoint learning. Through extensive experimental validation, we show that FedMatch significantly outperforms both local semi-supervised learning methods and naive combinations of federated learning algorithms with semi-supervised learning on diverse and realistic scenarios. As future work, we plan to further improve our model to tackle the scenario where pretrained models deployed each client adapts to a completely unlabeled data stream (e.g. on-device learning of smart speakers).

---

Published as a conference paper at ICLR 2021
Acknowledgements This work supported by Samsung Research Funding Center of Samsung Electronics (No. SRFC-IT1502-511 Samsung Advanced Institute of Technology. Samsung Electron- ics Co., Ltd., Next-Generatin Information Computing Development Program through the National Research Foundation of Kor)a(NRF) funded by the Ministry of Science, ICT & Future Plannig (No. (01663C447952644) the National Research Foundation of Korea(NRF) grant funded by the Korea govermmen((MSITT (201 ASA 1059921), and Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UDI19003 IRD). Also, this work was supported by Institute of Information communications Technology Planning Evaluation (IITP) grant funded by the Korea goveramen(MMSIT) (No.2019.0-000755 Artificial Intelligence Graduate School Program(KAISTT)
REFERENCES
Abdullatif Albaseer, Bekir Ciftler, Mohamed Abdallah, and Ala Al-Fuqaha. Exploiting unlabeled data in smart cities using federated learning. 01 2020.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.099 1.09785, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning, arXiv preprint 190Xvv1005.022 2019b.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems 32, pp. 5049-5059. Curran Associates, Inc., 2019c.
Yang Chen, Xiaoyan Sun, and Yaochu Jin. Communicationn-ffiiinn federated deep learn- ing with asynchronous model update and temporally weighted aggregation. arXiv preprint arXiv: /903.07424, 2019a.
Yujing Chen, Yue Ning, and Huzefa Rangwala. Asynchronous online federated learning for edge devices. arXiv preprint 19Xiv:1911.02134 2019b.
Muhammad EH Chowdhury, Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muhammad Ab- dul Kadir, Zaid Bin Mahbub, Khandaker Reajul Islam, Muhammad Salman Khan, Atif Iqbal, Nasser Al-Emadi, et al. Can help in screening viral and covid-19 pneumonia? arXiv preprint arXiv:2003.13145. 2020.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical data augmentation with no separate search. CoRR, abs/1909. 13719, 2019. URL attp://arxiy: org/abs 10009117111
Geoff French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In International Conference on Learning Representations, 2018. URL htkLttttp:////eeeeeiiii net /
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, pp. 529-536-666 Cambridge, MA, USA, 2004. MIT Press.
Neel Guha, Ameet Talwikar, and Virginia Smith. One-shot federated learning. 02 2019.
Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. A survey towards federated semi-supervised learning. 02 2020.
Dong-Hyun Lec. Pscudo-label: : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, arXiv preprint arXiv: 18/iv:88/2..0 18/2.0//22.007 2018.
10

---

Published as a conference paper at ICLR 2021
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication- -efficiinnnff learning of deep networks from decentralized data. In AISTATS, 2017.
Takeru Miyato, Shin ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. PAMI, 2018
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg. I. Guyon, and R. Garnett (eds.), Advances Neural Information Processing Systems 29, pp. 1163-1171. Curran Associates, Inc., 2016.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4548-4557, Stockholmsmmäsann, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http: ppw.eedeiceeeeessooooeeeeees.sse mlr www......./.v.e.... /t8.rssssv0//eoe serra1// html.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi- ani-super learning with consistency and confidence. arXiv preprint arXiv:2001.0766: 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-195) 1958, 2014. URL http:// http:///mmlr.r////...eee. httgg/apers//vii /v15/
srivastava14 ahtml.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- erated learning with matched averaging. In International Conference on Learning Representations, 2020. URL https: / wttp//ppeevvveeeeeeeeeeeee (tte//oooommn.-kIglgggg_1
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh- Thang Luong, and Quoe V. Le. Unsupervised data augmentation for consistency training. 2019.
Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Funho Yang, and Sung Ju Hwang. Federated continual learning with weighted inter-client transfer. In arXiv preprint arXiv:2003.03 196, 2020a.
Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust contin- ual learning with additive parameter decomposition. International Conference on Learning Representations, 2020b. URL httpt:tt/////// /ttp://ppeeeeeeee review.ww..eee.e........ net.
www.eeewwwweeeeeeeeeeeeeeeee forum? d=r1 jd
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. 2019.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint 1rXiv:1806.0053: 2018.

---

Published as a conference paper at ICLR 2021
Organization We describe detailed experimental setups in Section A, such as baselines (Sec- tion A.1). model architecture (Section A.2), and the training configurations (Section A.3). We also provide additional analysis and experimental results in Section B, including analysis on communi- cation costs (Section B.1) and number of labels per class (Section B.2), experiments on real-world dataset (Section B.3), different backbone architecture (Section B.4), fraction of clients per communi- cation round (Section B.5).
A EXPERIMENTAL DETAILS
We describe our experimental setups in detail, such as our baseline models, network architecture that used for all base models and our method, and the detailed training setups.
A.1 BASELINE MODELS
We consider UDA (Xie et al., 2019) and FixMatch (Sohn et al., 2020) as our baselines, since they are state-of-ther SSL models and are based on the consistency-bart based mechanisms that are conceptually similar to our inter-client consistency loss. We reimplement UDA with the Training Signal Annealing (TSA) and exponential scheduling for its best performance as reported in their paper (we use RandAugment (Cubuk et al., 2019) for consistency regularization with random magnitude). We also reimplement FixMatch algorithms with strong augmentation as RandAugment (Cubuk et al., 2019). For weak augmentation (fip--ad--shift, however, the performance has significantly dropped when we apply the weak augmentation, we use original images rather than weakly augmenting the images. We fix confidence threshold 7=0.85 for all FixMatch and our model experiments. For federated learning frameworks, we use FedAvg (McMahan et al., 2017) and FedProx (Li et al., 2018) algorithms since they are the standard baselines for federated learning and can be easily combined with the SSL baselines. Detailed hyper- parameter settings are described in Table 4.
A.2 NETWORK ARCHITECTURE
Table 3: Network Architecture of ResNet-9
We build ResNet-9 networks as our base architecture for all base model and our method. In the architecture, the first two convolutional neural lay- ers have 64 and 128 filters and the same 3 x 3 kernel sizes followed by 2 x 2 max-po max-pooling layer. Then we have skip connection between the subse- quent two convolution layers with 128 filters. We then double the filter size from 128 to 256 with the next conv layer and down-sample via the follow- ing 2 x 2 max-poolle max-pooling layer. We repeat the previous step, such that we have 512 filter size and 4 X 4 kernel size.
Layer
Filter Shape
Stride
Output
Input
N/A
N/A
x 32 3
Conv
3 3 x x 64
32 x 32 x 64
Conv
3 x x 64 x 128
1
32 x 32 x 128
Pool I
2 2
2
16 16 x 128
Conv
x 128 x 128
16 x 128
Conv
3 3 x 128 x 128
1
16 x 16 x 128
Conv
3 x x 128 x 256
1
16 X 16 x 256
Pool 2
2 2
2
8 X 8 256
Conv
3 x x 256 x 512
1
8 X 8 512
Pool 3
2x2
2
4 4 512
Conv
3 x 3 x 512 x 512
1
4 4 512
Conv
3 x 3 x 512 x 512
1
4 512
Pool 4 Softmax
4
4
1 1 512
512 10
N/A
TxTXI
Then, we perform another skip connec- tion through the two subsequent kernel size from 4 x 4 to x 1, then perform softmax classifier with the last fully connected layer. All layers are equally initialized based on the varaiance scalining method. The model architecture is described in Table 3.
layers with 512 filters. As final step, we down-sample the
A.3 TRAINING DETAILS
We use Stochastic Gradient Descent (SGD) to optimize our model with initial learning rate 1e-3, We also adopt adaptive learning rate decay which is introduced by (Serra et al., 2018). The learning rate strategy gradually reduces the learning rate by a factor of 3 for every 5 epochs that validation loss does not consecutively decreases. We use L2 weight decay regularization on the base architecture with L2 factor to be Ic-4. All comparison shown in Table 4. In the table, denote I.PC number of labels per class for each
and other training setups are equally set for fair
12

---

Published as conference paper at ICLR 2021
Features
g
Examples
Train set TD
Split 12 into Hand
Dataset Partition for Experimental Tasks split dataset into a set labeled data and a set of unlabeled data 2. U is divided into K subsets which are distributed to K clients (Batch Task), For streaming tasks, we further split all instances in each subset into T subsets for 7 streaming steps. For class-imin meas-immalanced tasks, additionally control the number of instances per class for cach client.
Figure Illustration
Table Hyper-IPaie Parameters & Training Setups We provide all hyper-yppaaa parameters and training setups for all
baseline models and our method. Detailed ppraee-rrrmeterss are also available in the code.
Labels-at- Client Scenario
Methods
Ir
wd
A
Au
Aices
AL
d
LPC
B
Belient
B.S
Berrver
SL
le-3
le44
10
5
10
100
le-2
UDA
le-3
le44
10
5
10
100
le-2
FixMatch
le-3
le44
5
10
100
le-2
FedMatch
le-3
le-4
10
lc-2
le-4
5
10
100
Labels-at-Server Scenario
SL
le-3
le-4
10
TO0
100
100
lc-2
UDA
le-3
le-4
10
100
100
lc-2
FixMatch
le-3
le-4
10
100
100
100
le-2
FedMatch
le-3
le-4
10
le-2
le-5
10
100
100
100
client (or at server). B and B" denote batch-size of labeled set and unlabeled set 24 μ is a hyper-parameter for FedProx framework. We additionally provide visual illustration of dataset configuration. Please see Figure
B ADDITIONAL ANALYSIS AND EXPERIMENTAL RESULTS
In this section, we additionally provide more analysis and experimental results, such analysis on communication costs and number of labels per class, experiments on real-world dataset, different backbone architecture, fraction of clients per communication round.
B. THE EFFICIENT COMMUNICATION OF FEDMATCHI
Since the actual bit-level compression techniques are rather implementation issues, which are beyond research scope, only consider the reduction of the amount of information that needs to be transmitted between the server and the client. To minimize the communication costs, we not only learn // be sparse, but also subtract the parameters between server and client, such that A = vif d6C and Ao = of of then send only the difference, Axh and Ao, as sparse matrices from both directions of server-to-client (S2C) and client-to-server (C2S). Here. S2C and C2S costs are the sums of Ao and Av. When transmitting the difference for each parameter to either way, we discard almost unchanged values in element-wise manner, that only meaningful neural values can be updated either server- or client-side. We observe that the range of the threshold values is from le-5 to 5e-5, such that the model performance is well preseeeed and significantly harmed, while maximizing the reduction of communication costs.
As shown in Figure 8, we observe that both the S2C and C2S costs are gradually decreased during the learning phases both batch and streaming datasets under labels-at-client (Figure 8 (a) and (b)) and labels-at-server scenarios (Figure 8 (c) and (d)). This is because each parameter separately learns different tasks (i.e. supervised and unsupervised learning) effectively, which results in rapid convergence to optimal points, respectively. Further, for the labels-at-server scenario, since labeled data is not available at client, client even does not need transfer Ag to the server (see Figure (c) right and (d) right), which is extremely efficient than the labels-at-client scenario where both Ag and Agh must be transferred to the server. For both scenarios, indeed, S2C contains the cost
13

---

Published as a conference paper at ICLR 2021
300
Server Client Cost
Client Server Cost
Communication Coet 8
200
Client
220
Client to Server Cost
Communitaiin
200
S2C
40
Communiccation cost [%: Communicaioo DoP
100
C25
S2C
150
bd
C25
150
Communication
100
100
50
50
50
25
100
150
50
150
20
100
20
40
60
80
Communication Round
Communicatio Round
Communication Round
Communication Round
(a) Batch-NonIID (Labels-at-Clicee Client)
(b) Streaming NonlII (Labels-at-Client
300
Server
100
to Server Cost
Communication Cost [8)
200
Communication (%)
100
Server
S2C
C25
S2C
C2S
200-
Ao
20
Communicttion
150
Axp
100
by
100
s0
25
150
200
100
40 Communication Round
Communication Round
Communication ation Round
Communiration Round
(c)
(Labels-at- Server)
(d) Streaming NonllDD (Labels-at-Servel
Figure 8: Communication Cost Curves FedMatch (ResNet-9) Corresponding to the Table and measure the communication costs each parameters, Ao and At, during training phase. The communication costs under the labels-at-clicnt scenario are visualized in (a) and (b) on the upper row. (c) and (d) on the lower row represent the communication costs under sabels-at-serves scenario.
COVID11 Daraeet Steeees-CClent
COVID Datenet 1: 8- Server
COVTD-19 Radiography Dataset
Labels-at-Client
Labels-at-Server
Acc.noc
Methods
Acc. %)
Acc.(%) %6)
FodProx*UDA
FedProx_ UDA
Flutatc
FeoProx, FlkMatch
F.Prx-UDA
74.24 + 0.25
80.11 = 0.18
FedMltch (Durs)
FedMatch (Durs)
E.Px-FixMMr
70.02 0.28
72.15 0.14
Commuttctton 120
Commideration Sound
FedMatch
78.67 + 0.23
84.32 + 0.11
(a) Labels-at-Client
(b) Labels-at-
Figure 9: Experimental Results on COVID-19 Radiography Dataset. Left: Performance comparison of our method (FedMatch) with the naive federated supervisd learning algorithms TFddPox-UDA/FixMauch
Right: Test accuracy curves corresponding to the left performance table. Our method trains stably and consistently outperforms all hase models.
-ド= of helper agents, such that 1 = Σ vi o/. However, as shown in Figure 8, transmitting multiple helper agents (H=2 our experiments) does not significantly affect the total S2C costs thanks to novel decomposition techniques as well efficient subtracting method, such that model reconstruction can possible without meaningful information loss at either server- or client-side.
B.2 FURTHER ANALYSIS OF THE NUMBER OF LABELS PER CLASS
We explain our analysis of the number of labels per class under Section 6.2, and here we further provide additional experimental re- sults. We conduct experiments with our method without the de- composition technique. As shown in Table 5, our method without de- composition technique (indicated as FedMatch (w/o) shows not much performance improvement when the number of labels per class increases from to 10 (around 3.x%p) than from 1 to 5 (around 9.x%p) and from 10 to 20 (10.x%p), which are the similar tendency with the baseline models in Table 5. However, with the decomposition technique, method shows consistent performance improvement, which implies that our proposed technique has the effectiveness to handle inter-tase task interference and preserve reliable knowledge in the novel federated semi-supervised learning scenarios.
Table 5: Analysis of Number Labels per Class
Number of Labeled Examples per Class
5
10
20
Methods
Acc(%)
Acc.(%)
Acc.(%)
Acc, (%)
UUAA
31.95
47.45
41.4
47.
FedPrx*FxMtch
30.01
47.2
34.25
44.5
FedMaich (w/o)
37.7
47.31
31.15
62.7
TedMatch
337.65
34.5
60.65
66.1
14

---

Published as a conference paper at ICLR 2021
Table 6: Performance Comaprison utilizinng AlexNet-Like architecture We use 100 clients for 100 rounds for streaming task and 200 rounds for batch tasks. We measure global model accuracy, while varying experimen- tal settings (i.e. fraction of available clients and the accessibility of labeled data).
Experiments based on AlexNet- Architecture
Streaming-Nonie NonIID (F=1.0)
Batch- Match-IID (Labels-at- CliCnet
Cheienttt-Cheent
Labels-at- -t-erer
F=0.05
F=0.10
F=0.20
Methods
Acc.(%) %)
Acc. (%)
Ace, ))
Acc,(%)
Acc.( (%)
FedAvg-SL.
68.20 0.29
70.51 a.11
47.23 0.31
47.87 am
48.73 0.15
FedProx-SL
68.47 =
70.55 a.7
47.54 + 0.28
48.01 t017
49,20 = 0.64
FedAvg-(11 -
52.25 1 0.04
46.28 0.330
35.27 0.29
35.20 153
36,21 1.12
FedProx-UDA
52.84 0.15
46.35 1:331
34,94 1 0.46
36,67
1.73
35.80 1:43
SadAvg-FixMaatc
37.09 0.89
52.67 1.78
32.33
36.27
1:3
37.61 1.05
FedProx-FixMateh
57.12 0:41
51.51 0:300
36,83
1 0.23
36.37
1 0.99
37.40
1: 0.188
FedMatch (Ours)
63.84 + 0.18
59.12 + 0.35
41.67
0.32
41.97 + 0.34
42.18 0.27
B.3 EXPERIMENTS ON REAL-WORLD DATASET
To show our method consistently work with real-world dataset, we further conduct experiment on COVID-19 Radiography Dataset (Chowdhury et al., 2020) which is real-world dataset that consists of X-ray images from COVID and non-COVID patients. The COVID-19 dataset contains 219 X-ray images from the patients diagnosed of COVID-19, 1341 images from normal (healthy) patients, and 1341 images from patients diagnosed of viral pneumonia. We use 10 clients with a fraction of 1.0 (communication rate). We use 5 labeled examples per class for each client, leaving the rest of the image as unlabeled. We find this setting to be realistic as the datasets are, since we may not not have skilled radiologists that can fully label the X-ray images taken the local hospitals. We compared our method against baselines which naively combine semi-supervised learning and federated learning models during training 100 rounds. As shown in the left table of Figure 9, our method consistently outperforms all base models with large margins (around 4%p--0%p) in both scenarios. The test accuracy curves in Figure 9, we can see that our method trains faster than the base models and shows more stability during training. We believe that these additional experimental results further strengthen our paper.
B.4 BACKBONE ARCHITECTURE
Most existing works federated learning considers smaller networks since the focus is on-device learning of low-resource devices, and thus we utilize a smaller backbone networks than ResNet-9, To verify that our method also successfully works on the smaller & different architecture, we adopt AlexNet-Like (Serra et al., 2018), of which the first three layers are convolutional neural layers with 64, 128, and 256 filters with the 4, 3, and 2 kernel sizes followed by the two fully-connected layers of 2048 units, while 2 x 2 max-poooo max-pooling layers are followed after each convolutional layer. In Table 6, for both Streaming-NonIle
and Batch-IID tasks, our methods still outperforms all naive Fed-SSL. models with the similar tendency with that of the results based on ResNer-9. This shows that our methods can be applicd to the smaller and different base networks, and still effectively utilize inter-client and reliable knowledge across multiple clients than naive algorithms.
B.5 FRACTION OF AVAILABLE CLIENTS PER COMMUNICATION ROUND
To see the effect of participation rate of clients, we increase the fraction of available clients per com- munication round in a range of 0.05, 0.10, and 0.2. This means, for every round, server can connect to the arbitrary 5, 10, and 20 available clients out of 100 clients, and perform distributed learning on each individual local data through each client and updates global knowledge by aggregating the locally-learne knowledge. The experimental results are shown in Table 6 Batch-IID. We observe that the performances of all models are slightly improved when the faction increases. This natural that the more knowledge the client updates, the more the global performance is improved. We are not able to find any extraordinary phenomenon on the fraction of the number of clients per round.
15


Reviews:
[{"review_id": "yJmqbNr4cqf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In many real-world applications, local data are not always well labeled or fully labeled, and most of them are unlabeled. This paper introduced a novel learning paradigm, Federated Semi-Supervised Learning (FSSL), to handle the federated learning from distributed and partially labeled data within the clients. Under this paradigm, the paper studies two different scenarios where partly labeled data appear on client nodes and merely appear on the sever node. In the introduction, the authors clearly presented the motivations of introducing FSSL and showed the differences between the proposed method and some existing ones.\n\nIn the FSSL, the key technique is Federated Matching (FedMath), which learns inter-client consistency between clients, and decomposes parameters to reduce both interference between supervised and unsupervised tasks, and communication cost. The Inter-Client Consistency Loss was most directly taken from some existing method. Thus, the critical innovation is the Decomposition of parameters, which has several benefits to the learned models. Basically, the proposed method is intuitively and technically sound.\n\nThe authors conducted a batch of experiments to evaluate their proposed method on three different tasks under both labels-at-client and labels-at-server, compared with some baselines. The experimental results are positive. However, there are some minor issues in the experiment section. The authors mentioned that the proposed model can work under the scenario that some labels are not correct (noisy labels). However, this point was not verified in the experiment section at different error labeling rates. It is shown that FedProx-UDA/FixMatch degrade when the labeled data increases from 5 to 10 in Figure 6(d). What causes this degradation and how the proposed method avoid it?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "vXd586WcUsW", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of learning a joint model for different local data sets. Each model is trained individually based on each individual local data set, and the inference is performed by regularizing the results of different models.  The main challenges come from the facts that each local data set belongs to different client hence the data is not shared and the labeled data in some local data set may be not sufficient. To utilize the unlabeled and labeled data, the paper proposes using unsupervised, semisupervised and supervised learning models. To incrementally training the model with different types of data, authors  propose to decompose the model parameters as supervised and unsupervised model. The description of the problem is complicated and little confusing. \n\nThe problem seems interesting, but one of my main main concerns is that I do not think authors solve the original challenge. The original challenge is that clients do not want to share data for model training. But the method proposed utilizes all the local data sets to learn the model.\n\nMy another main concern is that the novel contribution of this work is not strong. My impression of the method proposed in this work is pretraining the model with labeled data and fine-tuning it with unlabeled data with constraints to make the fine-tuned model as close as the pretrained model. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nIUI7yfle-Q", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new semi-supervised federated learning algorithm (FedMatch). Two scenarios are studies: 1) label-at-client (labeled and unlabeled data are at client). 2) label-at-server (labels are at server and unlabeled data are on client). The authors propose a disjoint learning which decomposes the model \\theta into supervised \\sigma and unsupervised $\\psi$ such that $\\theta = \\sigma + \\psi$ . the model is trained, in an alternating manner, to minimize the two loss functions for the supervised eq (4) and unsupervised model components eq (5) with inter-client consistency. The experimental results show an improvement w.r.t. to SOTA in terms of performance.\n\nThe paper is sometime difficult and its clarity could be improved. I would consider the novelty of the method, from machine learning prospective, to be somewhat borderline as it uses known elements and adapt them to the introduced learning scenarios but overall the proposed solution is interesting as it seems to work well in practice. \n\nsome detail comments: \n\n- $\\pi$ in one of the terms in the norm needs to be removed in inter-class consistency loss ||p θ (y|π(u)) − p θ (y|π(u))||\n- The disjoint learning scenario was not well introduced in paragraph \"Parameter Decomposition for Disjoint Learning\"\n- a missing section reference in (see section ??).\n- Related to the reduction of communication costs: the bit-encoding of the model difference between the server and client is not clarified, i.e., for example if the difference is coded using 64 bits and also the same encoding for the models there is no gain in communication. \n- In Table 1, the bold should be indicative of the best method per column", "extra_scores": {}, "extra_text": {}}, {"review_id": "2BygUJTZlsb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "AFTER READING THE AUTHORS' REPLY, I HAVE CHANGED MY RATING TO 6.\n\nEven though authors introduce an interesting approach for a problem that has many potential practical applications, the paper suffers from two main weaknesses:\n1) it is poorly organized, which makes it very hard to follow\n2) it lacks a compelling, real-world dataset\n\nIn terms of paper organization, in this reviewer's opinion, it may be beneficial to decouple and present in a serial manner the solutions to the two main scenarios (i.e., labels at client vs server). The parallel presentation makes the story harder to follow, as the reader has to keep switching context from one scenario to the other. The very long captions of Figures 2 & 3 are not helpful, and Section 4 comes a bit out of nowhere, given that the pseudo-code for the two proposed algorithms only appears in the appendix. \nA better approach would be to use the body of the paper for an illustrative running example and the pseudo-code of the algorithms, while relegating the details to the appendix.\n\nIn terms of the empirical validation, it is disappointing to see  that you are using synthetic datasets. The paper would greatly benefit from having at least one real world application domain in which the proposed approaches \"move the needle.\" There are far too many papers in which a novel approach does great on synthetic data without impacting the state-of-the-art results on real-world domains.\n\n\nOther:\n- caption of Figure 1: the data is \"available\" rather than \"given\" to the local client (twice, under both \"a)\" and \"b)\")\n- page 2: you make references to Figure 6 a & b, but you mean Figure 1 a & b\n- page 3: in line 3 of the 1st paragraph of 3.1, you refer to \"D\" without defining it\n- please spell-check & proof-read the paper: \n    - \"Cleint\" --> \"Client\" in the header of Table 1; \n    - \"manly\" --> \"mainly\" on page 6\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes the Federated Matching algorithm as a novel method to tackle the problems in federated learning. The paper is well-written and original, and it contributes to the state-of-the-art. 
Decision: Accept (Poster)