Unsupervised Federated Learning is Possible

1 INTRODUCTION . Federated Learning ( FL ) has received significant attention from both academic and industrial perspectives in that it can bring together separate data sources and allow multiple clients to train a central model in a collaborative but private manner ( McMahan et al. , 2017 ; Kairouz et al. , 2019 ; Yang et al. , 2019 ) . So far , the majority of FL researches focused on the supervised setting , requiring collected data at each client to be fully labeled—this may hinder the spread of FL in practice since labeling large-scale training data can be extremely costly due to laborious mannual annotations , and sometimes may not even be possible for privacy concerns , e.g. , in medical diagnosis ( Ng et al. , 2021 ) . In this paper , we are interested in a challenging unsupervised FL problem where only unlabeled ( U ) data are available at each client . Our goal remains to train a classification model that can predict accurate class labels in the FL setup . It is often observed that each client collects their own data at different time points or from different geographic locations , e.g. , a hospital ( data center ) may store patient data every month or a particular user ( mobile device ) may take photos at different places . Therefore , we consider a realistic setting that the U data at a client come in the form of separate data sets : the data distribution of each U set and the number of U sets available at each client may vary . Without any labels , FL becomes significantly harder than before . In this work , we show that the aforementioned unsupervised FL problem can be solved if the class prior probability shifts between the available U sets at each client , and these class priors are known to the clients . Such a learning scenario can be conceivable in many real-world problems . For example , the hospital may not release the patients ’ diagnostic labels due to privacy reasons , but the morbidity rates of diseases ( corresponding to the class priors ) may change over time and can be accessible from related public medical reports ( Croft et al. , 2018 ) ; moreover , in many cases it is possible to estimate the class priors much more cheaply than the ground-truth labels ( Quadrianto et al. , 2009a ) . Based on these findings , we propose the federation of unsupervised learning ( FedUL ) ( cf . Figure 1 ) , where the original clients with U data are transformed to surrogate clients with ( surrogate ) labeled data and then supervised FL can be applied . The difficulty is how to infer our wanted model for the original classification task from the surrogate model learned by the surrogate task . We solve this problem by bridging the original and surrogate class probabilities with certain injective transition functions . Then we implement it by adding a transition layer to the output of each client model , so that the learned model is guaranteed to be a good approximation of the original class probability . On one hand , FedUL is a very general and flexible : it works as a wrapper that transforms the original clients to the surrogate ones , such that it is compatible with many supervised FL methods , e.g. , FedAvg , and add no additional communication cost . On the other hand , FedUL is computationally efficient and easy-to-implement : since the added transformation layer is fixed and determined by the class priors , FedUL adds no more burden on hyper-parameter tuning and/or optimization . Our contributions can be summarized as follows : • Methodologically , we propose a novel FedUL method that employs a surrogate FL task , providing new perspectives for dealing with U data in FL . • Theoretically , we prove that the optimal global model learned by FedUL with only U data converges to the optimal global model learned by supervised FL with fully labeled data under mild conditions . • Empirically , we demonstrate the effectiveness of FedUL on benchmark and real-world datasets . Related Work : FL has been extensively studied in the supervised setting . One of the most popular supervised FL paradigms is FedAvg ( McMahan et al. , 2017 ) which aggregates the local updates at the server and transmits the averaged model back to local clients . In contrast , FL in the unsupervised setting is less explored . Recent studies have shown the possibility of federated clustering with U data ( Ghosh et al. , 2020 ; Dennis et al. , 2021 ) . However , these clustering based methods rely on geometric or information-theoretic assumptions to build their learning objectives ( Chapelle et al. , 2006 ) , and are sub-optimal for classification goals . Our work is intrinsically different from them in the sense that we show the possibility of federated classification with U data based on empirical risk minimization ( ERM ) ( Vapnik , 1998 ) , and therefore the optimal model recovery can be guaranteed . Learning from U datasets has also been studied in the classical centralized learning . Mainstream directions include learning with label proportions ( LLP ) ( Quadrianto et al. , 2009b ) and Um classification ( Lu et al. , 2021 ) . The former learns a multi-class classifier from U datasets based on empirical proportion risk minimization ( EPRM ) ( Yu et al. , 2014 ) , while the latter learns a binary classifier from U datasts based on ERM . EPRM is inferior to ERM since its learning is not consistent . Yet , it is still unexplored how to learn a multi-class classification model from U datasets based on ERM . To the best of our knowledge , this work is the first attempt to tackle this problem in the FL setup , which is built upon the binary Um classification in the centralized learning . 2 STANDARD FEDERATED LEARNING . In this section , we review standard FL of a classification model . We consider a K-class classification problem with the feature space X ⊂ Rd and the label space Y = [ K ] , where d is the input dimension and [ K ] : = { 1 , . . . , K } . Let x ∈ X and y ∈ Y be the input and output random variables following an underlying joint distribution with density p ( x , y ) , which can be identified via the class priors { πk = p ( y = k ) } Kk=1 and the class-conditional densities { p ( x | y = k ) } Kk=1 . Let f : X → R K be a classification model that assigns a score to each of the K classes for a given input x and then outputs the predicted label by ypred = argmaxk∈ [ K ] ( f ( x ) ) k , where ( f ( x ) ) k is the k-th element of f ( x ) . In standard FL setup with C clients , each client c ∈ [ C ] has access to a labeled training set Dc = { ( xci , yci ) } nc i=1∼pc ( x , y ) of sample size nc and learns its local model fc by minimizing the following risk : Rc ( fc ) : = E ( x , y ) ∼pc ( x , y ) [ ` ( fc ( x ) , y ) ] , ( 1 ) where E denotes the expectation , ` : RK × Y → R+ is a proper loss function , e.g. , the softmax cross-entropy loss , i.e. , ` ce ( fc ( x ) , y ) = − ∑K k=1 1 ( y = k ) log ( fc ( x ) ) k = − log ( fc ( x ) ) y , and 1 ( · ) is the indicator function . In practice , ERM is a common practice that computes an approximation of Rc ( fc ) based on the client ’ s local data Dc by R̂c ( fc ; Dc ) = 1nc ∑nc i=1 ` ( fc ( x c i ) , y c i ) . The goal of standard FL ( McMahan et al. , 2017 ) is that the C clients collaboratively train a global classification model f that generalizes well with respect to p ( x , y ) , without sharing their local data Dc . The problem can be formalized as minimizing the aggregated risk : R ( f ) = 1 C ∑C c=1 Rc ( f ) . ( 2 ) In practice , we typically employ a server to coordinate the iterative distributed training as follows : • in each global round of training , the server broadcasts its current model f to all the clients { fc } Cc=1 ; • each client c copies the current server model fc = f , performs L local step updates : fc ← fc − αl · ∇R̂c ( fc ; Dc ) , ( 3 ) where αl is the local step-size , and sends fc − f back to the server ; • the server aggregates the updates from all clients { fc − f } Cc=1 to form the new server model using FedAvg : f ← f − αg · ∑C c=1 ( fc − f ) , ( 4 ) where αg is the global step-size . 3 FEDERATED LEARNING : NO LABEL NO CRY . In this section , we formulate the novel problem setting of FL with only unlabeled datasets , propose our method called federation of unsupervised learning ( FedUL ) , and provide theoretical analysis of FedUL . All the proofs are given in Appendix C . 3.1 PROBLEM FORMULATION . In this paper , we consider a challenging setting : instead of fully labeled training data , each client c ∈ [ C ] observes Mc ( Mc ≥ K , ∀c ) 1 sets of unlabeled samples , Uc = { Umc } Mc m=1 , where Umc = { xc , mi } nc , m i=1 denotes the collection of the m-th U set for client c with sample size nc , m . Each U set can be seen as a set of data points drawn from a mixture of the original class-conditional densities : Umc ∼ pmc ( x ) = ∑K k=1 πm , kc p ( x | y = k ) , ( 5 ) 1Note that we do not require each client has the same number of U sets . We only assume the condition Mc ≥ K , ∀c to ensure the full column rank matrix Πc , which is essential for FL with U datasets . where πm , kc = p m c ( y = k ) denotes the k-th class prior of the m-th U set . These class priors form a full column rank matrix Πc ∈ RMc×K with the constraint that each row sums up to 1 . Note that we do not require any labels to be known , only assume the class priors within each of the involved data sets are available , i.e. , the training class priors πm , kc and the test class prior π k = p ( y = k ) . Our goal is the same as standard FL , i.e. , the C clients are interested in collaboratively training a global classification model f that generalizes well with respect to p ( x , y ) , but using only unlabeled datasets Uc on each local client .

Summary: This work presents a novel federated learning scheme to address the problem of learning from only unlabeled data. The main idea is clean and interesting, which constructs a global (server) model by aggregating the surrogate clients’ tasks from observing only unlabeled data for the classification tasks. The unlabeled data are transformed at each client to make them compatible with supervised federated learning; consequently, the learned models are transformed accordingly. Existing federated aggregation techniques, e.g. FedAvg, are applied on these transformed clients (where they call surrogate clients). Theoretical results are provided for learning the optimal model and experiments on benchmark and real data show superior performance compared to baselines.