Adversarial Collaborative Learning on Non-IID Features
Authors: Qinbin Li
Bingsheng He
Dawn Song
Venue: ICLR 2022 Submitted (2022)
Keywords: Federated Learning
Collaborative Learning

Abstract: Federated learning has been a popular approach to enable collaborative learning on multiple parties without exchanging raw data. However, the model performance of federated learning may degrade a lot due to non-IID data. While most existing studies focus on non-IID labels, federated learning on non-IID features has largely been overlooked. Different from typical federated learning approaches, the paper proposes a new learning concept called ADCOL (Adversarial Collaborative Learning) for non-IID features. Instead of adopting the widely used model-averaging scheme, ADCOL conducts training in an adversarial way: the server aims to train a discriminator to distinguish the representations of the parties, while the parties aim to generate a common representation distribution. Our experiments on three real-world datasets show that ADCOL achieves better accuracy and is much more communication-efficient than state-of-the-art federated learning algorithms on non-IID features. More importantly, ADCOL points out a promising research direction for collaborative learning.

Under review as a conference paper at ICLR 2022
ADVERSARIAL COLLABORATIVE LEARNING ON NON- IID FEATURES
Anonymous authors Paper under double-bling review
ABSTRACT
Federated learning has been a popular approach enable collaborative learning on multiple parties without exchanging raw data. However, the model performance federated learning may degrade a lot due to non-IID data. While most existing stud- ies focus on non-lID labels, federated learning on non-llD features has largely been overlooked. Different from typical federated learning approaches, the paper pro- poses new learning concept called ADCOL (Adversarial Collaborative Learning) for non-IID features. Instead of adopting the widely used model-an -averggig scheme, ADCOL conducts training in an adversarial way: the server aims to train dis- criminator to distinguish the representations of the parties, while the parties aim generate common representation distribution. Our experiments world datasets show that ADCOL achieves better accuracy and is much more communication-effccientt than state-of-the-art federated learning algorithms IID features. More importantly, ADCOL points out a promising research direction for collaborative learning.
three real-
INTRODUCTION
Deep learning data hungry. While data are always dispersed in multiple parties (e.g., mobile devices, hospitals) in reality, data are not allowed to transfer to central server for training due to privacy concerns and data regulations. Collaborative learning among multiple parties without the exchange of raw data has been an important machine learning topic.
Federated learning (McMahan et al. 2016 Kairouz et al. 2019 Li et al. 2019b(a) has been a popular form of collaborative learning without exchanging raw data. A basic federated learning framework is FedAvg (McMahan et al. 2016). which uses a model-averaging scheme. In each round, the parties update their local models and send them to the server. The server averages all local models to update the global model, which is sent back to the parties as the new local model in the next round. FedAvg has been widely used due its effectiveness and simpleness. Most existing federated learning approaches are designed based on FedAvg.
However, as shown in many existing studies (Hsu et al. 2019 Li et al. 2020 Wang al. 2020b Karimireddy et al.] 2020 Acar et al., 2021 Li et af. 2021b). b). the performance of FedAvg and its alike algorithms may be significantly degraded in non-lID data among parties. While many studies try to improve FedAvg on non-IlD data, most of them (Li et al., 2020; Wang et al. 20200] Karimireddy al. 2020; Acar et al., 2021 Li al. 202 Wang et al.) (020a) focus on the label imbalance setting, where the parties have different label distributions. In their experiments, they usually simulate the federated setting by unbalanced partitioning the dataset into multiple subsets according to the
labels.
As summarized in (Hsieh et al. 2020 Kairouz et al. 2019). besides the label distribution skew, feature imbalance is also an important case of non-lll) data. In the feature imbalance setting, the feature distribution (x( varies across parties. This setting widely exists reality, e.g., people have different stroke width and slant when writing the same word. Another example in practice is that medical images different hospitals have different intensity and contrast. However, compared with non-IID labels, federated learning non-IID features has been less explored. Since the feature distributions are different across parties, there exists bias in the local training. It is challenging to mitigate the bias while exploiting the heterogeneous feature distributions in the training.

---

Under review as a conference paper at ICLR 2022
In this paper, we think out of the model-averaging scheme used in federated learning, and propose a novel learning concept called adversarial collaborative learning. While the feature distribution of each party is different, to mitigate the local bias, we aim to learn common representation distribution among parties. Specifically, the server aims to train a discriminator to distinguish the local representations by the party IDs, while the parties train the local models such that the generated representations cannot be distinguished by the discriminator. Our experiments show that ADCOL outperforms state-of-the-ar federated learning algorithms (Li et al. 20216 2020 Fallah et al. 2020 Collins et al. 2021) on three real-world tasks. More importantly. ADCOL points out promising research direction on collaborative learning. Inspired by our idea of conducting collaborative learning in an adversarial way, we believe there will be more interesting future works in this direction.
2 BACKGROUND AND RELATED WORK
2.1 NON-IID DATA
We use P. y) to denote the data distribution of party i, where x is the features and is the label. According to existing studies (Kairouz et al., 2019; Hsich al., 2020), we can categorize non-IID data in federated learning into the following four classes: (1) non-lll) labels: the marginal distribution (y)) varies across parties. (2) non-IID features: the marginal distribution P. (x(x) varies across parties. (3) concept drift: The conditional distributions P.(y/ or P. (x(xx)) varies across parties. (4) quantity skew: the amount of data varies across parties. In this paper, we focus on non-IID features, which widely exist in reality. For example, while the pneumonia distribution of patients may be close among different hospitals, the distributions of CT images (such as resolution, color, length and width) may vary across hospitals, which are collected from different equipment and environments.
2.2 FEDERATED LEARNING ON NON-IID LABELS
Non-III data is key challenge in federated learning. There have been many studies trying to improve the performance of federated learning under non-lll) data. However, most existing studies (Li et al.,
2020:
Wang et al., [2020a) Hsu et al., 2019; Li et al. 2021a Acar al. 2021] Karimireddy et al.
2020 focus on label imbalance, where they usually simulate the federated setting with heterogeneous label distributions in the experiments. For example, FedProx (Li et al., 2020) introduces a proximal term in the objective of local training, which limits the update of the local model by the distance between the local model and the global model. FedMA (Wang et al. 2020a) applies Bayesian non-parametric methods to update the global model in a layer-wise manner in the model aggregation phase. FedDyn (Acar et al., 202T) introduces a dynamic regularization term the local training based the global model and the local model from the previous round. More recently, MOON (Li et al. 202 proposes monel-contrastive federated learning. which corrects the local training by contrastive loss which maximizes the agreement between the representation learned by the global model and the representation learned by the local model. As we will show in the experiments, existing algorithms have severe performance degradation on parties with non-IID features. More recently, a contemporary study (Zhang et al.] (22 proposed FedUFO, where each party trains a discriminator to apply feature and objective consistency constrains address the non-IlD data issue. However, FedUFO needs to transfer each local model to all the other parties, which causes massive communication overhead. Moreover, FedUFO is still based on FedAvg, while we propose a novel new collaborative training scheme based on adversarial training in this paper.
2.3 FEDERATED LEARNING ON NON-IID FEATURES
To the best of our knowledge, there is only one study named FedBN (Li et al. (021b) that focuses on non-IID features. FedBN introduces lightweight modifications on FedAvg. Instead of transferring and averaging all model parameters in FedAvg, FedBN keeps all the local batch normalization parameters and does not synchronize them with the global model. The operations for non- non-batch normalization parameters are the same as FedAvg. Our proposed ADCOL is based on a novel approach: adversarial learning for a common representation among multiple parties. ADCOL outperforms FedBN on three real-world datasets in both accuracy and communication--eficiinnn.
2

---

Under review as a conference paper at ICLR 2022
2.4 PERSONALIZED FEDERATED LEARNING
There are basically two kinds of objective in federated learning: learning a single global model or learning personalized local model for each party (i.e., personalized federated learning (Fallah
2020; Dinh et al. 2020 Hanzely et al. 2020 Zhang et al. 2021b; Huang et al. [202]] Collins ett al. al., 2021)). ). When the training data is non- -ID across parties, personalized federated learning is
reasonable since single global model may not be good for every local distribution. In this paper,
same as FedBN, ADCOL is designed under the personalized federated learning setting, i.e., each
party aims to learn a local model for its own local task.
3 METHOD
3.1 PROBLEM STATEMENT
Suppose there are N parties, where party i has local dataset D {xx, y² The feature distributions P(x') are different among parties while the label distributions P(y") are similar among parties. The parties conduct collaborative learning over D USE[N]] D' with the help of a central server without exchanging the raw data. The goal of each party is to train machine learning model which has good accuracy on its local test dataset.
3.2 MOTIVATION
The key idea of ADCOL is intuitive. While the feature distributions are different among parties, to mitigate the local bias, ADCOL aims to learn a common task specifpe representation distribution. Inspired by GANs (Goodfellow et al. (2014) and adversarial learning for distribution matching (Tznng ct al. 2017 Peng et al. 2019b), ADCOL adopts an adversarial way to restrict the representation distribution.
Next we explain the motivation from distribution perspective. The ultimate goal each party in typical federated learning is to learn the oracle optimal distribution (y/xx) Here we introduce the representation z of the data x, which is supposed to be sufficient for the prediction of labels (Federici et al., (020). Then, x, y. and forms a Markov chain
y. Thus, decomposite the oracle optimal into two parts: p" (z|x) and p" (ylz). The first part is to learn a good representation and the second part is to predict the label by the representation with simple linear layers. To learn the optimal representation distribution p" (z|x), the objective of each party is
min Ex- Ex (KL(((/)(x()))) 0,) || pz((x)),
(I)
0,
where L z is the KL divergence loss. We adopt KL divergence due to its popularity for evaluating the distance between probability distributions. However, the optimal representation distribution is unknown. Thus, we approximate the objective by two aspects: 1) By training in novel adversarial way, we use the similarity between the representations of different parties to regularize the representa- tion, i.e., the parties learn common representation 2) We ensure that the generated representation contains necessary information for the prediction of y by training on the labeled local data. Next, we describe our model architecture and training algorithm in details.
3.3 MODEL ARCHITECTURE
There are two kinds of models in ADCOL: the local models trained in the parties and the discriminator trained in the server. As ADCOL works from the perspective of representation, the architecture of the local model is similar as existing studies (Chen et al., 2020: He et al.] 2020) Chen and Hc (02T) on self-supervised representation learning. The local model has three components: a base encoder, projection head, and predictor. The base encoder (e.g., ResNet-50) extracts representation vectors from inputs. Like SimCLR (Chen et al. 2020) and SimSam (Chen and He 2021), an additional projection head is introduced to map the representation to space with fixed dimension. The final predictor is used to output predicted probabilities for each class. For ease of presentation, we use F(-) to denote the whole model and G(-) to denote the model before the predictor (i.e., G(x) is the mapped representation vector of input x). For the discriminator, we simply use a multi-layer perceptron in experiments.
3

---

Under review as a conference paper at ICLR 2022
Base encoder Projection head Prodictor
server
discriminator
discriminator
Representations
outputs targets
(a) Step 1: The server sends the discriminator to the parties.
(b) Step 2: The parties update their local models
Representations
Representations
Representations
discriminator
Representations
Representations
oulputs
targets
(c) Step 3:
The parties send representations to the server.
(d) Step The server updates the discriminator.
Figure 1: The ADCOL framework
3.4
THE OVERALL FRAMEWORK
and Algorithm 1 There are four steps in each round: (1) The overall framework is shown in Figure 1 and
The server sends the discriminator to the parties. (2) The parties update their local models. (3) The parties send representations to the server. (4) The server updates the discriminator.
Step 1
In the first step, the server sends the discriminator to parties (line 4 of Algorithm I.
In the second step. the parties update their models using their local datasets (lines 10-17 of Algorithm[[] addition the objective which aims minimize the cross-entropy loss (i.e., (CE) on the local dataset, ADCOL introduces additional regularization term which aims to maximize probability that the discriminator cannot distinguish the local representations. For each input x, ADCOL feeds the representation G(x) to the discriminator. ADCOL expects the discriminator to output probability vector [ N (i.e., the probability of each class N) such that cannot distinguish which party that the representation comes from. Thus, ADCOL uses Kullback-Leibler (KL) divergence loss to measure the difference between the output of the discriminator D(G(x)) and the target IN The final loss of input (x, y) is computed
Step 2
Lo (F)
plxea1
1 N D(G(x)))
where μ is hyper-parameter to control the weight of KL divergence loss, fcg is the cross-entropy loss. and ε is the KL divergence loss.
Step 3
After local training, the parties feed their data into the local models and transfer the representations to the server (line 5 of Algorithm T..
Step 4 The server updates the discriminator using the received representations (lines 6-9 of Algo- rithm [I] Specifically, the server build training set, where the feature values the representa- tions and the labels are the party IDs that the representations come from. The training set is used to update the discriminator using stochastic gradient descent.
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1:
The ADCOL algorithm
Input: number of communication rounds T, number of parties N, number of local epochs E, learning rate hyper-paramen
Output: The local models F; [N]
Server executes:
PartyyLoca//rrininnii D): for epoch e= 1,2, ..... do for each batch b = fx, y of D' do
for t 2,, for i = 1,2,.., N in parallel do send the discriminator D to party
12
lCE + Cross R G2(x)
13
R. x SonryyooaTTaainng(.. D)
14
R + ((R,,))
CKL <- (
for each batch b= iR of R do + Cross (ntropy D n
16
t foE + +ulKz
F F; -ive
17
return Gx(x') to server
3.5 COMPARISONS WITH EXISTING FEDERATED LEARNING APPROACHES
ADCOL is quite different from typical federated learning algorithms. Instead of transferring and aver- aging models, ADCOL transfers the representations from the parties to the server. More importantly, the training is conducted in adversarial way between the parties and the server. The server tries distinguish the representations while the parties try to generate non-distinguishabl representations. Here we compare ADCOL with FedAvg terms of communication size and privacy.
For simplicity, do not consider party sampling during training (i.e., all parties participate in learning in each round). We use S₁ to denote the size of the local model. Then, the communication size of FedAvg is 2n including the server sends the model all parties and the parties send their local models to the server. We use n to denote the total number of examples (i.e.,
Communication Size
Σn i= [D"), d to denote the dimension of the representations, and Sp to denote the size of the discriminator. Suppose each float value costs bytes to store. In each round, the total communication size of ADCOL is (4nd + NSp), including the parties send the representations to the server and the server sends the discriminator to the parties. Thus, when n (2S, Np)N/4d, the communication size of ADCOL is larger than FedAvg in each round.
Note that the above threshold is usually large in practice. For example, in our experiments, when N = 6, d 2048, Sr 142M B, and Sp 5MB, the threshold is equal to 204346. As will show in the experiments, ADCOL is much more communication-efiice ADCOL requires smaller bandwidth requirement as the size of each representation is much smaller than the local model.
than FedAvg. Moreover,
Privacy While FedAvg transfers models and ADCOL. transfers the representations, FedAvg and ADCOL both suffer from possible attacks the exchanged information. There are many existing studies (Shokri et al, 2017; Nasr et al.] 2019) that infer sensitive information from exchanged gradients/models. Also, there studies (Salem et al. 2020) on reconstruction attacks wn the output of a model. While existing studies have shown that the mutual information between the input data and the final representation is small (Shwartz-Ziv and Tishby 2017). it is not clear that which one is more private against inference attacks, which can be an interesting future direction. Local differential privacy (Kairouz al.) (2014) can be applied protect the representation, which has been adopted in previous studies (Lyu et al.) 2020). We have conducted experiments to show the feasibility of differential privacy to protect the communicated representations in Appendix B.13
THEORETICAL ANALYSIS
As shown in Equation 2). the local loss has two parts: the cross-entropy loss part to update the whole network F and the KL divergence loss part to update the representation generator G. Ideally, to achieve minimum of é, each part should achieve minimum. Since the cross-entropy loss part is same FedAvg, focus the effect of the KI divergence loss. For simplicity, ignore the
5

---

Under review as a conference paper at ICLR 2022
eross-entropy loss and study the KL divergence loss in our theoretical analysis The local objective of party is:
1 N
min BxND, B.LDDKKKll (K ((G(())))
(3)
The objective of the discriminator is
N max > Ex-D, log(Di
(4)
where Di(-) is the i-th output of the prediction vector (.)) (i.e., the probability of class i). Here we analyze the convergence property of the training process. All the proofs are available Appendix [
Theorem 1. We use Pc, denote the distribution of the representations generated in party and Pc, (z) is the probability of representation z in distribution Pc.. Then, the optimal discriminator D of Equation is
Po (z)
(2)
Ei (z)
Theorem 2. Given the optimal discriminator D from Equation 33 is achieved if and only if
(5) the global minimum of Equation
Pa
Po2
Pax
Theorem D and Theorem show that to achieve the minimum of the objectives of the local parties and the discriminator, the parties will generate the same representation distribution, which matches the goal of ADCOL.
Theorem 3. Suppose Pg is the optimal solution shown in Theorem If G. (Vi E [) and D have enough capacity, and Pa, is updated to minimize the local objective (i.c., Equation (3), given the optimal discriminator D from Equation (5) then Pg. converges to Pa-
The above theorem provides insights the convergence of the training. In practice, we optimize the parameter the local networks rather than Po, itself, which is reasonable due to the excellent performance as claimed in (Goodfellow et al. 2014).
Note that there are collapsing solutions for Equation (3) and Equation The representations of each party simply be constant vectors, which achieve global minimum of Equation (3) Thus, the crossyyntropy loss is necessary Equation (2). which ensures that the generated representations meaningful.
EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Baselines We compare ADCOL with six baselines including SOLO (i.e., each party trains the
model individually without collaborative learning), FedAvg (McMahan et al.) 2016), FedBN (Li al. 2021b), FedProx (Li al. 2020), Per-FedAvg (Fallah et al. 2020), and FedRep (Collins et al. 2021) Here FedBN is the ofate-of-the-art FL approach on non- IID features. FedProx is a popular Fl. approach for non-lllD data. Per-FedAvg and FedRep
state-of-the-art personalized FL approaches. Like FedAvg (McMahan et al. 2016). we use weighted average according to the data volume of each party for all baselines.
Models approaches use the same local model architecture for a fair comparison. The architecture of the local model is similar as SimSam (Chen and Hc [02I), which has the following three components: (1) Base encoder: ResNet-50 (He et al. 2016). (2) Projection head: 3-layer MLP with BN applied to each fully- fully-connected layer. The input dimension is 4096. The dimension of the hidden
Note that G is part of F and the two losses not independent of each other. For simplicity, only analyze the KI. divergence loss study its effect.

---

Under review as a conference paper at ICLR 2022
Table 1: The comparison of top-1 test accuracy among different approaches on Digits. We run FL approaches for 100 rounds (all approaches have converged). We run three trials and report the mean and standard derivation. Besides the test accuracy on each party, we also report the mean accuracy of all parties denoted as "AVG".
Digits
MNIST
SVHN
USPS
SynthDigit
MNIST M
AVG 69.5%
SOLO
074%
34.8%10.89
94.8%:10.1%%
63.0%=0.44
67.2%110.4%
FodAvg
94.4% +0.5%
+9.4%+0.9%
94.3%++0.25%
74.4% 0.5%
70.3% 1.2%
78.6%++0.65
FedBN
94.1%
59.9%
94.1%:10.1%%
73.9%
11.3%:11.1%
78.7%
FedProx
94.1% +0.4%
5..8%
94.3%++0..15 1%
73.4%
-3.4%-
71.6%++0.95
78.6%%+0.449
Per- Per-FedAAg
88.9% -075%
36.6%:1 11.6:111..5 .3%
89.5%:+0.2%
58.3%
-077%
54.5%.41.38 .33
65.6%.40..89 8%
FedRep
92.6%
42.0%%:11.0%
93.1%%t0..1%
61.1% 0.5%
50.8% 1.4%
67.9%::t0.8%
ADCOL
94.7%=0.66%
58.2%:11.000
40.2%
76.0%
0.33%
76.7%:110.8%
0..2%
layer and the output layer is 2048. (3) Predictor: a 2-layer MLP with BN applied to its hidden layer.
The input dimension is 2048. The dimension of its hidden layer is 512. The discriminator a 3-layer MLP. The input dimension is 2048. The dimension of the hidden layers is 512. The output dimension equal the number of parties.
Datasets We use the same datasets in the study of FedBN. There are three weal-world tasks in our experiment (1) Digits: The Digits task has the following five digit data sources from different
P
domains: MNIST (LeCun al. 1998), SVHN (Netzer et al. 2011), USPS (Hull 1994), SynthDigits
(Ganin and Lempitsky 2015), and MNIST-M (Ganin and Lempitsky 2015). (2) Office- Caltech- 10
(Gong al. 2012): The dataset has four data sources acquired using different camera devices or in different real environments with various backgrounds: Amazon, Caltech, DSLR, and WebCam. (3) DomainNer (Peng af. 201 The dataset contains natural images coming from six different data sources with different image styles: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch.
Setup By default, the number of parties is equal to the number of data sources, where each party has data from one of the data sources. For each dataset, we randomly split 1/5 of the original dataset as the test dataset, while the remained dataset is used as the training dataset. Same as (Li et al. 2021b), data of different parties have heterogeneous features but share the same labels and label distribution, which naturally forms the non-llD feature setting. The number of local epochs is set to 10 by default for all FL approaches. The number of epochs is set to 300 for SOLO. For ADCOL and FedProx, we tune E (10, 0.1, 0.01, 0.001} and report the best results. For FedRep, we tune B (i.e., step size for the second batch training) from 0.001, 0.01 and report the best results. We use the prediction layers as the shared representation in FedRep We use Py/orch v1.81 (Paszke et al. 2019) to implement all approaches. We use the SGD optimizer for training with a learning rate of 0.01. The SGD weight decay is set to 10-5 and the SGD momentum is set to 0.9. The batch size is set to 64, 32, and 32 for Digits, server with 8 NVIDIA GeForce RTX 3090 and a cluster with 45 * NVIDIA GeForce RTX 2080 Ti.
10, and DomainNet, respectively. We run the experiments on a
Due to the page limit, we only present some representative experimental results in the main paper. For more experimental results and details, please refer Appendix B
5.2 OVERALL COMPARISON
Table reports the test accuracy of different approaches on three tasks. We have the following observations. First, ADCOL is more effective than the other approaches. It can achieve the best test accuracy most datasets. Moreover, ADCOL can outperform the other approaches by more than 2% accuracy on average. Second, while the parties may not benefit from FL. approaches in some cases (e.g.. Caltech- 10), ADCOL always achieves better accuracy than SOLO, which demonstrates the robustness of ADCOL. Last, the personalized FL approaches (i.c., Per-FedAvg and FedRep) have poor performance on the non-IID feature setting, which are even worse than SOLO.
FedBNN uses four tasks in their experiments. We only use three tasks because one task is not publicly available in FedBN's GitHub website.
7

---

Under review as a conference paper at ICLR 2022
Table 2: The communication round and communication of each approach achieve the same target performance as the minimum converged accuracy among FedAvg, FedBN, FedProx, and ADCOL as shown in Table 1 (i.e., 94.1% MNIST). We use the slash cell indicate that the approach cannot reach the target performance in 100 rounds/30 GB. The speedup is computed by
dividing the communication size of FedAvg by the communication size ADCOL.
Digits
MNIST
SVHN
USPS
SynthDigit
MNIST _M
AVG
FedAvg
11
54
5
7
28
FedBN
11
73
68
7
22
FedProx
64
42
8
12
10
31
#round
Per-FedAvg
FedRep
ADCOL
19
9
21
FedAvg
3.12
15.34
1.42
3.12
1.99
7.95
FedBN
3.12
20.73
1.42
19.31
1.99
6,25
FedProx
18.18
11.93
2.27
3.41
2.84
8,80
cost (GB)
Per-FedAvg
FedRep
ADCOL
0.21
0.95
0.07
0.21
0.10
0.23
Speedup
14.95
16.21
21.52
14.95
20.08
34.42
5.3 COMMUNICATION EFFICIENCY
To show the communication efficiency of ADCOL, like existing studies (Karimireddy et al. 2020 Lin et al. 2020), compare the number of communication rounds and communication size each approach to achieve the same target performance. The results on Digits are shown in Table The results on the other two datasets are available in Appendix B.4] We can observe that no approach consistently outperforms the other approaches in terms of the number of communication rounds. However, the communication size of ADCOL is always much smaller than the other approaches. ADCOL can save at least 10 times the communication costs achieve the same accuracy as FedAvg. The speedup can even be up to x4x on Digits. The results demonstrate that ADCOL is much more communication -effeiiinnnnntn than the other FL approaches.
5.4 SENSITIVITY STUDIES
We study the effect of different factors on ADCOL and the other baselines using the Digit task.
Scalability and Heterogeneity We adopt the same approach as (Li et 2021b) to study the effect of number of parties and heterogeneity. We divide each dataset into ten parts randomly and equally and allocate each part into one party. The parties from the same dataset are treated as IID and the parties from different datasets are treated as non-IID. We add two parties from each dataset each time, which results the number parties E {10, 20, 30, 40, 50). Moreover, the degree of heterogeneity decreases as the number of parties increases since the number of IID parties increases. The test accuracies are reported in Figure 2a We can observe that the accuracy of all approaches can be slightly improved when increasing the number of parties due to the reduced heterogeneity and increased total amount of data. Given a different number of parties, ADCOL consistently outperforms the other baselines. Although the number of classes to distinguish increases for the discriminator when increasing N, ADCOL still shows a good and stable performance with a large number of parties.
Number of Local Epochs We vary the number of local epochs E E (1,2,5, 10, 20) and report the results Figure 2b We run all approaches for 100 rounds. the number of local epochs too small, the local update is small in each round and the convergence speed is slow. Thus, the accuracy of all approaches is relatively low after running for 100 rounds with a small number of local epochs. ADCOL still consistently outperforms the other approaches with different number of epochs.
Effect of Local Dataset Size We vary the percentage of the original local dataset used in each party from 20% to 100%. The results are shown in Figure 2c The improvement of ADCOL is more
8

---

Under review as a conference paper at ICLR 2022
90
90
Par-FedAvg
FedAvg
Per-FedAvg
FedAvg
FedRep
FedBN
FedRep
85
FedBN
ADCOL
85
FedProx
ADCOL
FedProx
test accuracy (%)
accuracy
75
70
70
65
65
20
40
10
number of parties
number of epochs
(a) Scalability
(b) Number of local epochs
100
100
u=0
p-1
v=0..1
1-110
80
test test accuracy (%)
80
test accuracy
60
40
SOLO
Per-FedAvg
50
20
FodAvg
FodRop
FedBN
ADCOL
FedProx
60
100
MNIST
SVHN
USPS
SynthDigitMNISTAN
percentage (%)
(c) Dataset size
(d) Hyperparameter
Figure 2: Effect of different factors. We run three trials and report the mean accuracy across parties and its standard derivation.
significant when the size of the local dataset small. If the size of the local dataset is large, each party can already achieve satisfactory accuracy by SOLO. The accuracy of all approaches is close when the percentage 100%. It is not necessary to conduct collaborative learning in such a case.
Effect of We vary {0, 0, 10} and report the accuracy of ADCOL shown in Figure
2d
We can observe that ADCOL can achieve the best accuracy when μs 1. If p is too small, the KT. divergence loss of Equation (2) has little effect the local training. Then, the goal of learning a common representation distribution may not achieve. If µ is too large, the cross-entropy loss of Equation (2) has little effect the local training, and the representations may be useful for classification at all (e.g., all representations are a constant vector). ADCOL with 10 may even be worse than SOLO (i.e., μ 0). Thus, an appropriate y important ADCOL. Through our experimental studies, we find that setting μ = 1 is good default choice.
6 CONCLUSION
In this paper, we propose ADCOL, a novel collaborative learning approach for non-lID features. ADCOL trains the models an adversarial way between the parties and the server from a perspective of representation distributions. The parties aim to learn common representation distribution, while the server aims to distinguish the representations by party IDs. Our experiments on three real-world tasks show that ADCOL achieves higher accuracy than the other state-of-the-art federated learning approaches on non-IID features. ADCOL provides new insights collaborative learning.
Ethics Statement Not applicable.

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement We have provided the experimental details in Section 5.1 and Section B.I reproducibility. Moreover, we will make the code publicly available.
REFERENCES
Durmus Alp Emre Acar, Yue Zhao. Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regularization. In International Conference on Learning Representations, 2021. URL
forum?
id Ad-BBvvQMR623
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint urXiv:200..0700, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In International Conference on Machine Learning. PMLR, 2021.
Canh T Dinh, Nguyen H Tran, and Tuan Dung Nguyen. Personalized federated learning with moreau envelopes. arXiv preprint arXiv:2006.0882 2020.
Cynthia Dwork. Differential privacy. Encyclopedia of Cryptography and Security, pages 338-340, 2011.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33, 2020.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. International Conference Learning Representations, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-2033 2016.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference computer vision and pattern recognition, pages 2066-2073. IEEE, 2012.
Ian J Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661 2014.
Filip Hanzely, Slavomír Hanzely, Samuel Horváth, and Peter Richtárik. Lower bounds and optimal algorithms for personalized federated learning. arXiv preprint arXiv:2010.02372. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IFFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pages 4387-4398. PMLR, 2020.
10

---

Under review as a conference paper at ICLR 2022
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non- con-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.063333 2019.
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on paltern analysis and machine intelligence, 16(5):550-554, 1994.
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. Extremal mechanisms for local differential privacy. Advances in neural information processing systems, 27:2879-2887, 2014.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 2019.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for on-device federated learning. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-23 23222224,, 1998.
Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection. arXiv preprint arXiv:1907.09693, 2019a.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrasivee federated learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021a.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. arXiv preprint 1r0iv:/908.0700 2019b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Third Conference on Machine Learning and Systems (MLSys), 2020.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fed(bn): Federated learning on non-(iid) features via local batch normalization. In International Conference on Learning Representations, 2021b. URL /ttpps://peerrevier wwt.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances Neural Information Processing Systems, 33, 2020.
Lingjuan Lyu, Xuanli He, and Yitong Li. Differentially private representation for nlp: Formal guarantee and an empirical study on privacy and fairness, EMNLP. 2020.
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. coamuumication-effcceene learning of deep networks from decentralized data. arXiv preprint arXiv: 1eXXvv::000000 05629, 2016.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2079 IEEE Symposium Security and Privacy (SP), pages 739-753. IEEE, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, reeh-performncc deep learning library. Advances in neural information processing systems, pages 8026-8037, 2019.
11

---

Under review as a conference paper at ICLR 2022
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 1406-1415, 2019a.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation. arXiv preprinl arXiv: wrXiv:/911.022 1911.0005, 2019b.
Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, and Yang Zhang. Updates-leak: Data set inference and reconstruction attacks in online learning. In 29th {USENIX) Security Symposium ( (USENIX) Security 20), pages 1291-1308, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18. IEEE, 2017.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 100000000,,,000, 2017.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176, 2017.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Fed- erated learning with matched averaging. In International Conference on Learning Representations, 2020a. URL http:///ppeeeeee.eeeeeeee
/ net/ forumorrmmoooo did. uug 1 SFDS
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor.
Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems, 33, 2020b.
Lin Zhang, Yong Luo, Yan Bai, Bo Du, and Ling- Yu Duan. Federated learning for non-iid data unified feature learning and optimization objective alignment. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 4420 4400, 2021a.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized federated learning with first order model optimization. In International Conference on Learning Representations, 2021b. URL httttpss///peen.eeee...ttt http:///oooee...ehe0100000000000
12

---

Under review as a conference paper at ICLR 2022
A THEORETICAL ANALYSIS
Theorem 1. We use Pc. to denote the distribution the representations generated in party and Pc. (z) is the probability of representation z in distribution Pa.. Then, the optimal discriminator D of Equation
Pa (z)
(1)
Σ (z.
Proof. From the view of the distribution of representations z, we can reformulate Equation 4 the objective to maximize:
and
Σ
Pc. (z)
(8)
Let V(D) En (z) log(D:(z)).
Note that Σ -i=1 Di(z) 1. Let F(D) V(D) X(1
ΣN (z)). We have
OF(D) Pc.
(9)
(D( D2(z)
Let (F(D) for [1, we have (D,(z)
Pa. (z)
(z)
Pan (z) Dx
=
(10)
(z D2(z)
Thus, V(D) can achieve maximum when
Pa (z) Σ (z)
(z(z)
(11)
Theorem Given the optimal discriminator D* from Equation 77. the global minimum of Equation 3 is achieved if and only if
Po = Pc2 PaN
(12)
Proof. From Equation
3). the local objective of party l is minimize
N log(N- (N- D+(Gx(x)))
w(G+) EX
(13)
N
N (Gx
-x N
log(
Σj=1 Pa, (x))
Note that ΣN N (Gx())
Po, (Gx(x))
1. Similar to the proof in Theorem
Equation 13 can achieve
minimum when
Pai (Gx(x))
(Gx(()
Pa, (Gx(x))
1 (14)
EN Pa, (Gx(x)
E Pa, (Gx(x))
Σ Pa, (Gx())
N
NN Pa, (Gx(x)
For Vk N] and Vi E N], we have Pa, (x)()))
Given representation
Z, we have
En Pc, (z) N
Po, (z)
-...= Por :...
(15)
Thus, Po, Pa,
Pox-
n
13

---

Under review as a conference paper at ICLR 2022
Table 3: The statistics of all studied datasets.
#training samples
#testing samples
MNIST
56,000
14,000
SVHN
79,431
19,858
Digits
USPS
7,438
1,860
SynthDigit
402,209
97,791
MNIST M
56,000
14,000
Amazon
766
192
Caltech
898
225
Caltech-10
DSLR
I25
32
WebCam
236
59
Clipart
2,103
526
Infograph
2.626
657
Painting
2,472
619
DomainNet
Quickdraw
4.000
1,000
Real
4,864
1,217
Sketch
2,213
554
Theorem 3. Suppose Pg is the optimal solution shown in Theorem 2 If G. (Vi E N), and D have enough capacity, and Pa, is updated to minimize local objective (i.e., Equation (3), given the optimal discriminator D* from Equation 7 then Pc, converges to Pa.
Proof. In Equation
13 consider W(G U(P..) function of Pa.- Then
(U.)).)
Exps Pc.
(16)
0Pa.
NPo, Exdi Poe)
We have
(U.)
20
(17)
aP.. 2
Pa. (Pa. + E Po,)
Thus, U(P..) ) is convex in Pa.. Therefore, with sufficiently small updates of Pa., Pc. converges to Pa. concluding the proof.
B ADDITIONAL EXPERIMENTAL RESULTS
B. ADDITIONAL EXPERIMENTAL DETAILS
In each experiment, like FedBN (Li et al. 2021b), to remove the effect of quantity skew, we truncate the size of all datasets to their smallest number with random sampling. For Digits, we resize all images to 28 x 28 3 and normalize them with mean 0.5 and standard derivation 0.5 for each channel. For Office-Caltech-II 10, we resize all images to 64 x 64 x with random horizontal flip and random rotation. For DomainNet, we resize all images to 64 64 x 3 with random horizontal flip and random rotation. Like FedBN, we take Digits as the benchmark task for most studies.
The statistics of all the datasets shown in Table The label distributions are shown in Figure The portion samples with each class close to 0.1. The label distribution is balanced among all datasets (i.e., parties) from task and extract the feature distributions of each dataset. Then, we use t-SNE to visualize the representation as shown in Figure
the parties. We train ResNet-50
M
We can observe that the feature distribution of each party is different.
B.2 CALTECH- 10 AND DOMAINNET
Table and show the test accuracy different approaches on Caltech- and DomainNet, respec- tively. We observe that ADCOL still outperforms the other approaches in most cases.
14

---

Under review as a conference paper at ICLR 2022
-011 323 010
630
130 130
+-022 630
931
0.00 830 631 012 6.5 310 333
0.30
1 1353
1004
DE 0. 114 511 011 611 414
3359
101 014 0: 511 611 011 533 020
121 029 010
9-033
020
631
-0005-009 012 645 339 609 612 911 0.0 8.0 822
-827
-107
K.-333 610 938 618 010 613 6/6
0333
333 211 009 921 0L1 0.9 630 312 609 646
1993
Digits
(b) Caltech-10
(c) DomainNet
Figure 3: The label distributions of each task. The value in each cell of row and column j represents the percentage of samples with class j in Party
Party
Patty
(a) Digits
(b) Caltech-10
(c) DomainNct
Figure 4: The feature distributions of each task.
B.3 TRAINING CURVES
The training curves of different approaches on Digit shown in Figure 5 We can observe that ADCOL is much more communication-effiice than the other approaches. ADCOL can convergence with much smaller communication size than the other approaches.
B.4 COMMUNICATION EFFICIENCY
Like Table 2 we show the communication efficiency of ADCOL on Caltech- 10 and DomainNet in Table and T We can observe that ADCOL much more communication--fficcee than the other approaches. The speedup can be even up 300 times.
B.5 PARTY SAMPLING
Party sampling is technique usually used in the cross-device setting, where a fraction of parties is sampled participate federated learning in each round. Here we set the sample fraction to 0.4 in Digit and choose FedAvg and FedBN as the baselines. The training curves are shown in Figure We can observe that all approaches have an unstable accuracy during training due to sampling. Moreover, FedAvg and FedBN have a very poor accuracy, which shows that existing federated learning approaches cannot well support party sampling on non-III features. ADCOL significantly outperforms the other approaches.
increase the number parties to 100 (i.e., divide each dataset to 20 subsets) and vary the sampling rate from {0.1, 0.2, 0.5, 1]. We run all approaches for 200 rounds. The results are shown in Table X We can observe that when the sampling rate decreases, the performance all approaches decreases. Moreover, the training is more unstable if the sampling rate is smaller. However, ADCOL still significantly outperforms FedAvg and FedBN. It is still challenging task to develop effective algorithm the cross-device setting with low sampling rate.
15

---

Under review as a conference paper at ICLR 2022
Table 4: The comparison of top-1 test accuracy among different approaches Caltech-10
Caltech-10
Amazon
Caltech
DSLR
WebCam
AVG
SOLO
50.8%
36.9%
10.5%
74.6%
58.8%:10.6%
FedAvg
24.0%
36.9%
81.3%10.44%
82.7%
56.2%.11.0%%
FedBN
33.3% 1.5%
33.8%=188%
81.3%=0.55%
80.6%
57.1%=1.0%
FedProx
43.2% 1.6%
33.1%10.8%
82.6%=10.49
83.1%10.7%
60.5%11.7%%
Per-FedAvg
33.9% + 1.6%
32.4%
1.5%
62.5%
+0.8%
74.6%
+.88%
50.8%+1..3
FedRep
16.1%= 1.9%
22.7%
1.7%
56.3% 1.0%
57.6%
1.1%
38.2%++1.49
ADCOL
14.2%
38.2%
1.3%
10.6%
83.1 11% 10.5%
62.6% 10.9%
Table 5:
The comparison of top-1
1 test accuracy among different approaches DomainNet.
NomainNet
Clipart
Intogruph
Puinting
Quickdraw
Real
Sketch
AVG
SOLO
01.7%
20.28:
30.9%%
18.2%
0..5%
20.8% 5.55%
31 40.9%%
FedAvg
33.3%112 2%
20.7%
20.2%
56,2%;1 2%
40.3%200.8%
22.6%%
33.7%
FedBN
13.
0..8%
27.8%
61.9%
41.9% 1.3%
23.6%%
1.2%
1.1%%
FedProx
37.8%=108
20.8%
28.1%6:11.0%
20.0%
43.6%
22.4% 1: 0%
29.5%
10%
Per-IedAwg
18.2%
10.0%
27%
10.4%
40.3%
22.0%%
25
31.8%
1.3%
FedRep
27.9%=1225 1.2%
19%%%
24. 11.1%%
17.9%
31.1%
16.8%
22.8%
1.4%
ADCOL
39.9%=1.00%
20.9%
33.9%:11.25
61.7%::10.8%
39.3%
23.9%
36.8%:11..0
B.6 STUDY ON THE DISCRIMINATOR
One natural question how to increase the information contained the discriminator to improve the performance of ADCOL. We have tried two approaches.
Changing Model Architecture One approach to increase the capacity of the discriminator. We change the model architecture ResNet-50. The results are shown Table The performance of ADCOL cannot be improved by increasing the capacity of the discriminator.
Increasing the Number of Discriminators The other one approach increase the number of discriminators. Suppose the number of discriminators is Na and the current round is t. Then, we use discriminators from round max(1, loss is computed as
Na) to round (t - 1) the local training. The KL divergence
N₂ K
1 N. {
Σ
(G(()),
(18)
N₂
where D' is the discriminator trained in round max(1, - i). The results are shown in Table 10 ADCOL cannot benefit from more discriminators. When the number of discriminators is larger, the accuracy of ADCOL even worse. It is future work to investigate how integrate more useful information into the discriminator.
B.7 DIMENSION OF REPRESENTATIONS
Same as SimSam (Chen and He 202I), we set the dimension of representations (i.c., the output dimension of the projection head, the input dimension of the discriminator) 2048 by default. As shown in Table T1 we report the performance of ADCOL varying the representation dimension. ADCOL can benefit from larger representation dimension, where the representations more informative. The mean accuracy can be improved by about 5% by increasing the dimension from 512 2048.
B.8 NON-IID LABELS
We test the performance of ADCOL on non-lll) label settings. Specifically, we sample ~ Dir v (0.5) and allocate a Pa.j proportion of the instances class k to party j. where Dir(0.5) is the Dirichlet distribution with a concentration parameter 0.5. The results are shown in Table 12 ADCOL cannot achieve better performance than FedAvg. Intuitively, the task-specific representations of images from different classes should be very different. If the label distribution varies across parties, the representation distribution naturally also varies lot. The intuition of ADCOL, which aims to learn common representation distribution, appropriate non-III) label settings.
16

---

Under review as a conference paper at ICLR 2022
60
top-1 accuracy (%)
top-1 accuracy (%)
50
80
70
30
60
50
Fedtrox
20
AUCOL
FodAvg
Fodlug
FodAvg
Per FodAuq
FedttN
Feditep
PedtN
reditep:
40
Communication size (GB)
Communication size (GB)
(a) MNIST
(b) SVHN
98
96
top-1 accuracy (%)
accuracy 1%)
70
94
92
50
(00-1
30
FedProx
20
ADCDL
TedPrax
82
FedAug FedBN
Pe--Pedog
Ped/wg
FedRep
PedBN
FedRep
80
Communication size (GB)
Communication size (GB)
(c) USPS
(d) SynthDigit
top-1 accuracy (%)
top-1 accuracy (%)
70
60
50
50
40
30
20
ADCOL
FedProx
20
ADCOL
FesProx
Fedligg
Pen Fedling
Fedfigg
Per FedAung
FodBN
FodRep
FodBN
FedRep
Communication size (GB)
Communication size (GB)
(e) MNIST-M
(i) AVG
Figure The training curves of different approaches on Digit.
B.9 IMBALANCED LOCAL DATA SIZES
In the main paper, we truncate the local data to the same size to control non-related factors. Here we show experiments with imbalanced number of samples across parties. We use 10% original data size for all parties without truncating the data. The results are shown Table [3] ADCOL is still the best approach in most cases.
17

---

Under review as a conference paper at ICLR 2022
Table 6: The communication round and communication of each approach to achieve the same larget performance on Caltech-10. 10.
Caltech- 10
Amazon
Caltech
DSLR
WebCam
AVG
FedAvg
7
22
12
14
14
FedBN
21
12
19
19
FedProx
14
29
13
29
22
#round
Per-FedAAg
23
FedRep
ADCOL
3
18
31
12
20
FedAvg
1.99
6.25
3.41
3.98
3.98
FedBN
0.85
5,96
3,41
5.40
5,40
FedProx
3.98
8.24
4.26
8.24
6.25
cost (GB)
Per-FedAvg
6.53
FedRep
ADCOL
0.02
0.10
0.17
0.07
0.11
Speedup
120.48
63.11
19.99
60.24
36.15
Table 7: The communication round and communication cost of each approach to achieve the same target performance on DomainNet.
DomainNel
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
AVG
FedAvg
22
38
23
47
FedBN
14
7
5
54
29
18
41
FedProx
13
8
16
32
12
fround
Per-FedAvg
31
19
34
64
FedRep
ADCOL
11
6
44
26
FedAvg
6.25
1.42
2.56
10.79
6.53
4.83
13.35
FedBN
3.98
1.99
1.42
15.34
8.24
5.11
11.64
FedProx
3.69
2.27
4.54
9.09
3.41
cost (GB)
Per-FedAvg
8.80
5.40
9.66
18.18
16.19
FedRep
ADCOL
0.06
0.02
0.05
0.04
0.26
0.11
0.15
Speedup
96.9
60.6
48.5
306.9
25.3
43.4
87.6
B.10 COMPUTATION OVERHEAD
As shown Table 14 the training time of ADCOL is larger than the other approaches. ADCOL requires the training of discriminator in the server side, while the other approaches only need to average the models in the server side. However, in practice the server usually has much power- ful computation resources than the parties. Thus, the computation overhead in the server side is affordable.
B.11 SHARING THE PREDICTOR LAYERS
The parties only send the representations to the server in ADCOL. While ADCOL aims learn common representation distribution z, interesting extension is to share the predictor layers between
Table 8: The performance of different approaches varying the sampling rate. We run all approaches for 200 rounds and report the final mean accuracy and standard deviation with three runs. Sampling Rate
FedAvg
FedBN
ADCOL
0.1
12.1% 1.9.9999990
1.3%%.8..19
34.1%
6.5 %
0.2
15.3% 1..6%
31.9%
46.7% % 4.2% %
0.5
60.2%
6..2%
67.3% -2.4%
77.6%+1.49%
1..7%
79.1 % 1099%
18

---

Under review as a conference paper at ICLR 2022
accuracy (%)
80
(0p-7 accuracy (%)
80
70
70
60
60
50
40
top
30
20
ADCOL
FedBh
ADCOL
Fedah
Tedkurg
Teddug
10
175
#Communication rounds
Communication size (GB)
(a)
(b)
Figure 6: The training curves with party sampling (sample fraction = 0.4). We report the mean test accuracy across all parties.
Table 9: ADCOL with different discriminator architectures. USPS
Discriminator MLP
MNIST
SVHN
SinihDigii
MNIST-M
95.1% 4 0.5%
55.6% + 0.8%
96.0%
0.3%
73.6%
0.5%
76.3% 0.5%
79.4% + 0.4%
ResNet-50
94.7%
0.6%
58.2% + 1.0%
95.4%
0.2%
76.0%
0.3%
76.7% 0.8%
80.2% + 0.5%
parties and the server, which ideally helps in regularizing p(y|z). The results are shown in Table We can observe that ADCOL without sharing the predictor layers is generally more effective than sharing the predictor layers. In practice, the distribution p(y[x;) is not exactly the same across parties. Thus, it is not necessary regularize p(ylz) among the parties. Leaving the parties fine tune their own predictor layer more capable to learn the personalized local distribution.
15
B. REPRESENTATION SAMPLING
An approach reduce the communication cost sample a subset of representations and send them to the server. Here we change the sampling rate from 220%, 60%, 80%, 100%). The final accuracy and the communication efficiency are shown in Table We can observe that the communication cost of ADCOL can be significantly reduced with representation sampling technique. Moreover, there is little accuracy loss when the sampling rate is large than 60%.
B. DIFFERENTIAL PRIVACY
We conduct experiments apply differential privacy (Dwork] 2011) to protect the communicated representations. Specifically, we normalize the representations and add noises from Lap(0, 1/e) before sending them the server, where Lap(0, 1/e) is the Laplace distribution with mean and scale Then, in each round, the transferred representations satisfy c c-differential privacy (Lyu et al. 2020p. We run 50 rounds for all approaches. For each round, we sample 1 of the representations of each party without replacement so that the privacy cost of each round is not accumulated. The results are shown Table 17 We can observe that the accuracy of ADCOL is very close to the non-private version with modest privacy budget (i.e., 10). It is promising apply differential privacy if there are concerns about the information leakage of the representations.
50
C DISCUSSION
Relation to Domain Adaptation Domain adaptation aims train model source domain (or nouti- domain), which has a good accuracy on a target domain. A classic and popular approach in domain adaptation is perform adversarial training, i.e., training discriminator to encourage
19

---

Under review as a conference paper at ICLR 2022
Table 10: ADCOL with different number of discriminators.
Number of discriminators
10
20
MNIST
94.7% + 0.6%
95.1% - 0.6%
91.0% + 0.3%
88.6% = 0.8%
SVHN
58.2% + 1.0%
45.4% = 1.1%
52.4% 1 1.2%
46.5% = 1.4%
USPS
95.4% + 0.2%
95.2% + 0.1%
95.3% : 0.2%
90.1% = 0.4%%
SynthDigit
76.0% + 0.3%
73.4% + 0.4%
67.2% = 0.8%
73.4% + 0.5%
MNIST-M
76.7% + 0.8%
76.4% = 0.7%
72.1%
= 0.9%
57.0% = 1.2%
AVG
80.2% + 0.5%
77.1% + 0.6%
75.6% + 0.8%
71.1% + 1.0%
Table 11: The test accuracy of ADCOL with different representation dimensions.
Dimension 512
1024
2048
MNIST
93.2% + 0.6%
94.9% + 0.5%
94.7% + 0.6%
SVHN
48,9% +: 1.2%
50.5% + 1.4%
58.2% + 1.0%
USPS
94.8% + 0.2%
95.5%
0.3%
95.4%
0.2%
SynthDigit
72.9% + 0.8%
79.4% = 0.7%
76.0% + 0.3%
MNIST-M
68.9% + 0.9%
73.1% = 0.7%
76.7% + 0.8%
AVG
75.7% + 0.7%
0.7% 78.7%
80.2% 0.5%
Nomain-invarian features (Ganin et al. 2016; Peng et al. 2019b). For the difference between our approach and domain adaptation, Peng al. (2019b) proposed FADA, which extends domain adaptation a federated setting. One connection between our approach and domain adaptation is that each party can be viewed
source domain, and the target domain is the unknown oracle optimal. Then, our approach is to extract domain-invarian features from multiple source domain, which is used regularize the training. To highlight the differences between our approach and the domain adaptation techniques, we compare our approach with the federated domain adaptation study (Peng ct al. 2019b) (FADA) and show the main differences: (1) Setting: FADA aims to train model on multiple source domain, which has a good accuracy on target domain. Our study aims to train personalized model for each party, which has good accuracy on its local data. (2) Discriminator: FADA uses multiple discriminators, where each discriminator is used for binary classification for one source-target domain pair. Our study uses a single discriminator for the multi-classification among all parties. Moreover, we have provided the theoretical analysis on the convergence properties. (3) Framework: FADA uses adversarial training to generate features. Our study uses adversarial training to regularize the local training in federated learning.
and domain domain-specific
Intuitively, we cannot directly compare ADCOL and FADA in the experiments since the settings are different. In our experiments, there is no a single target domain for testing in FADA. One method is to treat each party as a target domain and applying FADA N times, where N is the number of parties. However, the computation and communication overhead is significantly large. Moreover, such an approach does not utilize the labels of the target dataset. We have compared ADCOL and FADA using the above method and the results FADA. Moreover, the test accuracy of FADA is even smaller than local training in many cases since it does not exploit the labels of the target dataset.
shown in Table 18 ADCOL significantly outperforms
Limitations ADCOL is a collaborative learning method for non-lID features. As shown in Ap- pendix B.& the performance of ADCOL is poor compared with federated learning approaches on non-IlI label setting. Note that ADCOL aims learn a common representation distribution. Intu- itively, the task-specific representations of images from different classes should be very different, which can easily classified by a small MIP. Thus, if the label distribution varies across parties, the representation distribution naturally also varies lot across parties. The current objective of ADCOL does not into the non-III label setting.
As shown in Section 3.5 the communication size of ADCOL is related to the number of examples. If the number of examples is very large and the size of the model small, the communication cost of ADCOL will be larger than other federated learning approaches. However, local training can usually achieve satisfactory performance if the dataset size is very large. In such cases, besides ADCOL, existing federated learning approaches may also not help.
20

---

Under review as a conference paper at ICLR 2022
Table 12: The accuracy of different approaches non-Ill label settings.
SOLO
FedAvg
ADCOL
CIFAR-I0
59.3% + 8%
89.0% + 2.4%
79.2% + 4.7%
CIFAR-I00
33.5% 2.1%
57%
2.8%
36.1% + 2.8%
Table 13:
The performance of differerent approaches in the imbalanced data size setting.
SOLO
FedAvg
FedBN
ADCOL
FedProx
Per-FedAvg
FedRep
MNIST
93.9%
98. 1%
98.0%
98.7%
96.7%
94.5%
97.6%
SVHN
84.2%
88.3%
88.4%
87.1%
69.1%
76.7%
86.5%
USPS
94.2%
93.1%
93.6%
95.6%
94.0%
92.6%
94.1%
SynthDigit
94.0%
95.4%
95.6%
96.1%
93.7%
90.0%
95.6%
MNIST M
90.9%
89.7%
92.4%
94.2%
86.0%
82.5%
89.4%
AVG
91.4%
92.9%
93.6%
94.3%
87.9%
87.3%
92.6%
Table The total training time running all approaches for 100 rounds.
FedAvg
FedBN
ADCOL
FedProx Per-FedAvg
FedRep
Total Training time (hour)
6.5
11
8.5
Table 15: The comparison between sharing the predictor layers and not sharing the predictor layers.
w/ sharing predictor layers
w/o sharing predictor layers
MNIST
95.4%
94.7%
SVHN
48.8%
58.2%
USPS
95.2%
95.4%
SynthDigit
75.9%
76.
MNIST M
73.4%
76.7%
AVG
77.7%
80.2%
Table 16: ADCOL with different representation sampling rates. We present the final converged mean
accuracy and the number of communication rounds and communication costs achieve the target accuracy 78%.
Sampling rate
accuracy
#communication rounds
communication cost (GB)
20%
78.4%
32
0.07
60%
79.1%
26
0.17
80%
79.8%
23
0.20
100%
80.2%
21
0.23
Table 17: Differentially private ADCOL.
privacy budget non-private
2
5
10
MNIST
94.7%
89.1%
92.7%
93.1%
SVHN
58.2%
50.4%
56.7%
56.9%
USPS
95.4%
88.2%
93.0%
93.1% 1%
SynthDigit
76.0%
64.1%
69.4%
73.2%
MNIST M
76.7%
66.8%
71.9%
74.3%
AVG
80.2%
71.7%
76.7%
78.1%
Insights and Future Work The key insights from ADCOL are (I) a GAN-style training scheme and (2) regularization from view of representation distribution. While ADCOL does not have requirement on the vanilla local training algorithm, it can also be extended to self-supervised federated learning, where the cross- coosrpy loss is replaced by the loss used in self-super learning (e.g., contrastive loss (Chen et al. 2020 Chen and He (02T)). Moreover, while ADCOL only works on non-IID feature settings currently, the adversarial collaborative training scheme potentially be
21

---

Under review as a conference paper at ICLR 2022
Table 18: The comparison between SOLO, ADCOL,, and FADA Digits.
MNIST
SVHN
USPS
SynthDigit
MNIST_M M
AVG
SOLO
87.9%
34.8%
94.8%
63.0%
67.2%
69.5%
ADCOL
94.7%
58.2%
95.4%
76.0%
76.7%
80.2%
FADA
85.6%
40.1%
89.6%
68.8%
60.5%
68.9%
applied to address other data settings by modifying the objectives of local training and server training. There are many research opportunities based the findings of this paper.
22


Reviews:
[{"review_id": "uVnDlZVPf8g", "reviewer": "Reviewer_tbKF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n(1) It proposed a novel adversarial collaborative learning for federated learning on non-IID features.\n(2) It provided the convergence state of the training process.\n(3) Experiments showed it outperformed baselines.\n\nWeaknesses:\n(1) The overall idea is marginally novel, as it simply adapts the training procedures of “domain-adversarial neural network” [ref 1] into the federated learning. This idea has also been explored in previous work [ref 2]. In this case, [ref 2] also focused on the non-IID features in FL. It is more convincing to use [ref 2] as one of the baselines in the experiments.\n[ref 1] Ganin, Yaroslav, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. \"Domain-adversarial training of neural networks.\" The journal of machine learning research 17, no. 1 (2016): 2096-2030.\n[ref 2] Peng, Xingchao, Zijun Huang, Yizhe Zhu, and Kate Saenko. \"Federated Adversarial Domain Adaptation.\" In International Conference on Learning Representations. 2020.\n(2) One major concern of this method is to the privacy issue. It suffers much more seriously from the privacy leakage by sharing the hidden representations, compared to sharing only parameters in FedAvg. This might break the privacy constraint of federated learning. The communication cost also significantly increases due to the shared representations.\n(3) The theoretical results provided the convergence state of the training process, but it is not clear whether the proposed FL method can achieve the convergence result.\n(4) The train/test split for each client is not given.\n", "summary_of_the_paper": "This work presented an adversarial collaborative learning named ADCOL for federated learning on non-IID features. It learns the invariant feature representations for all parties.", "summary_of_the_review": "The novelty of the proposed model is incremental, and the privacy issue of sharing features is not solved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jdPmPz7U12X", "reviewer": "Reviewer_VD2k", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1). This paper studies an important problem -- the non-iid features problem in federated learning. The authors have some good ideas to tackle the problem.\n2). The paper presents both theoretical and numerical results. The experiments show that the proposed method has some advantages over some baselines.\n3). The paper is well-written in general.\n\nWeaknesses:\n1).  Using adversarial learning to add the non-iid features problem in federated learning is not new. The proposed method is similar to the FedUFO algorithm in  \"Federated Learning for Non-IID Data via Unified Feature Learning and Optimization Objective Alignment\" [Zhang et al. ICCV'21] (not cited). The authors might want to compare the proposed method with FedUFO. \n\n2). The proposed method requires the server to distinguish the devices' local representation.  In practice, only a small number of devices are selected for each round of federated learning. In addition, the number of the selected devices might also be different. It is questionable whether the devices can learn a common representation efficiently under these scenarios. Even if they can, these dynamics might significantly degrade the performance of the ADCOL algorithm. The authors might want to explain why the proposed method can be applied to these scenarios. The appendix has shown some results related to device sampling, but it only has one large sampling rate (0.4). In practice, the sampling rate is much lower (e.g., below 0.2).\n\n3). While the experiments appear to be comprehensive, there is a lack of study on how different degrees of non-iid influence the proposed algorithm's performance. Besides, the authors might want to increase the number of devices and adjust the device sampling rate.\n", "summary_of_the_paper": "This paper studies federated learning when different devices have non-iid features. To address the heterogeneity problem,  it proposes a federated learning scheme called ADCOL based on adversarial learning.  In ADCOL, the devices transfer local representations to the server while sending the discriminator to the devices. The server aims to distinguish the devices' local representations, while the devices aim to train local models that generate non-distinguishable representations. To make the representations non-distinguishable, ADCOL adds an additional regularization term to the devices' loss functions. This practice aims to maximize the probability that the discriminator cannot distinguish the local representations. The experimental results show that the proposed method has some advantages over several baselines.", "summary_of_the_review": "While this paper has some nice ideas to address the non-iid feature problems in federated learning, its contributions are limited due to novelty and practicability issues. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P5hYHKg6nLq", "reviewer": "Reviewer_gd59", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- The paper addresses an important problem, namely the one of federated learning that accounts for the fact the real-world data for collaborative learning is often non-iid in the features.\n- The paper is well-written and motivated, comparison to prior work is very well-done.\n- The proposed adversarial training method is intuitive and seems to perform well in practice.\n- The issues of communication complexity and privacy are discussed, even if those can in some situations be limitations of the proposed method.\n\nCons:\n\n- It is indeed possible that for certain modes (for example the local sample sizes being large as compared to the number of users N), the proposed method can be communication-inefficient.\n\n- Similarly, privacy can be seen as an issue. It is true that classic FL is also prone to privacy problems, however the proposed method sends multiple features, of the same input data, to the server at every round. Intuitively, it should be easier to infer information about the original data with so much information, rather than just with sharing (one) locally updated model at every round.\n\nSuggestions:\n\n- The paper will benefit from further explanation on why the proposed method (finding representation that obfuscates the differences of the distributions of the local datasets) is expected to work. The idea is intuitive and is in lines with classic domain adaptation techniques. However, expanding the discussion on that will be beneficial.\n\n- Similarly, equation (1) and the KL distance come a bit out of nowhere in the text. It will be nice to justify using the KL, as opposed to other distance measures.\n\n- In Section 3.1, it will be useful to repeat again the assumption that only the input distribution changes, since this is where you formally introduce the studied problem.\n\n- The conclusion (and in some sense the comparison to standard FL throughout the paper) seems to suggest that the method is more computationally efficient than other FL methods. The validity of this claim clearly depends on the values of multiple parameters of the problem, such as n, N, d etc. In this sense, the technical discussion is Section 3.5 is much more helpful to the reader and I would recommend against overselling the merits of the proposed method.\n\n- Related to the last point, perhaps the authors could comment on the possibility of communicating the representations of only a subset of the local data of each client at every round?", "summary_of_the_paper": "The paper proposes an adversarial training method for learning from non-iid data in federated learning. Both a theoretical and an experimental analysis of the method are provided.", "summary_of_the_review": "Overall, this is a very well-written paper that studies an important topic and proposes an interesting new idea. While there are some limitations, they are quite well-discussed (although improvements such as those mentioned in the \"Suggestions\" section are desirable) and I think that the paper will spark some interesting discussions and new ideas in the context of FL.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E5kWkGjXfPG", "reviewer": "Reviewer_3uWW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### strengths:\n1. The motivation of ADCOL (Adversarial Collaborative Learning) is both clear and reasonable. Indeed, the features in real-world scenarios are usually non-IID and how to apply federated learning on the non-IID data has always been an important question in practice. Therefore, it is quite necessary and challenging to design a thorough federated framework. \n2. The experimental evaluation seems to be convincing. Authors choose several typical fundamental tasks (digits, office-caltech-10, DomainNet) for experimental evaluation. The results demonstrated the effectiveness of the proposed ADCOL method.\n\n### weaknesses:\n1. In ADCOL, parties deliver representations of data instead of local models. It may lead to more serious privacy breaches. It needs more work to protect the local data.\n2. The advantage of communication size is not persuasive. In fact, smaller models than used in SOLO are often used\n3. The statements in this paper are not complete, especially in the experiment sections. There is a lot of confusion here, such as the statics of all the datasets,  the method for averaging results from different data sources (arithmetic average or weighted average according to data volume)  and methods for adjusting consistent label distribution but non-IID feature distribution, and the final distribution of label and feature.\n", "summary_of_the_paper": "This paper focuses on federated learning on non-IID features. This is a crucial problem when applying federated learning. The authors propose a new federated learning scheme, called ADCOL (Adversarial Collaborative Learning) for non-IID features. Specifically, the server is designed to train a discriminator to distinguish the local representations from local parties. While the local parties train the local models and expect the representations not to be distinguished by the discriminator. Authors conduct experiments on multiple parties where the data have heterogeneous features but share the same labels and label distribution and the results clearly show the effectiveness.", "summary_of_the_review": "As shown in the Main Review, the motivation of the paper is clear and reasonable, and also, the experimental evaluations are convincing. While there are also some issues that need further refinement to be accepted, especially the statements in the experiment.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes an adversarial method to address non-IID heterogeneity on federated learning. Distinct from existing methods, the mitigation is implemented by training and local communicating learned representations, so the metric of success is indistinguishability of representations across devices. 

There are four reviewers, all of whom agree that the method addresses an interesting and timely issue. However, reviewers are mixed on the paper score, and many raised concerns about communication overhead, apparent privacy costs, and convergence concerns with the adversarial methods. There is also some limited concern of novelty compared to existing methods. The authors provide a good rebuttal addressing these issues -- either based on experimental evidence (adding differential privacy), comparing communication overhead tradeoffs as it varies with model and sample size, and some existing convergence analysis. However, after reviews and discussion, the reviewers are unconvinced that the method is sufficiently robust to the concerns outlined. Authors are encouraged to address the highlighted technical concerns in any future submission of this work.
Decision: Reject