Collaborative Learning via Prediction Consensus
Authors: Dongyang Fan
Celestine Mendler-Dünner
Martin Jaggi
Venue: NeurIPS 2023 poster (2023)
Keywords: Collaborative training
decentralized learning
consensus reaching

Abstract: We consider a collaborative learning setting where the goal of each agent is to improve their own model by leveraging the expertise of collaborators, in addition to their own training data. To facilitate the exchange of expertise among agents, we propose a distillation-based method leveraging shared unlabeled auxiliary data, which is pseudo-labeled by the collective. Central to our method is a trust weighting scheme that serves to adaptively weigh the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. We demonstrate empirically that our collaboration scheme is able to significantly boost individual models’ performance in the target domain from which the auxiliary data is sampled. At the same time, it can provably mitigate the negative impact of bad models on the collective. By design, our method adeptly accommodates heterogeneity in model architectures and substantially reduces communication overhead compared to typical collaborative learning methods.

Collaborative Learning via Prediction Consensus
Dongyang Fan
Celestine Mendler-Dünnerr Martin Jaggi'
EPFL, Switzerland
Max Planck Institute for Intelligent Systems, Tübingen, Germany
$ELLIS Institute Tübingen, Germany
martin.jaggi jaggi)@epfl...... (dongyang.fal fan, martin.
celestineStue ellis.en
Abstract
We consider a collaborative learning setting where the goal of each agent is to improve their own model by leveraging the expertise of collaborators, in addition to their own training data. To facilitate the exchange of expertise among agents, we propose a distillation-base which is seeudo-labeled by the collective. Central to method is trust weighting scheme that serves to adaptively weigh the influence of each collaborator on the pseudo-labels until consensus how to label the auxiliary data is reached. We demonstrate empirically that our collaboration scheme able to significantly boost individual models' performance in the target domain from which the auxiliary data sampled. At the same time, it can provably mitigate the negative impact of bad models the collective. By design, our method adeptly accommodates heterogeneity in model architectures and substantially reduces communication overhead compared typical collaborative learning methods.
method leveraging shared unlabeled auxiliary data,
Introduction
This work considers decentralized learning setting where each agent has access labeled dataset and a local model. The agents may differ in the data distribution they have access to as well as the quality of their local models. In addition, we assume shared unlabeled dataset X sampled from a target distribution Q is available to all agents. The central question is how can agents effectively exchange information to benefit from each other's local expertise in order to improve their predictive performance on the target domain Q?
Towards this goal, our work takes inspiration from social science on how panel of human experts collaborate on a task. Humans typically engage discourse to exchange information, they share their opinions, and based how much they trust their peers, each individual will then adjust their subjective belief towards the opinion of peers. When repeated, this process gives rise to a dynamic process of consensus finding, as formalized by DeGroot [1] Central to the consensus mechanism of DeGroot is the concept of trust. It determines how much individual agents influence each other's s opinion, and thus the influence of each agent the final consensus.
Our proposed algorithm mimics this mansensus-findig mechanism in the context of collaborative learning, inspired by (2). In particular, our consensus procedure is aimed how to label the shared dataset X Therefore, we carefully design strategy by which each agent determines the trust towards others, given its local information, to optimally leverage each agent's 's expertise to collectively pseudo-label X This mechanism of knowledge distillation is then combined with techniques from self-training in order transfer the shared knowledge from the pseudo-labels into the local models in an iterative fashion.
Code available http:p////hhhha.///////ooooooooa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Crucial to our approach is that instead of aiming for consensus on model parameters, such as typically done in federated learning, we leverage the abundance of unlabeled data enforce consensus in prediction space over X" A key benefit of information exchange in prediction space is a reduction in communication complexity and the ability to control privacy leakage, but also the ability to seamlessly cope with both data heterogeneity and model heterogeneity.
Problem setup. We consider a collaborative learning setup with N agents. Each agent E [N] holds local training data, sampled from a local data distribution P.. We denote the local training data as (X,, 3r), where the matrix X, composes of n, local datapoints and yt denotes the vector of corresponding labels. The number n, needs not to be the same across agents. In addition, we assume agents have access to shared unlabeled dataset X sampled from a target distribution Q. We use n* to denote the number of datapoints in X". The ultimate success measure we consider is each agent's s prediction performance on the target distribution Q. We work under the assumption that sharing of raw labeled data is not desirable due to privacy concerns, data ownership, or storage constraints and we wish to keep communication at a minimum to avoid overheads. Our setting recovers the goal of both decentralized and federated learning, with or without personalization [4, when Q is defined as the prediction task the mixture of all local data distributions. In addition, we allow agents to differ in their model architectures. We do not require a coordination server but assume agents have all-to-all communication available.
Contributions. The key contributions of our work can be summarized in the following aspects:
1. We propose a novel collaborative learning algorithm based on irediction-consenan which effectively addresses statistical and model heterogeneity in the learning process, and provably mitigates the negative impact of low-quality participating agents.
2. Our algorithm is able to significantly reduce communication overhead and privacy-sensitive information sharing in comparison to other collaborative learning methods while achieving superior empirical performance.
3. We show theoretically that consensus can be reached via our algorithm and justify the condi- tions for good consensus to be achieved.
2 Related work
In federated learning a central server coordinates local updates toward learning a global model. Local nodes upload gradients model parameters, instead of data itself, to maintain certain level of privacy. McMahan et al. [5] describe the classic FedAvg algorithm. Follow-up works mainly and robustness towards Byzantine attacks. Apart from communicating gradients or model parameters, several works discuss alternatives to allow for heterogeneous model architectures. These methods are based on variants of model distillation [9] [0]. reaching an agreement in the representations space [II] or output space [12]. Similar to our work, both assume access shared unlabeled dataset. However, we go beyond naive averaging to determine agreement to account for heterogeneity in model or data quality.
focus on addressing challenges from non-i.i.d. local data 6
7 g
In contrast to federated learning, the fully decentralized learning setting does assume the existence of a central server. Instead, decentralized schemes such as gossip averaging are used to aggregate local information across agents [13 [4]. Despite the lack of a global state, such methods can provably converge to the desired global solution, leading to a gradual consensus among individual models (4. In this context, Dandi et al. [15. Le Bars et al. [16] optimizes the communication topology adapt to data heterogeneity but do not offer any collaborator selection mechanisms. Bellet et al. [ allows personalized models on each agent, but assumes prior information about task-relatedness, as opposed to learned selection. Gossip algorithms typically assume a fixed gossip mixing matrix given by e.g. physical connections of nodes [17 [8] 19.. These approaches fail consider data-dependent communication as with task similarities and node qualities. Several recent works have addressed this issue by proposing alternative methods that consider these factors. Notably, Li et al. (20 directly optimizes the mixing weights by minimizing the local validation loss per node, which requires labeled validation sets. Sui et al. (1 uses the E-step of EM algorithm to estimate the importance of other agents to one specific agent i, by evaluating the accuracy of other agents' models on the local data of agent i. This way of trust computation does not allow the algorithm to be applied to target distributions that differ from the local distribution, differentiating it from our work. Moreover, for

---

both Li et al. (20]. Sui et al. (1] the aggregation is performed in the gradient space, therefore not allowing heterogeneous models.
Our work relates to semi-supervised learning as it involves partially unlaheled data. Most relevant self-training methods [22] that first train model using laheled data, then use the trained model to give pseudo-labels to unlabeled data. The pseudo-labels can further be fed back to the training loop to attain a better model. Wei et [23] shows that under expansion and separation assumptions, self-training with input consistency regularization can achieve high accuracy with respect to ground- truth labels. When more than one learner is involved, co-training [24] appears as an extension to self-training, benefiting from the knowledge of learners from independent views in labeling a set of unlabeled data. Diao al. [25] incorporates SSL into federated learning. In setting where agents are with unlabeled data and the center server is with labeled data, experimental studies demonstrate
that the performance of a labeled server is significantly improved with unlabeled clients. Farina [26) presented collective learning framework for distributed SSL, where they combine predictions shared dataset via weights evaluated from local models' performances on local validation datasets. While their algorithm bears similarities to ours, it is important to note that it is exclusively tailored to scenarios which the target domain matches the global distribution. In a similar spirit, we want to leverage unlabeled data in fully decentralized setting.
Finally, Mendler- Bendler-Dünner al. have previously formalized collective prediction as dynamic consensus finding procedure. They demonstrated that such an approach can lead to significant gains over naive model averaging. We extend their approach from test-time prediction to collaborative model training.
3 Method description
Our proposed method is designed take advantage of shared unlabeled data in the context of collaborative learning through knowledge distillation. Therefore, it emulates human opinion dynamics collectively pseudo-label the shared auxiliary data. These labels are then incorporated in the local model update steps towards collectively improving the performance on the data distribution from which the shared data is sampled.
3.1 Collective pseudo-labeling
To describe the pseudo labeling step, let us use fe. to denote the local model of agent i E [N] parameterized by 0,. We write y, = fe. (X*) to denote the predictions of agent on the auxiliary data X Agents share these predictions with their peers. Naturally, the individual models may differ in these predictions and of the auxiliary data are not available. To combine the predictions into pseudo-labels for X", each agent locally decides how to weigh other agents' predictions by estimating their respective expertise on the target task. We refer to these weights trust scores and we Wij to denote the trust of agent i towards the predictions of agent j. It's worth noting that the trust between agents is not necessarily mutual, i.e.. can be asymmetrical: agent can trust agent without agent necessarily trusting agent back. We use W to denote the matrix trust scores. Given the trust scores, agent uses the following pseudo labels for the auxiliary data:
a priori unclear which model is most accurate, as ground truth labels
4/:
Σ Wtyūj.
(1)
Trust scores are determined locally by each agent based on query access to other agents' predictions and they are refined iteratively throughout training as models are being updated. The adaptive weight computation will be detailed in Section
3.2 Collaborative learning from pseudo labels
In the second step, the proxy labels for the auxiliary data are used to augment local model training. Therefore, in each step, the local optimization problem is augmented by a disagreement loss, and the new objective is given by
(X.), v4) (fo vb)
(2)
3

---

Algorithm Pseudo code of our proposed algorithm
For each agent E [N] we are given local model 0 0 labeled local dataset (X yz). and unlabeled shared data X
Input:
for / = 1,.,,, do
7
Each node E [AV] broadcasts their soft labels g/--1) ffor-n (X*) to all other nodes in parallel for each agent do
Calculate pairwise trust score wiy (j E [N]), based on the received soft decisions
(4)
using methods provided in Section
Get pseudo-labels on X from collaborators: (b( = E, wij 03 (1-11)
Do local training with collaborative disagreement loss
E arg min L()((X)), yi) + (fiss((o(
(3)
end for
where fe, (X) denotes the vector of agent i's predictions on the dataset X, C is the local training loss and dist(-) is a disagreement measure. We choose l2 distance for the disagreement measure in the regression case and cross-entropy for the classification case. À 0 is a trade-off hyperparameter that weighs the local loss and the cost of disagreement. This objective adheres to a conventional semi-supervised learning approach, however, we generate lseuds-labels a trust-based collective manner.
To iteratively refine the local models in the spirit of self-training, the pseudo labeling step and the local training step are performed in an alternating fashion as described in Algorithm T Starting from
pre-trained models (00), in each round E (1, T) model predictions the auxiliary data shared and then each agent aggregates them into a set pseudo labels to augment local data and
perform an update step.
Our algorithm is motivated conceptually by co-training 24 where was demonstrated that unlabeled data can be used to augment labeled data to boost model performance. Moreover, learning from collective pseudo-labels offers several additional benefits. Firstly, aggregation performed in the prediction space, climinating the need for all agents to have identical model architectures. Secondly, the communication cost of transmitting predictions is significantly lower than that of sharing model weights, and the same pseudo-labels tb: can be reused for multiple local epochs to further reduce the communication burden.
3.3 Convergence analysis
We study under what conditions Algorithm]] will reach a consensus among agents on how to label the auxiliary data. For the analysis, we focus on the over-parameterized regime T and we make the following assumption the local data distributions:
Assumption 1.
There is no concept shift between the local data distributions and the target domain Q from which the shared data is sampled, i.e.. (Y Q(Y XX- =x) for all i [N).
Together with over-parane pere-aaaetttrzatiinnn the assumption implies that the minimizer of the objective specified in ( can always reach zero loss. Further, this allows us to model the update of agents' predictions X a Markov process where the state transition matrix corresponds to the trust matrix W Therefore, it is convenient to write the update of the predictions on X performed by
the algorithm in matrix form, as (t)) (o(() W Adopting this notation we have for > 1
(t) (t (t-1) = W ((--)) (_
(4)
The following result provides sufficient conditions under which consensus will be reached by algorithm.
'ee say a model is wer-parameterized if its training emror can reach zero. Over paramtterizaiionn is reasonable assumption in the deep learning regime.

---

Theorem 1 (Consensus on predictions), Assume all agents' models are pprammtraeee and the data distributions satisfy Assumption 7 Then. for t - 30 Algorithm 7 converges to a consensus among the local models the predictions X", that is,
4
b (1)
Vi + j.
(5)
as long as W(t) is row-stochastic and positive for any t 0.
The proof is given in Appendix B and the main insight is that as long as W is row-stochastic and positive (that is Ej Wii = for any row i, and way > 0), the product of any W(l's is stochastic, irreducible, and aperiodic, and this leads to the differences between rows in (f) vanishing in time. Together with no concept shift, veerparramtterizatonn is important to guarantee that models can fit the consensus predictions on the unlabeled data while at the same time minimizing local losses. In contrast, in the under- parameterized setting, consensus and local loss minimization cannot necessarily be achieved the same time.
3.4 Information sharing in prediction space
key feature of our method is that agents do not share model parameters, but they communicate by exchanging prediction queries. If Algorithm I achieves consensus this means that agents arrive at solutions where they agree on predictions on X but this does not imply that they have learned the same model, or that they agree on predictions outside X To illustrate this, we provide a simple example where local data are generated using cubic regression with additive i.i.d. noise in the output, as shown Fig. [ this example, uniform weights are the ideal solution. We apply this optimal uniform trust, as the main purpose here is to illustrate the difference between information sharing in prediction space and parameter space. Each agent fits polynomial regression of degree 4, which leads to over-parumeterization of the model to fit the data. Full details of our example are given in Appendix A. We refer to the work of 2 for a similar setting with cadder-parametrrinee models. Here we note the most interesting observations in the over-parameterized regime.
Spert
sgari
sprrt
ROHOO
4330
watton
sgeni
eam:3
-3300
-5
-2
-10
(a)
(b)
(c)
(d)
(e)
Figure I: Local data distributions are shown in (a), and the initial fit on local data is shown in (b). (c)
and (d) predictions X after 5 rounds and 20 rounds of our algorithm update respectively. (e)
is the comparison of model fits in a larger range ((d) is zoom-in of the rectangular area of (e))
First, we observe that for T 2 20 the three agents reach a consensus on the predictions of X
However, the model parameters are not the same across the agents, as depicted in the rightmost panel. Further, considering the properties of the algorithm across rounds and the predictions in different regions of the input space in more detail, the following desirable behaviors observed:
a) In the region where agent has more data, it fits the local data more accurately and moves pseudo-labels closer to its own predictions.
b) In the region where agent has no or little data, agent only updates its model parameters to fit the pseudo-labels.
c) When local loss minimization and prediction consensus can be achieved at the same time, agents can arrive at models with a perfect agreement in the target prediction space.
Design of trust weights
In Section 3.3 we have shown that our algorithm is guaranteed to reach consensus on X under weak assumptions on the trust matrix W(O, In this section, we discuss how to design ( to encourage that the achieved consensus leads to high-quality labeling of X*.
5

---

Therefore, we focus on multi-class classification. We let fe, (x) denote the class probabilities obtained using model 0, for a datapoint æ. We choose the cross-entropy measure H((-)) -)) define the agreement loss function -ist(:,-)) in B X [2,,.., Fn] then fo (X) = (z)( fx R
where n is the number of samples in X and C' is the number of classes.
4.1 Trust evaluation through self-confidence
The quality of the local models could differ due various factors, such the amount of labeled data available during training, due to the expressivity of the local model, the training algorithm, or due to the relevance of the local data for the target task of labeling 2. Thus, a desirable property of the consensus solution is that malicious agents, or agents with low-quality models contribute less to the lseudo-labeling than agents with better models.
Hadjicostis and Dominguez-Gann Gareiia (27] differentiate between malicious and non maliciiou agents and they discuss the concept of trustworthy consensus, where only non-m con-malicious agents contribute to the consensus. In contrast to prior work, we do not aim for trustworthy agents to contribute equally. Instead, we specifically want consensus come from potentially unequal contribution of all agents, weighted according to their relevance. We allow for the trust matrix to be asymmetric. All agents determine trust from information given locally to the respective agent, which differs across agents. Central to any such strategy is that the capabilities of models on Q can be estimated appropriately. In the following, we discuss strategy of how determine trust from local data and prediction queries to other models.
As no label information X is available to evaluate trust, it is natural to use agents' own predictions on X as a local reference point. Then, each agent distributes their trust towards other agents based on the alignment of their predictions. We use weighted pairwise cosine similarity measure of alignment which motivates the following trust weight calculation:
(Jo: 1 (100), 12 11()))
(1) w
with
Σ
(x)
(6)
Σ,
T
1 (x) (2)) oo 1) (x)
The inclusion of the weighting factor 3/00
(x) and how to choose it will be discussed in Section 4.2
Self-confident trust. Naturally, pairwise cosine similarity leads to a trust matrix that has diagonal entries being the highest value among each row. We call this property self-confident, as each agent trusts itself the most. We now demonstrate that this property is not particularly restrictive. Even if constraining trust matrix be self-confident, it is still possible to design such a matrix that facilitates any consensus. The proof is given in Appendix D
Proposition 2. For any given consensus distribution n, it is always possible to find a trust matrix W that leads to which is both row stochastic and self-confident.
A second nice property our trust calculation has is that for an appropriate choice of ((() the proposed calculation of trust scores in (6) leads to scores that become more evenly distributed over time as agents gradually reach consensus.
Claim 3. Given Assumption 7 holds and that all agents are over-purameterizee. Assume (500 is chosen such that the trust matrix (tt) is row-stochastic and positive for all 2 0. Then, for the trust calculation in (6). we have ( loses self-confidence over time and finally converges uniform matrix:
tr(w(1)) s (r((((11) and
w 1=3 11
N
The proof is provided in Appendix C This claim characterizes the behavior of our dynamic trust scheme: while initially all agents distribute trust towards helpful collaborators and try to achieve a consensus on X Once consensus is reached, will have 2/6) g for any i, and (t) will become a matrix with uniform weights. This means no individual agent has increasingly high weight or the ability to manipulate the labeling.
Now that we know that a self-confident matrix can lead to the desired consensus, what other properties should our trust matrix have?

---

4.2 Robustness to low-quality nodes
If agents possess luw-quality local data, we aim to minimize their influence on the labeling of the auxiliary data throughout the algorithm. Proposition 4 gives sufficient condition for such desired consensus: there exists only one node with low-quality data, as long as it receives the lowest trust from other regular nodes, and the sum of trust receives is the smallest compared to the others, it will remain to have lowest importance in the consensus.
Proposition& 4. Given Assumption] 7 the trust matrix is row stochastic and positive, and all agents hold over per--paameterize models. Let be the only node with low-quality data and be the timestep that consensus is reached. If the following desirable properties hold for t <:
i) b receives the lowest trust from others than itself. i.e., Wjb = min; w for i b.
ii b-th column has the lowest column sum: E, w min E, 3
(0)
Then node b will have the lowest importance in the consensus.
The proof is given in Appendix where we also provide desired properties in the presence of multiple nodes with low-quality data, under some extra assumptions. Proposition 4 emphasizes the desired trust weights during training before consensus is reached.
When nodes with weak model architectures (such as pardeerprrmeterizeed models) are involved, achieving consensus is not assured. If such a consensus solution does exist, it will be constrained by the underfitting of weak nodes. Consequently, this solution would not serve as a stationary solution concerning the local training loss of a strong node. Nevertheless, we conjecture that these desired properties can still enhance training by mitigating the impact of the weak nodes.
4.3 Confidence weighting
In the following paragraph, we discuss the choice of the weights 8 in (6 Specifically, incorporate confidence weighting into the pairwise cosine similarity calculation to emulate the construction of transition matrix based a known consensus distribution.
Let us start by outlining an idealized trust calculation that effectively down-weighs agents with low quality data. We first construct an intermediate transition matrix & from pairwise cosine similarities of the agents' predictions on X" (with row normalization). For the low-quality node 6, we will have Φ5b being the lowest value in the j-th row, for any j * b. According to Proposition 4 in order to have low importance of low-quality workers in the consensus, we need to set the overall trust that receives to be the lowest among all the nodes. To achieve this, we need to assign the trust of regular workers towards the low-quality workers to a very small value, as it is difficult to alter self-confidence. If the consensus importance weight is known, one can easily calculate the corresponding trust matrix
Wegb = Ø5b min (1, T T(b) cbty
(j) 63b
which is a classical result from Metropolis chains
28 (also see Appendix Dj. We will have wys < 65
for j + b, as TŪ)
r(bb) should be sufficiently small.
Practical scheme. Since the consensus importance weight is unknown, we cannot attain the ideal trust matrix. Therefore, we propose alternative weighting scheme that achieves similar effects: we up-weight the similarity in the region where agent j has more confidence, i.c., where agent j's class probability assignments have lower entropy. By doing this, we encourage that the trust weights become more concentrated on themselves and helpful workers, and less concentrated on low-quality workers. We incorporate this into the trust weight caleulation (6) by choosing
8 ( (x) ((
where H denotes the entropy. We offer further intuition as well as justification of this weighting scheme in Appendix(F Moreover, we empirically demonstrate how our choice of trust matrix leads to low column sum for bad nodes in Section 5.2
7

---

8
5
2
(a) Local data distributions
(b) Dynamic Trust
Naive Trust
Figure 2: Decision boundary comparison between our dynamic trust update and naive trust update
633
633
Clam:
Figure 4: Learned trust matrix during training with diagonal entries masked and column sum reported the lowest bar. (left) Agents 2&9 have bad data. (right) Agents 5-9 have weak models.
(a) Cifar10
(b) FedISIC-2019
Figure 3: Class distributions among clients
Experiments
We start with a synthetic example to visualize the decision boundary achieved by our algorithm and then demonstrate its performance on real data in a heterogeneous collaborative learning setting.
5.1 Decision boundary visualization
Four classes are generated via multivariate Gaussian following P Ni E), where 140 (- 141 2) 42 = (-2, - M3 (2, -2) Σ 12x2. Four clients have local data sampled from a mixture of pets. For clients 0-3, we randomly flip 10% of the labels, and for client 3, we flip all labels. The unlabeled data X are sampled equally from P"'s. The data distribution shown in Fig- Z The base model used each node multi-layer perceptron
layers with 5, 10, and 4 neurons respectively. We now compare Algorithm Twth dynamic trust weight to with naive trust weight. When a client with low-quality data is involved, i.e. client in the toy example, trust update scheme gives better decision boundary to good agents after collaboration, blind trust towards low-quality clients will impair the effectiveness of pseudo labeling.
5.2 Deep learning experiments
Datasets and model architectures. We consider more challenging setting, where local data distributions are non-i.i.d. Two different statistical heterogeneities are considered: (1) Synthetic heterogeneity. We utilize the classic Cifar10 and Cifar100 datasets
29 and create 10 clients
from each dataset. To distribute classes among clients, we use Dirichlet distribution with
Unless specified otherwise, we employ ResNet20 [3 without pretraining. (2) Real-world data heterogeneity. A real-world dermoscopic lesion image dataset from the ISIC 2019 challenge
31 32 33
included here. The same client splits are used as (34) based on the imaging
acquisition system employed in six different hospitals. The dataset includes eight classes of lesions to
classify, with the class distribution among the clients displayed in Fig. Eb Following (34), we choose pretrained EfficientNet (35) the base model, and use balanced accuracy the evaluation metric. For every dataset, we construct X from equally contributed samples by every agent.
Comparison against baseline methods. We compare our methods with several baseline methods, including FedAvg (5), FedProx [7]. SCAFFOLD (8) (SCA), FedDyn [36), local training without collaboration (LT), and training with naive trust (Naive). Note with naive trust we are realizing soft majority voting, which represents the baseline method proposed from [1]. We adhere to the same
"The Dirichlet distributed samples are constructed using the codes from httpt:tttttttttte http:tthhhw..cmmmmiin//....
PFL- Non-Non-To IID

---

FedAvg
FedProx
SCA
FydDyn
LT
Naive
Ours-S
Ours-D
Cifar10
0.542
0.517
0.578
0.578
0.475
0.618
0.604
0.612
Regular
Cifar 100
0.261
0.240
0.317
0.310
0.178
0.311
0.319
0.308
Fed-ISIC
0.279
0.261
0.213
0.243
0.248
0.290
0.302
0.291
Low-
Cifarl0
0.541
0,530
0,570
0.575
0.470
0,596
0.605
0.608
Quality
Cifar100
0.254
0.240
0,289
0.308
0.171
0,285
0.300
0.306
Data
Fed-ISIC
0.229
0.242
0.221
0.243
0.217
0.247
0.249
0.269
Table I: Our methods compare to baseline methods. Blue denotes the algorithm with top accuracy and green denotes the method with 2nd best accuracy. "Ours S denotes the static version where the trust score is kept constant after first-time calculation (after 5 rounds of local training) and "Ours D" denotes the dynamic version where the trust score is updated per global round.
architecture setting, where the standard federated learning algorithms can be applied. To initiate the process, we allow each client to perform local training for global rounds, with the objective of obtaining sufficiently refined model that can be used for trust evaluation. From the 6th training round, the clients start collaboration. Over total of 50 global rounds, each consisting of local epochs, we report the averaged accuracy results from three repeated experiments in Table [ The evaluation metric is calculated on the dataset X is fixed as 0.5 in all experiments.
When all nodes share the same data quality and degree of statistical heterogeneity (denoted by "regular" in the table), our methods align closely with consensus through naive averaging. which is optimal in this case. When all nodes share the same degree of statistical heterogeneity but differ in data quality, exemplified by randomly selecting two nodes (indexed as 2 and 9) for a complete flip of local training labels, our dynamic trust update shows better overall performance!" proving the effectiveness of our approach in limiting the detrimental influences from nodes with low-quality data. We further plotted out the learned trust matrix in the dynamic update mode during one of the middle training rounds in the left plot of Fig. 4 Clearly, our algorithm able to give low trust weights the nodes with low-quality data, and the 2nd and 9th columns have the lowest column sum.
Adaptability to varying model architectures. We allo- cate a more expressive model architecture to the first half of the nodes and less expressive one to the other half. The former comprises ResNet20 and EfficientNet, which were the models of choice in the previous experiments. For the latter, we employ a linear model (i.e., one-layer fully connected neural network) with flattened image tensor as input and the output is of size equivalent to the number of classes. It is worth noting that if agents with strong and weak model architectures (as in cases of under-parameeterzzation) coexist, consensus might not cur, as suggested by our empirical findings illustrated in Fig. 5 Nevertheless, our trust-based collaborator selection mechanism consistently outperforms local training and simple averaging. The trust weight matrix learned during Cifar100 training is depicted in the right plot of Fig. revealing the presence of asymmetric trust. Specifically, the last 5 nodes exhibit higher level of trust towards the first nodes, while the opposite not true. The trust al- location is desired in identifying the helpers. We further refer to Appendix A.2 for more empirical evidence from toy polynomial regression example the presence of strong and weak architectures.
0.6
0.2
2
0.0
0.1
010
0.05
0.00
0.4
callab
E8 rolah
dyraric
trath dynariz
Malin
0.0
0
clents No.
Figure 5: Target accuracy comparison with 2 different model architectures with error bars (hatch pattern denotes fully connected NN is used). From top to bot- tom: Cifar 10, C00, FedISIC
Reduced communication costs. Gradient aggregation- based methods incur significant communication burden proportional to the number of model parameters (O(N x
[params()), which is particularly heavy given the parmpearaeteriized nature of modern deep learning.
Here report average accuracy of regular workers, excluding workers with low-quality data

---

accuracy X
0.3
0.30
accuracy on
0.2
0.25
Ours
0.1
FedAvg
0.20
3
10
20
0
10
20
50
#local epochs
#global rounds
Figure 6: Algorithm performances on Cifar100 for different algorithm configurations. (left) effect of varying number of local epochs on final performance; (right) algorithm performance as a function of the number of training rounds for 5 local epochs each
In contrast existing approaches, our proposed method significantly reduces the communication burden by enabling each node to transmit only their predictions the shared dataset. This results in communication overhead ( x n x C). clear that this value does not scale up with more complex models, and is much smaller than the model size. Moreover, our methods maintain their high performance even when the number of local epochs increases. On the other hand, FedAvg loses its effectiveness with less frequent synchronization, i.c. more local epochs between global aggregation rounds, as shown the left panel of Fig.
Conclusions and extensions
the context of decentralized learning, we leverage the collective knowledge of individual nodes to improve the accuracy of predictions with respect target distribution. Our proposed trust update scheme, based on celf-confidenee, ensures robustness against nodes with low-quality data. By achiev- ing consensus in the prediction space, method effectively handles diverse model architectures within local clients, while maintaining a low communication overhead, thereby exhibiting important practical potential. Despite coming from different perspective, trust-based collaborative pseudo- labeling method may provide some inspiration in the semi-supervised learning community. Notably, algorithm is intrinsically compatible with personalization, in terms of allowing some concept shift across clients. We leave this for future work.
Robustness. We have designed our algorithm with the assumption that all agents communicate hon- estly, meaning that no Byzantine workers intentionally provide incorrect information. Nevertheless, our method exhibits some resilience against a common Byzantine attack, known as the label flip (referred "low-quality workers" in our paper). For instance, even with out of 10 workers having 100% flipped labels, our algorithm maintains good performance. If there are malicious workers deliberately providing incorrect information, the nodes may refuse reach consensus, instead of reaching a detrimental bad consensus, assuming reasonable À is chosen. Consider the scenario in which detrimental consensus reached with malicious nodes involved; in this case, the consensus loss and local loss for regular nodes will not decrease in the same direction, making the consensus solution non-stationary. Notably, the "personal" component our loss function adds an element of robustness against malicious nodes.
Privacy Concerns. While previous works show that training data can be reconstructed from model parameters (37] or gradients (38), our algorithm requires less privacy-sensitive information sharing, shared dataset. While private information on training data due to memorization [39], there is a trade-off between the gain from collaboration and the amount of information that users are willing to share. As the number of outer rounds increases, we observe a notable improvement in accuracy within the context of X". However, this enhanced accuracy comes the cost of disclosing more information, relationship that is depicted in Fig. 6 An interesting extension would be to apply differential privacy to further guarantee privacy.
which is predictions
aware that model predictions
still leak
Acknowledgements. DF would like thank Anastasia Koloskova, Felix Kuchelmeister, Matteo Pagliardini, and Nikita Doikov for helpful discussions during the project and El Mahdi Chayti for proofreading. DF acknowledges funding from EDIC fellowship from the Department of Computer Science at EPFL. CM acknowledges support from the Tübingen Al Center. This project was supported by SNSF grant 100020_200342
10

---

References
[1] Morris H. DeGroot. Reaching a consensus. Journal of the American Statistical Association, 69 (345):118-12) 121, 1974. ISSN 01621459.
[2] Celestine Mendler-Dünner, Wenshuo Guo, Stephen Bates, and Michael Jordan. Test-time collective prediction.
Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 13719-13731, 2021.
[3] S. Fralick. Learning to recognize patterns without a teacher. IEEE Transactions on Information Theory, 13(1):57-64, 1967.
[4] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D D'Oliveira, Hubert Eichner, Salim EI Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Har- chaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. ArXiv preprint 1exiv:9912.04977. 2021.
[5] H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas. Federated learning of deep networks using model averaging. CoRR,
2016.
[6] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arxiv: 0058, 2018.
[7] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2020.
[8] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, volume 119, pages 5132-5143. PMLR, 2020.
[9] Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In International Conference on Neural Information Processing Systems, 2020.
[10] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for hetero- geneous federated learning. In International Conference on Machine Learning, volume 139, pages 12878 22878-12889, PMLR, 2021.
[11] Disha Makhija, Xing Han. Nhat Ho, and Joydeep Ghosh. Architecture agmostic federated learning for neural networks. In International Conference on Machine Learning, volume 162, pages 14860-14870. PMLR, 2022.
[12] Amr Abourayya, Michael Kamp, Erman Ayday, Jens Kleesiek, Kanishka Rao, Geoffrey I. Webb, and Bharat Rao. AIMHI: Protecting sensitive data through federated co-training. In Workshop on Federated Learning: Recent Advances and New Challenges (at NeurlPS), 2022.
[13] Aurélien Bellet, Rachid Guerraoui, Mahsa Taziki, and Marc Tommasi. Personalized and private peer-to-peer machine learning. In International Conference on Artificial Intelligence and Statistics, volume 84, pages 473-481. PMLR, 2018
[14] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Conference Machine Learning. volume 119, pages 5381-5393. PMLR, 2020.
[15] Yatin Dandi, Anastasia Koloskova, Martin Jaggi, and Sebastian U. Stich. Data-heterogeneityy aware mixing for decentralized learning. In OPT 2022: NeurlPS Workshop on Optimization for Machine Learning, 2022.
[16] Batiste Le Bars, Aurélien Bellet, Mare Tommasi, Erick Lavoie, and Anne-Marie Kermarrec. Refined convergence and topology learning for decentralized sgd with heterogeneous data. In

---

International Conference on Artificial Intelligence and Statistics, volume 206, pages 1672-1702. PMLR, 2023.
[17] Lin Xiao and S. Boyd. Fast linear iterations for distributed averaging. In IEEE International Conference on Decision and Control, volume 5, pages 4997-5002, 2003.
[18] S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah. Randomized gossip algorithms. IEEE Transactions Information Theory. (2(6))2508-2555)) 2530,, 2006.
[19] Mahmoud Assran, Nicolas Loizou, Nicolas Ballas, and Mike Rabbat. Stochastic gradient push for distributed deep learning. In International Conference on Machine Learning. volume 97, pages 344-353. PMLR, 2019.
[20] Shuangtong Li, Tianyi Zhou, Xinmei Tian, and Dacheng Tao. Learning to collaborate in decentralized learning of personalized models. In Conference on Computer Vision and Pattern Recognition ( (CVPR), pages 9756-9765, 2022.
[21] Yi Sui, Junfeng Wen, Yenson Lau, Brendan Leigh Ross, and Jesse C. Cresswell. Find your friends: Personalized federated learning with the right collaborators. Arxiv preprint arxiv:2210006559 2022.
[22] Dong-Hyun Lee. Pseudo-label The simple and efficient semi-supervised learning method for deep neural networks. 2013. URL https://app
18507866
[23]
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations (ICLR), 2021.
[24]
Avrim Blum and Tom M. Mitchell. Combining labeled and unlabeled data with co-training. In Peter L. Bartlett and Yishay Mansour, editors, Annual Conference on Computational Learning
Theory (COLT), pages 92-100. ACM, 1998.
[25] Enmao Diao, Jie Ding, and Vahid Tarokh. Semifl: Semi-supervised federated learning for unlabeled clients with alternate training. In Advances in Neural Information Processing Systems, volume 35, pages 17871-17884, 2022.
[26] Francesco Farina. Collective learning. Arxiv preprint arxiv: 2021.
[27] Christoforos N. Hadjicostis and Alejandro D. Dominguez-Garria. Trustworthy distributed average consensus. In Conference on Decision and Control (CDC), pages 7403-7408. Institute of Electrical and Electronics Engineers Inc., 2022.
[28] D.A. Levin, Y. Peres, and E.L. Wilmer. Markov Chains and Mixing Times. American Mathe- matical Soc., 2008
[29] Alex Krizhevsky. Learning multiple layers of features from tiny images. pages 32-33, 2009. UtLhttt:::/wwwt
toronto. "ertuu//ri//// wwwri//eeariiii wwwpniiing-eattee--eeeeee
pdf
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[31] Rosendahl C. Tschandl P. and Kittler H. The ham 10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions, 2018.
[32] Noel C. F. Codella, David Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, and Allan Halpern. Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), 2017.
[33] Mare Combalia, Noel C. F. Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C. Halpern Susana Puig, and Josep Malvehy. Ben20000: Dermoscopic lesions the wild, 2019.
[34] Jean Ogier du Terrail, Lamy-Safwan Ayed, Edwige Cyffers. Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, Boris Muzel- lec, Constantin Philippenko, Santiago Silva, Maria Teleńczuk, Shadi Albarqouni, Salman Avestimehr, Aurélien Bellet, Aymeric Dieuleveut, Martin Jaggi, Sai Praneeth Karimireddy, Marco Lorenzi, Giovanni Neglia, Marc Tommasi, and Mathieu Andreux. Flamby: Datasets and benchmarks for cross-sill silo federated learning in realistic healtheare settings, 2022.
12

---

[35] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, volume 97, pages 6105-6114 14. PMLR, 2019.
[36] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regularization. In International Conference on Learning Representations (ICLR), 2021.
[37] Niv Haim, Gal Vardi, Gilad Yehudai, michal Irani, and Ohad Shamir. Reconstructing training data from trained neural networks. In Advances in Neural Information Processing Systems, 2022.
[38] Zihan Wang, Jason Lee, and Qi Lei. Reconstructing training data from model gradient, provably. In International Conference on Artificial Intelligence and Statistics, volume 206, pages 6595- 6612. PMLR, 2023.
[39] Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 35:13263-13276 2022.
[40] J. Wolfowitz. Products of indecomposable, aperiodic, stochastic matrices. Proceedings of the American Mathematical Society, 14(5):73--777, 1963.
13

---

Appendix
A Polynomial regression examples
A.1 Example in Section 3.4
The underlying function is chosen as [xx) 0.5x 0.3x² 5x + 4. There three agents in total, each of whom has 50 data points. The local data points are generated using normal distributions: J1 (M222, 1), 22 (0, 1) and 0g (2, 1). To introduce noise in the labels, each agent adds a normally distributed eπror term with zero mean and unit variance, i.c. y/ = f(x,) + E with e 1).
set of 50 equally spaced data points in the range of -4 to 4, denoted as X used in the analysis. The algorithm is applied using fixed trust weights with 1/3 in each entry and À is chosen as 1.
A.2 Example with strong and weak architectures
The true underlying function is chosen as fz) 0.5z + 0.3.. 5x + 4. There are four agents in total, each of whom has 50 data points. The local data points are generated using normal distributions: I ~ (V 1), 22 ~ NO0 1), 23 ~ (2, 1) and x4 N(3, 1). To introduce noise in the labels, each agent adds normally distributed error term with zero mean and unit variance, i.e. y: - f(x; +e with NO 1).
set of 50 equally spaced data points the range of -4 6, denoted as X used in the analysis. The algorithm is applied using dynamic trust weights and À is chosen as 1. For the first three agents, polynomial model with maximum degree of four is fit, while for the fourth agent, a polynomial model with maximum degree of one is fit, signifying a weak node.
We that after 50 rounds of model training using our proposed algorithm with dynamic trust, agent 4's model is still underfitting due to its limited expressiveness. Agents 1-3 end up agreeing with each other and giving good predictions in the union of their local regions. While with naive trust weights, we see that the strong agents also get influenced in the region where they could perform well, as the underfitted model has stronger impact through collective pseudo-labeling.
10
#poit
agentt
90
30
CD-
agent
açenta
20
aquett
true re ation
true re settion
acend4
10
20
20
0-
-10
-20
-2
(a)
(b)
(c)
(d)
Figure 7: (a) local data distributions in each agent; (b) local model fit without collaboration; (c) model fits after 50 rounds of our algorithm with dynamic trust update; (d) model fits after 50 rounds with naive trust update
B Proof of Theorem 1
The proof is rooted in the results from the work of Wolfowitz 40), we recommend readers to check the original paper for more detailed references. Note, for the following texts, when say matrix W has certain properties, it is equivalent to saying a Markov chain induced by transition matrix W has certain properties.
Definition (Irreducible Markov chains). A Markov chain induced by transition matrix W is irreducible for all j. there exists some n such that W 0. Equivalently, the graph corresponding to W is strongly connected.
Definition B (Strongly connected graph). A graph said to be strongly connected if every vertex reachable from every other vertex.
14

---

Definition C (Aperiodic Markov chains). A Markov chain induced by transition matrix W is aperiodic if every state has a self-loop. By self-loop, we mean that there a nonzero probability of remaining in that state, ie. wi for every i.
Assumption 2. W 's are row-stochastic and positive, i.e. Σj Wij = for any row i, and wij >
Claim 5. Given Assumption 2 the matrix product of any n elements of (W()) are SIA (SIA stands for stochastic, irreducible and aperiodic) for n. 1.
Proof. According to the assumption that all ( are positive, and thus we have any product of W being positive in each entry, which is equivalent to the graph introduced by the product being fully connected. Being fully connected implies being strongly connected. According to Definitions A irreducibility follows.
By the product being positive, we also have its diagonal entries being all positive. According to Definition C aperiodicity follows.
The product of row-stochastic matrices remains row-stochastic: for A and B row stochastic, we have the product AB remains row-stochastic.
Σ A
aszbba) Σ aik Σ bkj) = 1, Vi
Thus, have any product of ( being irreducible, aperiodic and stochastic (SIA).
Theorem 6 (Rewrite of Wolfowitz 40 Let Ai, .... Au be square stochastic matrices of the same order and any product of the A's (of whatever length) is SIA. When k - oo, the product of A ... A& gets reduced to matrix with identity rows.
Following Assumptions we have (i()) = (t (lghtt))11 holds for all 2 From Claim we have
any products of ((''s being SIA. From Theorem 6 we have the product (t-1)) 1T. W
gets reduced to matrix with identical rows when / goes to infinity. That implies, to has identical rows. The statement thus proved.
C Proof of Claim 3
Definition D (Row differences). Define how different the rows of W are by
5(W)= = max max [Wiyj Wizjl
(8)
31.2
For identical rows, 8(W)=
Definition E (Scrambling matrix). W is a scrambling matrix if
X(W) = min 21.22
Σ
min(wi.h Wizs) <
In plain words, Definition E says that if for every pair of rows i and i2 in matrix W, there exists column j (which may depend on i, and i2) such that Wiy > 0 and Waij > 0, then W is a scrambling matrix. easy to verify that a positive matrix always a scrambling matrix.
Lemma (Adaptation of Lemma 2 from Wolfowitz [Q]). For any t.
((WW) (() s III XW(
(10)
Lemma [ states that multiplying with scrambling matrices will make the row differences smaller. tr (r((t)) E; (f) represents the sum self-confidences all nodes. As every ( is positive, we have all w scrambling. Thus, the differences between rows of W(D (--1), (/1) get smaller when gets bigger.
As (0( E, 1) 11y we have the predictions on X given by all nodes get similar over time. According caleulation of (t))) in Equation 6). which is based cosine
15

---

similarity between predictions, it follows that an agent's trust towards the others gets larger over time. ((+1)
That is, Σj Wij
Σj wij Since each row sums up to I, we have wi (t+1) (t)
w for all i.
According Theorem have (/( xb) t oc, for any i and According to the calculation of W, we have W with equal entries when t reaches infinity.
D Proof of Proposition
Recall stationary distribution (ι E RRXN) ) of a Markov chain being
lim W w (-00
(1I)
The proof follows from the construction Metropolis chains given stationary distribution. We will first give an example of how Metropolis chains work.
Example 2 (Metropolis chains (28) Given stationary distribution 00.3, 0.1}. how
could we construct transition matrix that leads to the stationary distribution?
Suppose & is symmetric matrix,
construct Metropolis chain P as follows:
(x. v) min (1,
s(v)
m(m))
p(z, v)
(12)
Exfe (x, 2) min
(1,
x(=)
y =
1/3
1/4
1/4
1/6
T4/9
1/4
1/4
1/18
1/4
1/3
1/4
1/6
1/4
4/9
1/4
1/18
Choose +-
could get P
1/4
1/4
1/3
1/6
1/4
1/4
4/9
1/18
1/6
1/6
1/6
1/2
1/6
1/6
1/6
1/2
can be verifiid that æ the stationary distribution of Markov chain with transition matrix P. If Φ
is not symmetric, we modify s(y)
x()) p(y.)) and the results remain unchanged.
to
(()) ((2.9)
Following Example 2 choose to be any self-confident doubly stochastic matrix. For all x, choose P calculated from [2), have
p(a, x) =1 Σ plx 2) min (1, [() Txx)
2 21
Σ zæ0
Φ(x,2)
(13)
zΦp
we see that probability distribution among each row gets more concentrated on the diagonal entries in P than 4. As already has high diagonal values, the claim follows.
E Proof of Proposition 4
Proposition 4 states sufficient conditions for ( to have such that a low-quality node is assigned lowest importance in æ, i.e. n, min; æ. From Equation (T1). comes from the product of trust. matrices. We start from a product of two such matrices.
Proposition 7. For row-stochastic and positive matrices A and B, and C AB, if in both A and B.
i) j-th column has the lowest column sum,
ü) (i,j)-th entry being the lowest value in i-th row for +
then have j-th column remains the the lowest column sum in matrix C and (f, entry being the lowest value in i-th row of C for + j.
Proof. Let C ABB, the column of column of C can be expressed as:
Σ Ciz Σ
>
anpbis
(14)
Σ
ask Dbk
16

---

for t * j. the column sum of C' is
Eon = -
R
(15)
Σ
We first show that j-th column remains the lowest column sum in C. For t f
Σa
Σ
Cij
ask
(bus bus)
(bk) bus) + 1 aig (bFt - bjz)
a+j) bus) + 1 azd (bye bرt)
kØj
(16)
Σ aij)
(bue bug) + (bge -bjs)
Kij
Σ aij Σ bkt Σ bus
(i8)
0
(i) holds because for + j. bat - baj > 0 and Σ, asj <; ask
(ii) holds because the j-th column has the lowest column sum in B
We then show that (ith entry remains the lowest value in i-th row of C for A For + j. we have
Σ askbur Σ assbbz
Cit - Cij
Σ
ask (hee buj) + ais(bbt b53)
(182)
aij (bur bus) + ais ais(bjr - bzs)
(17)
Σ (bke - buy) + (bye bUy)
=a+
Σ bit Σ bki
=ūij
(iv) > 0
(iii) holds since byc - buz > 0 and ask > asy for i $
(iv) holds because Ee bat Σ bkj
For time-inhomogenous trust matrix, Assumptions ensure the Markov chain update: 4/!0 (0) E 4/th ()) which is followed by consensus as proven in Theorem We see that b-th column remains the lowest column sum in the product W( (!) by iteratively applying Propo- sition 7 For / 2 т, multiplying consensus with any row stochastic preserves the consensus. Thus, the b-th column will remain to be the smallest column in the consensus. For the time-homogenout case, as long as W holds the same properties, one can easily verify that the same result still holds. Thus, Proposition is proved.
17

---

i) the gap between the sum of trust from regular nodes towards any regular node r and low- quality node b is larger than the gap between low-quality node b's self-confidence and its Extend to more than one node with low-quality data. For more than one low-quality node, what are the desired properties (sufficient conditions) for the transition (trust) matrices have? turns out that apart from the two conditions in a single low-quality node case, we need an extra assumption.
Proposition 8. Given Assumptions 22 and that all agents indices of regular nodes, and B be the set of indices of low-quality nodes, for t S T. W satisfies the following conditions:
parameettter. let R be the set of
i) any regular node's ' column sum is larger than any low-quality node's s: milleER E, Wir maX :
trust towards the regular node: ΣnER (wit) - (t)) > (wis -wbr (t)
(t)
)
iii) any node's 's trust towards regular node is bigger or equal than its trust towards low-
quality node other than itself: for any RR and any b B, we have wor w holds as long as n + b.
And after > T, W = 11 1. N. Then we have nodes in B having a lower importance in the consensus than nodes in R.
Proof. First, let us look at the multiplication of two such matrices when 1 < t <, for any r E R and b B, we have conditions (1)(2)(3) remain be true for the product (((-), We will verify them one by one in the following part:
Verification of condition (I): any regular node's column sum is larger than any low-quality node's in (((tt),) 1), For any E R and any b B, we have
ΣΣ
(t) w(t-
ΣΣ
(t) (t-1)
(w((t11)
wnb (((11)
W
Win
W (t))
" (((--)) w (1-11)
(0) (w(f-))
wht (
in
nER
(b))
(t-1)
(t-1))
W
Wbr
w/b
with (1))
(w (1-11)) + Σ
nr
wib (uprr
w/o
(0-1)
nER
+
Σ
(t) Win
(witt 1) (1-1)
w/b
NEBY (b)
Σ MER
() 1)
Σ
wit-
1)
+ w/ (t-1)
- -w/ (1-1)
fr
nb
nER
+ Σ
(!(!)
(w(lt1)
w ((--11 (t-1)
(EB
(6)
> 0
(i) holds because Σ, (t)
wm for any R is larger than E, w for any b E B, which follows from condition (1), and whr wab 0, which follows from condition (3).
(8)
(ii) holds following the conditions (2) and (3). From (2), EnEr w (1-1)
EnER w((-1) w
(t-1) 0, and from (3), wnr nb
(t-1) (1-1) for * b
Whb
18

---

Verification of condition (2):
Σ
(1-1)
Σ
with with (t-1) (01111
Σ
(e) (t-1)
Σ
Wpp wop (-11)
(1)
w
up
up w
pb
w
up
neR
P
P
Σ AER
W
+
Wbp
(t)
w
(t-11
np
(t)
+
wbp
pb
(1-1)
P
NER
(8)
+ wop
(8)
(wpt)))
(t-1)
nER
Σ
Σ
(t(t)
np
+ bh
(0)
(w
pr
10)
pb
Σ [B
Σ AER
w + m0()) )
(wtt
u (1(1-1))
pb
PER
AER
(t)
+ (t)
(t-1)
(1-1)
+
Wh
Wab
whr
Wpb
(166)
Wab + (t)
(6)
(w)
w 20 ((111)) +
(8)
(t)
(w
w (1111))
wbb
W + Wwbb
PER
MER
NER
+
Σ
+ wop
(2)
(w/t1))
w
(1-1)
({}}
AER
Σ
nb + 1/b
(1)
Σ
wit
Σ (t-1)
+ wbr
(1-1)
wib
(1-1)
MER
PER
PER
Σ
(1) whp
(wpt 1)
(1-1)
+
pb
(b)
AEE
(i0)
20 0
(iii) holds because for p a regular node, we have E ER Whp + (0) > EnER w nb + wg which
(t)
follows from condition (2), and wor (t-11) with (t-1) 2 0 for + b, following from condition (3).
(iv) holds because of conditions (2) and (3).
Verification of (3): for n # b, we want to show the trust towards regular node is bigger than (2) (t) (e) (t) whp pb towards a low-quality node b, that is Ep whp war
Ep
Σ
wi(t)
(t)
Σ
Σ
tllt) np
(t)())
pr
m pbb
(1)
+
Σ
wi(l) up
(w(())
v0(()
+
(t
nb
(w
w((
PER
PE
(b)
(1)
2
Σ
w
nb
(t)
(t)
w
pb
(1)
+ (t)
w
((()
+
np
(())
pr
w
R
PEBi
(t)
Σ
(t)
Σ
(()
(t)
(e)
w((t] ipp
(w(()
2 (0())
w
pr
pb
+
whr
wib
PER
PER
PEB((0)
(0i)
(v) holds because for o b, we have who (1) 2 wpb following from condition (3), and wpp for p * b.
(t)
(0) (1)
w 2 0
19

---

(vi) holds following from conditions (2) and (3).
follows that in the product W W(T-1),,, W a low-quality node will still have a lower column sum than any regular node. Because conditions (3)(2)(3 holds for any product of W 's as long as each of the ((() share the conditions listed by (1)(2)(3).
After > T, multiplying with a naive weight matrix does not change the column sum order, we will have all low-quality nodes have lower importance in the consensus than the regular nodes.
F Reasoning for confidence weighting factor ((0)
In this section, we justify our choice of 8 (x) in Section 4.3 i.c. we show via adding such term, we are able to downweight regular node's 's trust towards a bad node.
((() is row-normalized pairwise cosine similarity matrix, with jith entry before row normaliza-
tion
(FoR-n (10))( fo(-))))))
Σ x'EX- 15 (2: (x:) )ll2 S 1) (a')
(18)
After adding (x) = (1) we have ( with (i, entry before row normalization as
(Fo (fon-1(a n (x)), (0
1 Σ E'EX
(19)
n2 2
HlSo: 0/- 12 -((0))-)((x))) (xx)) 15 (x') (2)1l21 S 03 n1 (x2)
We want to show that the weighting scheme down-weights regular node i's trust towards low- quality node b, that is
(8) 3
As the comparison is made with respect to the same time step t, we drop the notation from now on. Let (4o: ... -1 be the cosine similarity between regular agent i and others inside agent i's confident region, and (bg...bb confident region. By confident region, we mean region with low entropy in class probabilities, i.e. the model is more sure about the prediction. Further, we make the following assumptions:
be the cosine similarity between and others outside agent i's
a) for æ' in agent i's confident region, we have low entropy of predicted class probabili- ties: H (xx)) 1/cr; while for x' outside agent i's confident region, we have H
1/e2. We further assume 0 C2 C1.
b) inside regular node i's confident region, has better judgment of the alignment score produced by cosine similarity, such that the cosine similarity with low quality is weighted lower inside:
by Σ3 bj
(20)
aj
to claim wab dia. we need to show
cas + c2bs
ab + bb E, + by)
(21)
E, + tzb,,)
Proof. Re-arrange Equation 20 get
by Σ aj>> > 0b Σb; bj
(22)
)
Multiply with cı on both sides, we have
(c2 c1)b Σ
aj
(02 ciaab Eb3
(23)
20

---

c2bs Σ aj + cιab Eb < cbb E aj + + E by
(24)
j
Now add ciab E, a + c2b, Ej by to both sides, we have
Σ
aj + o2bb Σ
aj + a E bj + +cbb Ebe =
ab
j
j
(25)
C1 Σ
aj + cbbb E aj c2ab E by +zbbb Σ bj
Combining the terms we have
(cias + o2bb)
Σ (a, +b,) (b))
E(era + t2bb)) ) (as + bb)
(26)
following which, we directly have
arbb + czbb E,(cia + tobb))
ab bb
(27)
E, (as + bj)
G Complementary details
G.1 Details regarding model training
All the model training was done using single GPU (NVIDIA Tesla V100). For each local iteration, we load local data and shared unlabeled data with batch size 64 and 256 separately. We empirically observed that larger batch size for unlabeled data is necessary for the training to work well. The optimizer used is Adam with learning rate 5e-3. For Cifarl0 and Cifar100, as the base model is not pretrained, we do 50 global rounds with 5 local training epochs for each agent per global round. For Fed-ISIC-2019 dataset, as the base model is pretrained EfficientNet, we do 20 global rounds. For the first 5 global rounds, we set trust more fairly. After that, À is fixed as 0.5. Dynamic trust is computed after each global round, while static trust denotes the utilization of the initially calculated trust value throughout the whole experiment.
arrive at good local models, such that every agent can evaluate
For Cifar10 and Cifar100, equal representation. For the rest, we spread them into 10 clients using Dirichlet distribution with 1. For Fed-ISIC-2019 dataset. we follow the original splits as in du Terrail al. (34], and let each client contribute 50 data samples to constitute X*
5% of the whole dataset to constitute X where each class has
We employ fixed for all experiments. To select X, randomly sample 10% of the full Cifar10 dataset, which we then split into local training data (95%) and X* (5%). The local training data is then spread into 10 clients using Dirichlet distribution with a The test global accuracy and value of À is plotted out in Fig. g We thus choose X = 0.5 for all our experiments, and it is experiments.
always able give stable performances according
accuracy on X
0.45
0.40
0.35
0.30
0.25
0.01
0.1
0.5
À
Figure 8: À versus algorithm performance
21

---

G.2 Limitations of the work
The main limitation of this work is the requirement of an extra shared unlabelled dataset, like in other knowledge distillation-basee decentralized learning works. Moreover, each agent needs to calculate their trust towards all other nodes locally. The extra computational complexity O(N2 n x C) per global round, where N stands for the number of agents, n' stands for the size of the shared dataset and C denotes the number of classes. The computation can be heavy if the number of clients gets large. But as we focus on cross-silo setting, N usually does not tend to be a big number.
22


Reviews:
[{"review_id": "Y493Mqeqtn", "reviewer": "Reviewer_vDF7", "summary": "In this paper, the authors consider a collaborative learning setting where agents want to improve their predictive performance on a shared target domain. The paper proposes a novel algorithm based on prediction-consensus, which effectively addresses statistical and model heterogeneity in the learning process. The algorithm works by having agents pseudo-label data from the target domain. The pseudo-labels are then used to compute a trust weighting scheme, which determines how much each agent's opinion should be weighted when reaching a consensus on how to label the unlabeled data. Theoretical results that show consensus can be reached via the algorithm and justify the conditions for good consensus to be achieved. Overall, the paper is a significant contribution to the field of collaborative learning. The proposed algorithm is a promising approach for improving the predictive performance of individual models in the presence of heterogeneity.\n", "questions": "* Assumption 1 states that there is no concept shift between the local data distributions. However, it seems to be a quite strong assumption. If there is no concept shift, then the problem of collaborative learning becomes much easier.\n* Depending on the choice of lambda, is it possible that the algorithm converges to a point where although the local models reach consensus, the consensus is not ideal?\n", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper proposes a novel algorithm for collaborative learning that is based on prediction-consensus.\n* The paper is well-written and easy to follow. The authors do a good job of explaining the motivation for the work, the proposed algorithm, and the experimental results.\n* The theoretical results in the paper are sound. The authors provide a rigorous analysis of the proposed algorithm and show that it can reach consensus under certain conditions.\n", "weaknesses": "* The experimental results in the paper are somewhat weak. The authors only report results on a few datasets and with a few model architectures. And the datasets seem over-simplified. It would be helpful to see more experimental results on a wider variety of datasets and with a wider variety of model architectures. This would help to give a better sense of the generalizability of the results.\n* Again on the experimental results, it would be great to study the effects of the hyperparameter lambda and distance function D. It would be helpful to see how the performance of the algorithm is affected by different hyperparameter settings. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing helpful feedback, which we genuinely appreciate.\n\nRegarding the comment concerning the over-simplicity of our chosen datasets, it is important to highlight that we have, in fact, incorporated a challenging dataset known as Fed-ISIC-2019 [30, 31, 32] from a real-world use case. This dataset comprises dermoscopic images collected from six different hospitals, thereby adding complexity to the experiments. Additionally, we employed the Cifar10/100 datasets with Dirichlet distributed splits, which are standard in collaborative learning experiments. The consistent performance observed on these datasets serves to demonstrate the potential practical benefits of our algorithm. \n\n**Q1**: In the collaborative learning setting, statistical heterogeneity remains an issue even without concept shift [5,6]. This arises due to the non-IID nature of data distributed across agents, causing each agent's local objective to deviate from the global one. Consequently, the averaged federated model may deviate from the global optima. Our algorithm effectively addresses scenarios with highly non-IID data. Furthermore, beyond concept shift, the presence of low-quality nodes (nodes with bad-quality data or weak model architectures) can impede the learning process, and our algorithm is specifically designed to address this concern.\n\n**Q2**: Yes, one could imagine with a large enough $\\lambda$, such that every agent’s goal is just to reach consensus, then any consensus could be a minimizer to the optimization problem. We showed a plot of accuracy on $X_S$ versus the choice of $\\lambda$ in the appendix (see Figure 8). We found that accuracy increases and then decreases again as $\\lambda$ is increased and the optimal value is around 0.5. We use this value for our experiment without further hyperparameter tuning. \n"}, {"review_id": "PZdM9f8q3X", "reviewer": "Reviewer_u6rQ", "summary": "The paper proposes a collaborative learning approach that leverages unlabeled auxiliary data to improve individual models through consensus. The trust weighting scheme adapts to each collaborator's influence, leading to a consensus on how to label the auxiliary data. The authors demonstrate that this collaboration scheme significantly boosts individual model performance and effectively mitigates the negative impact of bad models on the collective. Overall, the paper makes a valuable contribution to the field of collaborative learning and presents a promising approach for improving model performance through consensus.", "questions": "- How does the proposed approach scale to larger datasets and more complex model architectures, and can the authors provide more insights into the computational complexity of the algorithm?", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a novel approach for collaborative learning through consensus on unlabeled auxiliary data.\n- The collaboration scheme significantly boosts individual model performance and mitigates the negative impact of bad models on the collective.\n- The paper provides a thorough description of the algorithm and its implementation.\n- The experimental results demonstrate the effectiveness of the proposed approach.", "weaknesses": "- The paper could benefit from a more detailed analysis of the impact of different trust weighting schemes on the consensus.\n- The paper could benefit from a more detailed discussion of the assumptions and limitations of the trust-based iterative pseudo-labeling process.\n- The paper could provide more insights into the computational complexity of the proposed algorithm and potential scalability issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and helpful feedback. \nWe address your question regarding computational complexity of our method in the following:\n\n\nCompared to typical decentralized algorithms where model params/gradients are communicated, our method introduced extra computational complexity on the calculation of pairwise trust scores and the computation brought by including the augmented dataset ($X_S$ and the corresponding pseudolabels) into training. The first one is matrix calculation and can be made efficient on GPU, and the extra computing time mainly comes from the augmented dataset, which is 1.35x longer per global round in our experiments with one V100 GPU. We believe this would less be an issue with more computing resources.\n\n\nHowever, we want to emphasize that our method greatly reduced communication complexity, which is a bottleneck in decentralized training. “In federated optimization communication costs dominate — we will typically be limited by an upload bandwidth of 1 MB/s or less” [3]. Conventional collaborative learning algorithm communicates model params/gradients, which come at size of millions, and the communication complexity per global round is $\\mathcal{O}(N \\times |params|)$. While we communicate model predictions. The complexity is $\\mathcal{O}(n_S \\times C \\times N^2)$ for each global round, where $N$ stands for the number of agents, $n_S$ stands for the size of the shared dataset and $C$ denotes the number of classes. It is clear that this value does not scale up with more complex models, and is much smaller than the model size.\n\n\n\n\n"}, {"review_id": "CQjLqNMF8u", "reviewer": "Reviewer_TUxA", "summary": "This paper proposes a collaborative learning method that leverages unlabeled auxiliary data to facilitate the exchange of expertise among agents. The method adaptively weights the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. The authors demonstrate that their collaboration scheme significantly boosts individual model's performance with respect to the global distribution, compared to local training. They also show that their method is particularly effective in scenarios where the intrinsic beliefs of individuals counterbalance the averaging process and yield a diversity of opinions.", "questions": "1. Can you provide more details on the computational complexity of the proposed method? How does it scale to large-scale datasets or complex models?\n2. How does the proposed method compare to other state-of-the-art collaborative learning methods in terms of performance? Can you provide a comprehensive comparison?\n3. How robust is the trust weighting scheme to malicious agents? Have you considered scenarios where some agents may intentionally provide incorrect information?\n4. Have you tested the proposed method on real-world datasets? If so, can you provide some examples and discuss the results?\n5. How sensitive is the proposed method to the choice of hyperparameters? Have you conducted a sensitivity analysis to assess the impact of different hyperparameters on performance?", "limitations": "The paper briefly mentions some potential limitations of the proposed method, such as the lack of a comprehensive comparison with other state-of-the-art methods and the assumption of trustworthy and non-malicious agents. However, the authors do not provide a detailed discussion of these limitations or potential negative impacts of their work. While the paper does provide some insights into the strengths and weaknesses of the proposed method, a more thorough analysis of the limitations and potential negative impacts would be desirable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed approach utilizes unlabeled auxiliary data to enhance the exchange of expertise among agents, resulting in a significant improvement in individual model performance when compared to local training with respect to the global distribution.\n2. The method dynamically assigns weights to each collaborator's influence on the pseudo-labels, iteratively reaching a consensus on how to label the auxiliary data. This adaptive weighting effectively detects and mitigates the negative impact of poor models on the collective performance.\n3. The authors demonstrate the method's efficacy, particularly in scenarios where individuals hold diverse beliefs that counterbalance the averaging process. This diversity of opinions leads to improved performance in heterogeneous environments, showcasing the method's potential in such situations.", "weaknesses": "1. The paper lacks a comprehensive comparison with other recent state-of-the-art collaborative learning methods, making it challenging to evaluate the relative performance of the proposed method. Including such comparisons would enhance the understanding of its strengths and weaknesses.\n2. The paper would benefit from a more in-depth analysis of the computational complexity of the proposed method. This analysis would shed light on its scalability to large-scale datasets or complex models, which is crucial for practical implementation.\n3. The paper assumes the trustworthiness and non-malicious nature of all agents, which may not be realistic in real-world scenarios. To address this limitation, a more robust trust weighting scheme capable of handling malicious agents should be considered, ensuring the method's applicability in diverse environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and insightful questions. We address the raised questions below:\n\n\n**Q1**: Compared to typical decentralized algorithms where model params/gradients are communicated, our method introduced extra computational complexity on the calculation of pairwise trust scores and the computation brought by including the augmented dataset into training. The first one is matrix calculation which can be made efficient on GPU, and the extra computing time mainly comes from the augmented dataset, which is 1.4x longer in our experiments with one V100 GPU. \n\n\nHowever, we want to emphasize that our method _greatly reduced communication complexity_, which is the main bottleneck in decentralized training. “In federated optimization communication costs dominate — we will typically be limited by an upload bandwidth of 1 MB/s or less” [3]. Conventional collaborative learning algorithm communicates model params/gradients, which come at a size of millions or billions, and the communication complexity per global round is $\\mathcal{O}(N \\times |params|)$. While we communicate model predictions. The complexity is $\\mathcal{O}(n_S \\times C \\times N^2)$ for each global round, where $N$ stands for the number of agents, $n_S$ stands for the size of the shared dataset and $C$ denotes the number of classes. It is clear that this value does not scale up with more complex models, and is much smaller than the model size.\n\n\n**Q2**: Based on your comment we further included two SOTA (and classic) methods that were designed to address statistical heterogeneity in federated learning: FedDyn [R1] and SCAFFOLD [R2]. SCAFFOLD uses variance reduction to correct for the `client-drift' in its local updates, and FedDyn designs a dynamic regularization term to ensure the alignment of global and device solutions. In our first scenario where the same model architecture is applied, we did a more comprehensive comparison. Please note that the training loss of SCAFFOLD and FedDyn gets saturated when the number of local epochs is set to 5, which is our default setting. Instead, we did a quick hyperparameter search, and found the optimal numbers of local epochs for SCAFFOLD and FedDyn are 1 and 2 respectively, and report the corresponding accuracy on our target dataset. For all the other methods, we set the number of local epochs to 5 as reported in the paper. Our proposed method still occupies the top 1 accuracy in most of the cases. Please refer to Table 1 in the **global response PDF file**.\n\n**Q3**: In our paper we design the algorithm under the assumption that every agent communicates honestly, that is, no Byzantine workers that send intentionally incorrect information are involved. However, our method does exhibit some robustness against a typical Byzantine attack, which is label flip (we call workers with flipped labels low-quality workers in the paper). With 2 out of 10 workers having 100% flipped labels, we did not witness a big performance drop. \n\nAdditionally, here are some of our thoughts regarding your question: If malicious workers that send intentionally incorrect information are involved, then the nodes might reject to reach consensus, instead of reaching a detrimental bad consensus, assuming a reasonable $\\lambda$ is chosen. Imagine if a bad consensus is reached with malicious nodes involved, then for the regular nodes, the consensus loss and the local loss will not decrease in the same direction, and thus the consensus solution is not a stationary solution. Indeed, the \"personal\" part of our loss adds some degree of robustness to malicious nodes.\n\nWe agree that it would be interesting to investigate robustness against _intentionally_ malicious nodes more.  However, we would like to keep the focus of our work on the new protocol of information exchange, and leave robustness extensions for future work. \n\n\n**Q4**: Yes, Fed-ISIC-2019 [30, 31, 32] is a real-world dataset from the healthcare domain. It is a dermoscopic dataset collected from 6 different hospitals, which is a benchmark dataset from Flamby paper [33]. Our method shows consistent success in the classification task. Please further refer to Figure 1 and Figure 3(c) for more details.\n\n**Q5**: We offer sensitivity analysis of how the performance scales with $\\lambda$ and the number of local epochs. Accuracy in the target domain versus different choices of hyperparameter $\\lambda$ is plotted in Figure 8 in the appendix. And the influence of the number of local epochs is also presented in Figure 6 in the main text. Is there another study you would be interested in?\n\n\n[R1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, Venkatesh Saligrama. _Federated Learning Based on Dynamic Regularization_. ICLR 2021\n\n[R2] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, Ananda Theertha Suresh. _SCAFFOLD: Stochastic Controlled Averaging for Federated Learning_. ICML 2020\n\n"}, {"review_id": "C7DbJPHLI4", "reviewer": "Reviewer_Bri9", "summary": "To facilitate information exchange among agents and improve prediction accuracy on a shared target domain, this paper proposes a decentralized learning algorithm based on prediction consensus inspired from social science for leveraging each agent’s predictions. To be concrete, each agent firstly shares its prediction and weighs these predictions according to the trust scores, then these proxy labels are used to augment local model training via distillation. The authors show the trust measure based on cosine similarity can attain trust matrix with ideal properties, which could facilitate effective consensus. The efficacy is empirically demonstrated by showing that the proposed method is better than classical baselines under some heterogeneous conditions.", "questions": "please see weaknesses", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper considers a collaborative learning setting where each agent owns their own data and agents want to collaborate to improve their predictive performance over a target domain. The authors propose a decentralized algorithm based on distillation and trust weighting scheme, which is useful in cases where data sharing and model sharing are not allowed.  The key ingredient, which is called trust, is inspired from social science, which is an interesting topic. The theoretical analysis seems sound although I didn’t check all the details. The authors also conduct sufficient experiments to demonstrate the effectiveness of the proposed method. Overall, the paper is a well-written. ", "weaknesses": "In the collaborative setting, each agent owns their own data and wants to improve its predictive accuracy while keeping its data and model private, which is a significant problem. However, exchanging predictions may leak valuable information. Is there a more private way to share data between agents? \nThe goal of agents is to improve the prediction performance on a shared target domain.  Is it possible that ensemble algorithms like bagging work better here?  I think the motivation and necessity of collective prediction could be improved.  \nI didn’t clearly figure out why the proposed method could keep communication at a minimum as claimed in the paper. Could the authors give more detailed explanations?\nThe explanation of co-training in line 144 is not that convincing. As far as I know, dispersed and heterogeneous data on multiple agents cannot be simply viewed as multi-view data. Could the author give more explanations?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your helpful comments and feedback. Regarding the weakness pointed out by the reviewer, we offer the subsequent clarifications:\n\n**W1 Information leakage**: This is an interesting question. As you point out, exchanging predictions may still leak sensitive information about the model’s training data. However, compared to the conventional ways in collaborative learning, such as sharing model parameters or gradients, predictions contain much less information. Further our scheme, due to the information exchange through model queries, renders itself amenable to applying standard query perturbation approaches to guarantee differential privacy. Crucially, the number of queries each agent is willing to answer is entirely in their own control, which poses a significant advantage. We will add a discussion of privacy considerations. In the worst case privacy constraints would limit the number of outer rounds that can be performed (we further attached a plot of the number of outer rounds versus accuracy in the **global response PDF**). Studying this tradeoff between privacy and the progress we can make through information exchange more formally could be an interesting extension for future work. \n\n**W2  Comparison to other ensemble algorithms**: Ensemble algorithms like bagging would put equal weights on each agent’s predictions, which is equivalent to our naive trust scheme. This equal weighting will suffer from low-quality agents that possess weak models or bad-quality data, please refer to Figure 2 and Table 1 for supporting experimental results. This also motivates the design of our collaborative learning algorithm that is able to benefit from each agent’s local expertise and can learn to upweight high-quality agents and downweight low-quality agents in the consensus.\n\n**W3  Minimum communication costs**: Regarding our claim that “the proposed method could keep communication at a minimum”, we position ourselves in the modern deep learning regime where the size of model parameters comes in millions or billions. Compared to exchange model parameters/gradients, which induces communication complexity per global round $\\mathcal{O}(N \\times |params|)$, exchanging model predictions will induce communication cost $\\mathcal{O}(N^2 \\times n_S \\times C)$, where $N$ stands for the number of agents, $n_S$ stands for the size of the shared dataset and $C$ denotes the number of classes. The latter is significantly smaller. We will make the meaning of the claim more precise in this sense.\n\n**W4 Multi-view explanation**: Thanks for pointing this out. In our setting, we are not talking about combining models with different feature views. Instead, each agent contributes their knowledge covering different regions of the target domain, and thus combining their knowledge wisely leads to a better model over the full region of the target domain. We will make sure this is made more explicit.\n\n\n"}]

Author rebuttal:
We thank all the reviewers for their time and effort spent in giving comments and feedbacks on the paper, and the ACs for the help in the reviewing process. 

In addition to the separate responses, we further added a table and a graph in our global response PDF to better support our arguments experimentally. We kindly request your consideration of these additions for enhanced clarity and understanding.


Meta-review: The paper studies collaborative learning via prediction-consensus on auxiliary data. All the reviewers appreciate the novelty of the work and  the theoretical results. Reviewers asked for stronger empirical comparisons to state-of-the-art, more in-depth discussion on computational complexity, potential privacy concerns. Please consider address them in your manuscript.
Decision: Accept (poster)