SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongyang Fan<sup>1</sup> Celestine Mendler-Dünner<sup>2,3</sup> Martin Jaggi<sup>1</sup>  
<sup>1</sup>EPFL, Switzerland  
<sup>2</sup>Max Planck Institute for Intelligent Systems, Tübingen, Germany  
<sup>3</sup>ELLIS Institute Tübingen, Germany  
{dongyang.fan, martin.jaggi}@epfl.ch  
celestine@tue.ellis.eu
We consider a collaborative learning setting where the goal of each agent is to improve their own model by leveraging the expertise of collaborators, in addition to their own training data. To facilitate the exchange of expertise among agents, we propose a distillation-based method leveraging shared unlabeled auxiliary data, which is pseudo-labeled by the collective. Central to our method is a trust weighting scheme that serves to adaptively weigh the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. We demonstrate empirically that our collaboration scheme is able to significantly boost individual models' performance in the target domain from which the auxiliary data is sampled. At the same time, it can provably mitigate the negative impact of bad models on the collective. By design, our method adeptly accommodates heterogeneity in model architectures and substantially reduces communication overhead compared to typical collaborative learning methods.
This work considers a decentralized learning setting where each agent has access to a labeled dataset and a local model. The agents may differ in the data distribution they have access to as well as the quality of their local models. In addition, we assume a shared unlabeled dataset  $X^{*}$  sampled from a target distribution  $\mathcal{Q}$  is available to all agents. The central question is how can agents effectively exchange information to benefit from each other's local expertise in order to improve their predictive performance on the target domain  $\mathcal{Q}$ ?
Towards this goal, our work takes inspiration from social science on how a panel of human experts collaborate on a task. Humans typically engage in discourse to exchange information, they share their opinions, and based on how much they trust their peers, each individual will then adjust their subjective belief towards the opinion of peers. When repeated, this process gives rise to a dynamic process of consensus finding, as formalized by DeGroot [1]. Central to the consensus mechanism of DeGroot is the concept of trust. It determines how much individual agents influence each other's opinion, and thus the influence of each agent on the final consensus.
Our proposed algorithm mimics this consensus-finding mechanism in the context of collaborative learning, inspired by [2]. In particular, our consensus procedure is aimed at how to label the shared dataset  $\mathbf{X}^{*}$ . Therefore, we carefully design a strategy by which each agent determines the trust towards others, given its local information, to optimally leverage each agent's expertise to collectively pseudo-label  $\mathbf{X}^{*}$ . This mechanism of knowledge distillation is then combined with techniques from self-training [3] in order to transfer the shared knowledge from the pseudo-labels into the local models in an iterative fashion.
Code available at https://github.com/fan1dy/collaboration-consensus
Crucial to our approach is that instead of aiming for a consensus on model parameters, such as typically done in federated learning, we leverage the abundance of unlabeled data to enforce consensus in prediction space over  $\mathbf{X}^*$ . A key benefit of information exchange in prediction space is a reduction in communication complexity and the ability to control privacy leakage, but also the ability to seamlessly cope with both data heterogeneity and model heterogeneity.
Problem setup. We consider a collaborative learning setup with  $N$  agents. Each agent  $i \in [N]$  holds local training data, sampled from a local data distribution  $\mathcal{P}_i$ . We denote the local training data as  $(X_i, y_i)$ , where the matrix  $X_i$  composes of  $n_i$  local datapoints and  $y_i$  denotes the vector of corresponding labels. The number  $n_i$  needs not to be the same across agents. In addition, we assume agents have access to a shared unlabeled dataset  $X^*$  sampled from a target distribution  $\mathcal{Q}$ . We use  $n^*$  to denote the number of datapoints in  $X^*$ . The ultimate success measure we consider is each agent's prediction performance on the target distribution  $\mathcal{Q}$ . We work under the assumption that sharing of raw labeled data is not desirable due to privacy concerns, data ownership, or storage constraints and we wish to keep communication at a minimum to avoid overheads. Our setting recovers the goal of both decentralized and federated learning, with or without personalization [4], when  $\mathcal{Q}$  is defined as the prediction task on the mixture of all local data distributions. In addition, we allow agents to differ in their model architectures. We do not require a coordination server but assume agents have all-to-all communication available.
Contributions. The key contributions of our work can be summarized in the following aspects:
1. We propose a novel collaborative learning algorithm based on prediction-consensus, which effectively addresses statistical and model heterogeneity in the learning process, and provably mitigates the negative impact of low-quality participating agents.
2. Our algorithm is able to significantly reduce communication overhead and privacy-sensitive information sharing in comparison to other collaborative learning methods while achieving superior empirical performance.
3. We show theoretically that consensus can be reached via our algorithm and justify the conditions for good consensus to be achieved.
In federated learning a central server coordinates local updates toward learning a global model. Local nodes upload gradients or model parameters, instead of data itself, to maintain a certain level of privacy. McMahan et al. [5] describe the classic FedAvg algorithm. Follow-up works mainly focus on addressing challenges from non-i.i.d. local data [6, 7, 8] and robustness towards Byzantine attacks. Apart from communicating gradients or model parameters, several works discuss alternatives to allow for heterogeneous model architectures. These methods are based on variants of model distillation [9, 10], reaching an agreement in the representations space [11] or output space [12]. Similar to our work, both assume access to a shared unlabeled dataset. However, we go beyond naive averaging to determine agreement to account for heterogeneity in model or data quality.
In contrast to federated learning, the fully decentralized learning setting does not assume the existence of a central server. Instead, decentralized schemes such as gossip averaging are used to aggregate local information across agents [13, 14]. Despite the lack of a global state, such methods can provably converge to the desired global solution, leading to a gradual consensus among individual models [4]. In this context, Dandi et al. [15], Le Bars et al. [16] optimizes the communication topology to adapt to data heterogeneity but do not offer any collaborator selection mechanisms. Bellet et al. [13] allows personalized models on each agent, but assumes prior information about task-relatedness, as opposed to learned selection. Gossip algorithms typically assume a fixed gossip mixing matrix given by e.g. physical connections of nodes [17, 18, 19]. These approaches fail to consider data-dependent communication as with task similarities and node qualities. Several recent works have addressed this issue by proposing alternative methods that consider these factors. Notably, Li et al. [20] directly optimizes the mixing weights by minimizing the local validation loss per node, which requires labeled validation sets. Sui et al. [21] uses the E-step of EM algorithm to estimate the importance of other agents to one specific agent  $i$ , by evaluating the accuracy of other agents' models on the local data of agent  $i$ . This way of trust computation does not allow the algorithm to be applied to target distributions that differ from the local distribution, differentiating it from our work. Moreover, for
both Li et al. [20], Sui et al. [21] the aggregation is performed in the gradient space, therefore not allowing heterogeneous models.
Our work relates to semi-supervised learning as it involves partially unlabeled data. Most relevant are self-training methods [22] that first train a model using labeled data, then use the trained model to give pseudo-labels to unlabeled data. The pseudo-labels can further be fed back to the training loop to attain a better model. Wei et al. [23] shows that under expansion and separation assumptions, self-training with input consistency regularization can achieve high accuracy with respect to ground-truth labels. When more than one learner is involved, co-training [24] appears as an extension to self-training, benefiting from the knowledge of learners from independent views in labeling a set of unlabeled data. Diao et al. [25] incorporates SSL into federated learning. In a setting where agents are with unlabeled data and the center server is with labeled data, experimental studies demonstrate that the performance of a labeled server is significantly improved with unlabeled clients. Farina [26] presented a collective learning framework for distributed SSL, where they combine predictions on a shared dataset via weights evaluated from local models' performances on local validation datasets. While their algorithm bears similarities to ours, it is important to note that it is exclusively tailored to scenarios in which the target domain matches the global distribution. In a similar spirit, we want to leverage unlabeled data in a fully decentralized setting.
Finally, Mendler-Dünner et al. [2] have previously formalized collective prediction as a dynamic consensus finding procedure. They demonstrated that such an approach can lead to significant gains over naive model averaging. We extend their approach from test-time prediction to collaborative model training.
3 Method description
Our proposed method is designed to take advantage of shared unlabeled data in the context of collaborative learning through knowledge distillation. Therefore, it emulates human opinion dynamics to collectively pseudo-label the shared auxiliary data. These labels are then incorporated in the local model update steps towards collectively improving the performance on the data distribution from which the shared data is sampled.
3.1 Collective pseudo-labeling
To describe the pseudo labeling step, let us use  $f_{\theta_i}$  to denote the local model of agent  $i \in [N]$  parameterized by  $\theta_i$ . We write  $\hat{\pmb{y}}_i = f_{\theta_i}(\pmb{X}^*)$  to denote the predictions of agent  $i$  on the auxiliary data  $\pmb{X}^*$ . Agents share these predictions with their peers. Naturally, the individual models may differ in these predictions and it is a priori unclear which model is most accurate, as ground truth labels of the auxiliary data are not available. To combine the predictions into pseudo-labels for  $\pmb{X}^*$ , each agent locally decides how to weigh other agents' predictions by estimating their respective expertise on the target task. We refer to these weights as trust scores and we use  $w_{ij}$  to denote the trust of agent  $i$  towards the predictions of agent  $j$ . It's worth noting that the trust between agents is not necessarily mutual, i.e., can be asymmetrical: agent  $i$  can trust agent  $j$  without agent  $j$  necessarily trusting agent  $i$  back. We use  $W$  to denote the matrix of trust scores. Given the trust scores, agent  $i$  uses the following pseudo labels for the auxiliary data:
$$
\boldsymbol {\psi} _ {i} = \sum_ {j} w _ {i j} \hat {\boldsymbol {y}} _ {j}. \tag {1}
$$
Trust scores are determined locally by each agent based on query access to other agents' predictions and they are refined iteratively throughout training as models are being updated. The adaptive weight computation will be detailed in Section 4.
3.2 Collaborative learning from pseudo labels
In the second step, the proxy labels for the auxiliary data are used to augment local model training. Therefore, in each step, the local optimization problem is augmented by a disagreement loss, and the new objective is given by
$$
\mathcal {L} \left(f _ {\boldsymbol {\theta} _ {i}} \left(\boldsymbol {X} _ {i}\right), \boldsymbol {y} _ {i}\right) + \lambda \operatorname {d i s t} \left(f _ {\boldsymbol {\theta} _ {i}} \left(\boldsymbol {X} ^ {*}\right), \boldsymbol {\psi} _ {i}\right) \tag {2}
$$
Algorithm 1 Pseudo code of our proposed algorithm
Input: For each agent  $i \in [N]$  we are given a local model  $\pmb{\theta}_i^{(0)}$ , a labeled local dataset  $(X_i, y_i)$  and unlabeled shared data  $\pmb{X}^*$ .
Each node  $i \in [N]$  broadcasts their soft labels  $\hat{\pmb{y}}_i^{(t-1)} = f_{\pmb{\theta}_i^{(t-1)}}(\pmb{X}^*)$  to all other nodes in parallel for each agent  $i$  do
- Calculate pairwise trust score  $w_{ij}^{(t)} (j \in [N])$ , based on the received soft decisions using methods provided in Section 4
- Get pseudo-labels on  $X^{*}$  from collaborators:  $\psi_{i}^{(t)} = \sum_{j} w_{ij}^{(t)} \hat{y}_{j}^{(t - 1)}$
- Do local training with collaborative disagreement loss
$$
\boldsymbol {\theta} _ {i} ^ {(t)} \in \underset {\boldsymbol {\theta}} {\arg \min } \mathcal {L} \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {X} _ {i}\right), \boldsymbol {y} _ {i}\right) + \lambda \operatorname {d i s t} \left(f _ {\boldsymbol {\theta}} \left(\boldsymbol {X} ^ {*}\right), \psi_ {i} ^ {(t)}\right) \tag {3}
$$
where  $f_{\theta_i}(\mathbf{X})$  denotes the vector of agent  $i$ 's predictions on the dataset  $\mathbf{X}$ ,  $\mathcal{L}$  is the local training loss and  $\mathrm{dist}(\cdot)$  is a disagreement measure. We choose  $l_2$  distance for the disagreement measure in the regression case and cross-entropy for the classification case.  $\lambda > 0$  is a trade-off hyperparameter that weighs the local loss and the cost of disagreement. This objective adheres to a conventional semi-supervised learning approach, however, we generate pseudo-labels in a trust-based collective manner.
To iteratively refine the local models in the spirit of self-training, the pseudo labeling step and the local training step are performed in an alternating fashion as described in Algorithm 1. Starting from pre-trained models  $\pmb{\theta}_i^{(0)}$ , in each round  $t \in \{1,..,T\}$  model predictions on the auxiliary data are shared and then each agent aggregates them into a set of pseudo labels to augment local data and perform an update step.
Our algorithm is motivated conceptually by co-training [24] where it was demonstrated that unlabeled data can be used to augment labeled data to boost model performance. Moreover, learning from collective pseudo-labels offers several additional benefits. Firstly, aggregation is performed in the prediction space, eliminating the need for all agents to have identical model architectures. Secondly, the communication cost of transmitting predictions is significantly lower than that of sharing model weights, and the same pseudo-labels  $\psi_{i}$  can be reused for multiple local epochs to further reduce the communication burden.
3.3 Convergence analysis
We study under what conditions Algorithm 1 will reach a consensus among agents on how to label the auxiliary data. For the analysis, we focus on the over-parameterized regime and we make the following assumption on the local data distributions:
Assumption 1. There is no concept shift between the local data distributions and the target domain  $\mathcal{Q}$  from which the shared data is sampled, i.e.,  $\mathcal{P}_i(Y|X = x) = \mathcal{Q}(Y|X = x)$  for all  $i\in [N]$ .
Together with over-parameterization the assumption implies that the minimizer of the objective specified in (2) can always reach zero loss. Further, this allows us to model the update of agents' predictions on  $\mathbf{X}^*$  as a Markov process where the state transition matrix corresponds to the trust matrix  $\mathbf{W}^{(t)}$ . Therefore, it is convenient to write the update of the predictions on  $\mathbf{X}^*$  performed by the algorithm in matrix form, as  $\Psi^{(t)} = [\hat{\pmb{y}}_1^{(t)},\dots,\hat{\pmb{y}}_N^{(t)}]$ . Adopting this notation we have for  $t\geq 1$
$$
\Psi^ {(t)} = \boldsymbol {W} ^ {(t)} \Psi^ {(t - 1)} = \boldsymbol {W} ^ {(t)} \boldsymbol {W} ^ {(t - 1)} \dots \boldsymbol {W} ^ {(1)} \Psi^ {(0)}. \tag {4}
$$
The following result provides sufficient conditions under which consensus will be reached by our algorithm.
<sup>1</sup>We say a model is over-parameterized if its training error can reach zero. Over-parameterization is a reasonable assumption in the deep learning regime.
Theorem 1 (Consensus on predictions). Assume all agents' models are over-parameterized and the data distributions satisfy Assumption 1. Then, for  $t \to \infty$  Algorithm 1 converges to a consensus among the local models on the predictions on  $X^{*}$ , that is,
$$
\psi_ {i} ^ {(t)} = \psi_ {j} ^ {(t)} \quad \forall i \neq j, \tag {5}
$$
as long as  $\pmb{W}^{(t)}$  is row-stochastic and positive for any  $t\geq 0$
The proof is given in Appendix B and the main insight is that as long as  $\mathbf{W}$  is row-stochastic and positive (that is  $\sum_{j} w_{ij} = 1$  for any row  $i$ , and  $w_{ij} > 0$ ), the product of any  $\mathbf{W}^{(t)}$ 's is stochastic, irreducible, and aperiodic, and this leads to the differences between rows in  $\Psi^{(t)}$  vanishing in time. Together with no concept shift, over-parameterization is important to guarantee that models can fit the consensus predictions on the unlabeled data while at the same time minimizing local losses. In contrast, in the under-parameterized setting, consensus and local loss minimization cannot necessarily be achieved at the same time.
3.4 Information sharing in prediction space
A key feature of our method is that agents do not share model parameters, but they communicate by exchanging prediction queries. If Algorithm 1 achieves a consensus this means that agents arrive at solutions where they agree on predictions on  $X^{*}$ , but this does not imply that they have learned the same model, or that they agree on predictions outside  $X^{*}$ . To illustrate this, we provide a simple example where local data are generated using cubic regression with additive i.i.d. noise in the output, as shown in Fig. 1. In this example, uniform weights are the ideal solution. We apply this optimal uniform trust, as the main purpose here is to illustrate the difference between information sharing in prediction space and parameter space. Each agent fits a polynomial regression of degree 4, which leads to over-parameterization of the model to fit the data. Full details of our example are given in Appendix A.1. We refer to the work of [2] for a similar setting with under-parameterized models. Here we note the most interesting observations in the over-parameterized regime.
(b)
Figure 1: Local data distributions are shown in (a), and the initial fit on local data is shown in (b). (c) and (d) are predictions on  $X^{*}$  after 5 rounds and 20 rounds of our algorithm update respectively. (e) is the comparison of model fits in a larger range ((d) is zoom-in of the rectangular area of (e))
First, we observe that for  $T \geq 20$  the three agents reach a consensus on the predictions of  $X^{*}$ . However, the model parameters are not the same across the agents, as depicted in the rightmost panel. Further, considering the properties of the algorithm across rounds and the predictions in different regions of the input space in more detail, the following desirable behaviors are observed:
a) In the region where agent  $i$  has more data, it fits the local data more accurately and it moves pseudo-labels closer to its own predictions.
b) In the region where agent  $i$  has no or little data, agent  $i$  only updates its model parameters to fit the pseudo-labels.
c) When local loss minimization and prediction consensus can be achieved at the same time, agents can arrive at models with a perfect agreement in the target prediction space.
4 Design of trust weights
In Section 3.3 we have shown that our algorithm is guaranteed to reach consensus on  $\mathbf{X}^*$  under weak assumptions on the trust matrix  $\mathbf{W}^{(t)}$ . In this section, we discuss how to design  $\mathbf{W}^{(t)}$  to encourage that the achieved consensus leads to a high-quality labeling of  $\mathbf{X}^*$ .
Therefore, we focus on multi-class classification. We let  $f_{\pmb{\theta}_i}(\pmb{x})$  denote the class probabilities obtained using model  $\pmb{\theta}_i$  for a datapoint  $\pmb{x}$ . We choose the cross-entropy measure  $\mathcal{H}(\cdot,\cdot)$  to define the agreement loss function  $\mathrm{dist}(\cdot,\cdot)$  in (3). If  $\pmb{X} = [\pmb{x}_1,\dots,\pmb{x}_n]^\top$ , then  $f_{\pmb{\theta}}(\pmb{X}) = [f_{\pmb{\theta}}(\pmb{x}_1),\dots,f_{\pmb{\theta}}(\pmb{x}_n)]^\top \in \mathbb{R}^{n\times C}$ , where  $n$  is the number of samples in  $\pmb{X}$  and  $C$  is the number of classes.
4.1 Trust evaluation through self-confidence
The quality of the local models could differ due to various factors, such as the amount of labeled data available during training, due to the expressivity of the local model, the training algorithm, or due to the relevance of the local data for the target task of labeling  $\mathcal{Q}$ . Thus, a desirable property of the consensus solution is that malicious agents, or agents with low-quality models contribute less to the pseudo-labeling than agents with better models.
Hadjicostis and Dominguez-Garcia [27] differentiate between malicious and non-malicious agents and they discuss the concept of trustworthy consensus, where only non-malicious agents contribute to the consensus. In contrast to prior work, we do not aim for trustworthy agents to contribute equally. Instead, we specifically want consensus to come from potentially unequal contribution of all agents, weighted according to their relevance. We allow for the trust matrix to be asymmetric. All agents determine trust from information given locally to the respective agent, which differs across agents. Central to any such strategy is that the capabilities of models on  $\mathcal{Q}$  can be estimated appropriately. In the following, we discuss a strategy of how to determine trust from local data and prediction queries to other models.
As no label information on  $X^{*}$  is available to evaluate trust, it is natural to use agents' own predictions on  $X^{*}$  as a local reference point. Then, each agent distributes their trust towards other agents based on the alignment of their predictions. We use weighted pairwise cosine similarity as a measure of alignment which motivates the following trust weight calculation:
$$
w _ {i j} ^ {(t)} = \frac {\gamma_ {i j} ^ {(t)}}{\sum_ {j} \gamma_ {i j} ^ {(t)}} \quad \text {w i t h} \quad \gamma_ {i j} ^ {(t)} = \frac {1}{n ^ {*}} \sum_ {\boldsymbol {x} \in \boldsymbol {X} ^ {*}} \beta_ {i} ^ {(t)} (\boldsymbol {x}) \frac {\left\langle f _ {\boldsymbol {\theta} _ {i} ^ {(t - 1)}} (\boldsymbol {x}) , f _ {\boldsymbol {\theta} _ {j} ^ {(t - 1)}} (\boldsymbol {x}) \right\rangle}{\| f _ {\boldsymbol {\theta} _ {i} ^ {(t - 1)}} (\boldsymbol {x}) \| _ {2} \| f _ {\boldsymbol {\theta} _ {j} ^ {(t - 1)}} (\boldsymbol {x}) \| _ {2}}. \tag {6}
$$
The inclusion of the weighting factor  $\beta_{i}^{(t)}(\pmb{x})$  and how to choose it will be discussed in Section 4.2.
Self-confident trust. Naturally, pairwise cosine similarity leads to a trust matrix that has diagonal entries being the highest value among each row. We call this property self-confident, as each agent trusts itself the most. We now demonstrate that this property is not particularly restrictive. Even if constraining trust matrix to be self-confident, it is still possible to design such a matrix that facilitates any consensus. The proof is given in Appendix D.
Proposition 2. For any given consensus distribution  $\pi$ , it is always possible to find a trust matrix  $W$  that leads to it, which is both row stochastic and self-confident.