SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In plain words, Definition E says that if for every pair of rows  $i_1$  and  $i_2$  in a matrix  $W$ , there exists a column  $j$  (which may depend on  $i_1$  and  $i_2$ ) such that  $w_{i_1j} > 0$  and  $w_{i_2j} > 0$ , then  $W$  is a scrambling matrix. It is easy to verify that a positive matrix is always a scrambling matrix.
Lemma 1 (Adaptation of Lemma 2 from Wolfowitz [40]). For any  $t$ ,
$$
\delta \left(\boldsymbol {W} ^ {(t)} \boldsymbol {W} ^ {(t - 1)} \dots \boldsymbol {W} ^ {(1)}\right) \leq \prod_ {i = 1} ^ {t} \lambda \left(\boldsymbol {W} ^ {(i)}\right) \tag {10}
$$
Lemma 1 states that multiplying with scrambling matrices will make the row differences smaller.  $tr(\boldsymbol{W}^{(t)}) = \sum_{i} w_{ii}^{(t)}$  represents the sum of self-confidence of all nodes. As every  $\boldsymbol{W}^{(t)}$  is positive, we have all  $\boldsymbol{W}^{(t)}$ 's scrambling. Thus, the differences between rows of  $\boldsymbol{W}^{(t)}\boldsymbol{W}^{(t-1)}\dots \boldsymbol{W}^{(1)}$  get smaller when  $t$  gets bigger.
As  $\psi_i^{(t)} = \sum_j[\pmb{W}^{(t)}\pmb{W}^{(t - 1)}.. \pmb{W}^{(1)}]_{ij}\pmb{\psi}_j^{(t - 1)}$ , we have the predictions on  $\pmb{X}^*$  given by all nodes get similar over time. According to our calculation of  $\pmb{W}^{(t)}$  in Equation (6), which is based on cosine
similarity between predictions, it follows that an agent's trust towards the others gets larger over time. That is,  $\sum_{j} w_{ij}^{(t+1)} \geq \sum_{j} w_{ij}^{(t)}$ . Since each row sums up to 1, we have  $w_{ii}^{(t+1)} \leq w_{ii}^{(t)}$ , for all  $i$ .
According to Theorem 1, we have  $\psi_i^{(t)} = \psi_j^{(t)}$  as  $t\to \infty$ , for any  $i$  and  $j$ . According to the calculation of  $\mathbf{W}$ , we have  $\mathbf{W}^{(t)}$  with equal entries when  $t$  reaches infinity.
D Proof of Proposition 2
Recall stationary distribution  $(\pi \in \mathbb{R}^{1\times N})$  of a Markov chain being
$$
\lim  _ {t \rightarrow \infty} \boldsymbol {W} ^ {(t)} \dots \boldsymbol {W} ^ {(1)} \rightarrow [ \boldsymbol {\pi} ^ {\top} \dots \boldsymbol {\pi} ^ {\top} ] ^ {\top} \tag {11}
$$
The proof follows from the construction of Metropolis chains given a stationary distribution. We will first give an example of how Metropolis chains work.
Example 2 (Metropolis chains [28]). Given stationary distribution  $\pi = [0.3, 0.3, 0.3, 0.1]$ , how could we construct a transition matrix that leads to the stationary distribution?
Suppose  $\Phi$  is a symmetric matrix, one can construct a Metropolis chain  $P$  as follows:
$$
p (x, y) = \left\{ \begin{array}{l l} \phi (x, y) \min  \left(1, \frac {\pi (y)}{\pi (x)}\right) & y \neq x \\ 1 - \sum_ {z \neq x} \phi (x, z) \min  \left(1, \frac {\pi (z)}{\pi (x)}\right) & y = x \end{array} \right. \tag {12}
$$
$$
C h o o s e \boldsymbol {\Phi} = \left[ \begin{array}{c c c c} 1 / 3 & 1 / 4 & 1 / 4 & 1 / 6 \\ 1 / 4 & 1 / 3 & 1 / 4 & 1 / 6 \\ 1 / 4 & 1 / 4 & 1 / 3 & 1 / 6 \\ 1 / 6 & 1 / 6 & 1 / 6 & 1 / 2 \end{array} \right], w e c o u l d g e t \boldsymbol {P} = \left[ \begin{array}{c c c c} 4 / 9 & 1 / 4 & 1 / 4 & 1 / 1 8 \\ 1 / 4 & 4 / 9 & 1 / 4 & 1 / 1 8 \\ 1 / 4 & 1 / 4 & 4 / 9 & 1 / 1 8 \\ 1 / 6 & 1 / 6 & 1 / 6 & 1 / 2 \end{array} \right].
$$
It can be verified that  $\pi$  is the stationary distribution of Markov chain with transition matrix  $P$ . If  $\Phi$  is not symmetric, we modify  $\frac{\pi(y)}{\pi(x)}$  to  $\frac{\pi(y)}{\pi(x)} \frac{\phi(y,x)}{\phi(x,y)}$ , and the results remain unchanged.
Following Example 2, choose  $\Phi$  to be any self-confident doubly stochastic matrix. For all  $x$ , choose  $\pmb{P}$  as calculated from (12), we have
$$
p (x, x) = 1 - \sum_ {z \neq x} \phi (x, z) \min  \left(1, \frac {\pi (z)}{\pi (x)}\right) \geq 1 - \sum_ {z \neq x} \phi (x, z) = \phi (x, x) \tag {13}
$$
we see that probability distribution among each row gets more concentrated on the diagonal entries in  $P$  than  $\Phi$ . As  $\Phi$  already has high diagonal values, the claim follows.
E Proof of Proposition 4
Proposition 4 states sufficient conditions for  $\mathbf{W}^{(t)}$ 's to have such that a low-quality node  $b$  is assigned lowest importance in  $\pi$ , i.e.  $\pi_b = \min_i \pi_i$ . From Equation (11),  $\pi$  comes from the product of trust matrices. We start from a product of two such matrices.
Proposition 7. For row-stochastic and positive matrices  $A$  and  $B$ , and  $C = AB$ , if in both  $A$  and  $B$ ,
i)  $j$ -th column has the lowest column sum,
ii)  $(i,j)$ -th entry being the lowest value in  $i$ -th row for  $i \neq j$
then we have  $j$ -th column remains the lowest column sum in matrix  $\mathbf{C}$  and  $(i,j)$ -th entry being the lowest value in  $i$ -th row of  $\mathbf{C}$  for  $i \neq j$ ,
Proof. Let  $C = AB$ , the column sum of column  $j$  of  $C$  can be expressed as:
$$
\begin{array}{l} \sum_ {i} c _ {i j} = \sum_ {i} \sum_ {k} a _ {i k} b _ {k j} \tag {14} \\ = \sum_ {k} (\sum_ {i} a _ {i k}) b _ {k j} \\ \end{array}
$$
for  $t \neq j$ , the column sum of  $C$  is
$$
\begin{array}{l} \sum_ {i} c _ {i t} = \sum_ {i} \sum_ {k} a _ {i k} b _ {k t} \\ = \sum_ {k} ^ {\tau} \left(\sum_ {i} ^ {\kappa} a _ {i k}\right) b _ {k t} \tag {15} \\ \end{array}
$$
We first show that  $j$ -th column remains the lowest column sum in  $\pmb{C}$ . For  $t \neq j$ :
$$
\begin{array}{l} \sum_ {i} c _ {i t} - \sum_ {i} c _ {i j} = \sum_ {k} (\sum_ {i} a _ {i k}) (b _ {k t} - b _ {k j}) \\ = \sum_ {k \neq j} \left(\sum_ {i} a _ {i k}\right) \left(b _ {k t} - b _ {k j}\right) + \left(\sum_ {i} a _ {i j}\right) \left(b _ {j t} - b _ {j j}\right) \\ \stackrel {(i)} {>} \sum_ {k \neq j} (\sum_ {i} a _ {i j}) (b _ {k t} - b _ {k j}) + (\sum_ {i} a _ {i j}) (b _ {j t} - b _ {j j}) \\ = \left(\sum_ {i} a _ {i j}\right) \left(\sum_ {k \neq j} \left(b _ {k t} - b _ {k j}\right) + \left(b _ {j t} - b _ {j j}\right)\right) \tag {16} \\ = \sum_ {i} a _ {i j} \left(\sum_ {k} b _ {k t} - \sum_ {k} b _ {k j}\right) \\ \begin{array}{c} (i i) \\ > 0 \end{array} \\ \end{array}
$$
(i) holds because for  $k \neq j$ ,  $b_{kt} - b_{kj} > 0$  and  $\sum_{i} a_{ij} < \sum_{i} a_{ik}$ .
(ii) holds because the  $j$ -th column has the lowest column sum in B
We then show that  $(i,j)$ -th entry remains the lowest value in  $i$ -th row of  $\pmb{C}$  for  $i \neq j$ . For  $t \neq j$ , we have
$$
c _ {i t} - c _ {i j} = \sum_ {k} a _ {i k} b _ {k t} - \sum_ {k} a _ {i k} b _ {k j}
$$
$$
\begin{array}{l} = \sum_ {k \neq j} a _ {i k} \left(b _ {k t} - b _ {k j}\right) + a _ {i j} \left(b _ {j t} - b _ {j j}\right) \\ > \sum_ {k \neq j} ^ {(i i i)} a _ {i j} \left(b _ {k t} - b _ {k j}\right) + a _ {i j} \left(b _ {j t} - b _ {j j}\right) \\ = a _ {i j} \left(\sum_ {k \neq j} \left(b _ {k t} - b _ {k j}\right) + \left(b _ {j t} - b _ {j j}\right)\right) \tag {17} \\ = a _ {i j} \left(\sum_ {k} b _ {k t} - \sum_ {k} b _ {k j}\right) \\ \end{array}
$$
$$
\begin{array}{c} (i v) \\ > 0 \end{array}
$$
(iii) holds since  $b_{kt} - b_{kj} > 0$  and  $a_{ik} > a_{ij}$  for  $i, k \neq j$ .
(iv) holds because  $\sum_{k}b_{kt} > \sum_{k}b_{kj}$
For time-inhomogeneous trust matrix, Assumptions 1 2 ensure the Markov chain update:  $\psi_i^{(t)} = \sum_j w_{ij}^{(t)}\psi_j^{(t - 1)}$ , which is followed by consensus as proven in Theorem 1. We see that  $b$ -th column remains the lowest column sum in the product  $W^{(\tau)}W^{(\tau -1)}\ldots W^{(1)}$ , by iteratively applying Proposition 7. For  $t\geq \tau$ , multiplying consensus with any row stochastic preserves the consensus. Thus, the  $b$ -th column will remain to be the smallest column in the consensus. For the time-homogenous case, as long as  $W$  holds the same properties, one can easily verify that the same result still holds. Thus, Proposition 4 is proved.
Extend to more than one node with low-quality data. For more than one low-quality node, what are the desired properties (sufficient conditions) for the transition (trust) matrices to have? It turns out that apart from the two conditions in a single low-quality node case, we need an extra assumption.
Proposition 8. Given Assumptions 1 and 2 that all agents are over-parameterized, let  $\mathcal{R}$  be the set of indices of regular nodes, and  $\mathcal{B}$  be the set of indices of low-quality nodes, if for  $t\leq \tau$ $\pmb{W}^{(t)}$  satisfies the following conditions:
i) any regular node's column sum is larger than any low-quality node's:  $\min_{r\in \mathcal{R}}\sum_i w_{ir}^{(t)} > \max_{b\in \mathcal{B}}\sum_i w_{ib}^{(t)}$
ii) the gap between the sum of trust from regular nodes towards any regular node  $r$  and low-quality node  $b$  is larger than the gap between low-quality node  $b$ 's self-confidence and its trust towards the regular node:  $\sum_{n \in \mathcal{R}} (w_{nr}^{(t)} - w_{nb}^{(t)}) > (w_{bb}^{(t)} - w_{br}^{(t)})$ ,
iii) any node's trust towards a regular node is bigger or equal than its trust towards a low-quality node other than itself: for any  $r \in \mathcal{R}$  and any  $b \in \mathcal{B}$ , we have  $w_{nr}^{(t)} \geq w_{nb}^{(t)}$  holds as long as  $n \neq b$ .
And after  $t > \tau$ ,  $W^{(t)} = 11^{\top}\frac{1}{N}$ . Then we have nodes in  $\mathcal{B}$  having a lower importance in the consensus than nodes in  $\mathcal{R}$ .
Proof. First, let us look at the multiplication of two such matrices when  $1 < t < \tau$ , for any  $r \in \mathcal{R}$  and  $b \in \mathcal{B}$ , we have conditions (1)(2)(3) remain to be true for the product  $\boldsymbol{W}^{(t)}\boldsymbol{W}^{(t-1)}$ . We will verify them one by one in the following part:
Verification of condition (1): any regular node's column sum is larger than any low-quality node's in  $\mathbf{W}^{(t)}\mathbf{W}^{(t - 1)}$ . For any  $r\in \mathcal{R}$  and any  $b\in \mathcal{B}$ , we have
$$
\begin{array}{l} \sum_ {i} \sum_ {n} w _ {i n} ^ {(t)} w _ {n r} ^ {(t - 1)} - \sum_ {i} \sum_ {n} w _ {i n} ^ {(t)} w _ {n b} ^ {(t - 1)} \\ = \sum_ {n} \left(\sum_ {i} w _ {i n} ^ {(t)}\right) \left(w _ {n r} ^ {(t - 1)} - w _ {n b} ^ {(t - 1)}\right) \\ = \sum_ {n \in \mathcal {R}} \left(\sum_ {i} w _ {i n} ^ {(t)}\right) \left(w _ {n r} ^ {(t - 1)} - w _ {n b} ^ {(t - 1)}\right) + \sum_ {n \in \mathcal {B} \backslash \{b \}} \left(\sum_ {i} w _ {i n} ^ {(t)}\right) \left(w _ {n r} ^ {(t - 1)} - w _ {n b} ^ {(t - 1)}\right) \\ + \left(\sum_ {i} w _ {i b} ^ {(t)}\right) \left(w _ {b r} ^ {(t - 1)} - w _ {b b} ^ {(t - 1)}\right) \\ > \sum_ {n \in \mathcal {R}} ^ {(i)} \left(\sum_ {i} w _ {i b} ^ {(t)}\right) \left(w _ {n r} ^ {(t - 1)} - w _ {n b} ^ {(t - 1)}\right) + \sum_ {i} w _ {i b} ^ {(t)} \left(w _ {b r} ^ {(t - 1)} - w _ {b b} ^ {(t - 1)}\right) \\ + \sum_ {n \in \mathcal {B} \backslash \{b \}} \left(\sum_ {i} w _ {i n} ^ {(t)}\right) \left(w _ {n r} ^ {(t - 1)} - w _ {n b} ^ {(t - 1)}\right) \\ = \left(\sum_ {i} w _ {i b} ^ {(t)}\right) \left(\sum_ {n \in \mathcal {R}} w _ {n r} ^ {(t - 1)} - \sum_ {n \in \mathcal {R}} w _ {n b} ^ {(t - 1)} + w _ {b r} ^ {(t - 1)} - w _ {b b} ^ {(t - 1)}\right) \\ + \sum_ {n \in \mathcal {B} \backslash \{b \}} \left(\sum_ {i} w _ {i n} ^ {(t)}\right) \left(w _ {n r} ^ {(t - 1)} - w _ {n b} ^ {(t - 1)}\right) \\ \begin{array}{c} (i i) \\ > 0 \end{array} \\ \end{array}
$$
(i) holds because  $\sum_{i} w_{in}^{(t)}$  for any  $n \in \mathcal{R}$  is larger than  $\sum_{i} w_{ib}^{(t)}$  for any  $b \in \mathcal{B}$ , which follows from condition (1), and  $w_{nr}^{(t)} - w_{nb}^{(t)} > 0$ , which follows from condition (3).
(ii) holds following the conditions (2) and (3). From (2),  $\sum_{n\in \mathcal{R}}w_{nr}^{(t - 1)} - \sum_{n\in \mathcal{R}}w_{nb}^{(t - 1)} + w_{br}^{(t - 1)} - w_{bb}^{(t - 1)} > 0$ , and from (3),  $w_{nr}^{(t - 1)}\geq w_{nb}^{(t - 1)}$  for  $n\neq b$
Verification of condition (2):
$$
\begin{array}{l} \sum_ {n \in \mathcal {R}} \left(\sum_ {p} w _ {n p} ^ {(t)} w _ {p r} ^ {(t - 1)} - \sum_ {p} w _ {n p} ^ {(t)} w _ {p b} ^ {(t - 1)}\right) - \left(\sum_ {p} w _ {b p} ^ {(t)} w _ {p b} ^ {(t - 1)} - \sum_ {p} w _ {b p} ^ {(t)} w _ {p r} ^ {(t - 1)}\right) \\ = \sum_ {p} \left(\sum_ {n \in \mathcal {R}} w _ {n p} ^ {(t)} + w _ {b p} ^ {(t)}\right) w _ {p r} ^ {(t - 1)} - \sum_ {p} \left(\sum_ {n \in \mathcal {R}} w _ {n p} ^ {(t)} + w _ {b p} ^ {(t)}\right) w _ {p b} ^ {(t - 1)} \\ = \sum_ {p} \left(\sum_ {n \in \mathcal {R}} w _ {n p} ^ {(t)} + w _ {b p} ^ {(t)}\right) \left(w _ {p r} ^ {(t - 1)} - w _ {p b} ^ {(t - 1)}\right) \\ = \sum_ {p \in \mathcal {R}} \left(\sum_ {n \in \mathcal {R}} w _ {n p} ^ {(t)} + w _ {b p} ^ {(t)}\right) \left(w _ {p r} ^ {(t - 1)} - w _ {p b} ^ {(t - 1)}\right) + \sum_ {p \in \mathcal {B} \setminus \{b \}} \left(\sum_ {n \in \mathcal {R}} w _ {n p} ^ {(t)} + w _ {b p} ^ {(t)}\right) \left(w _ {p r} ^ {(t - 1)} - w _ {p b} ^ {(t - 1)}\right) \\ + \left(\sum_ {n \in \mathcal {R}} w _ {n b} ^ {(t)} + w _ {b b} ^ {(t)}\right) \left(w _ {b r} ^ {(t - 1)} - w _ {b b} ^ {(t - 1)}\right) \\ \geq \sum_ {p \in \mathcal {R}} ^ {\left. i i i\right)} \left(\sum_ {n \in \mathcal {R}} w _ {n b} ^ {(t)} + w _ {b b} ^ {(t)}\right) \left(w _ {p r} ^ {(t - 1)} - w _ {p b} ^ {(t - 1)}\right) + \left(\sum_ {n \in \mathcal {R}} w _ {n b} ^ {(t)} + w _ {b b} ^ {(t)}\right) \left(w _ {b r} ^ {(t - 1)} - w _ {b b} ^ {(t - 1)}\right) \\ + \sum_ {p \in \mathcal {B} \backslash \{b \}} \left(\sum_ {n \in \mathcal {R}} w _ {n p} ^ {(t)} + w _ {b p} ^ {(t)}\right) \left(w _ {p r} ^ {(t - 1)} - w _ {p b} ^ {(t - 1)}\right) \\ = \left(\sum_ {n \in \mathcal {R}} w _ {n b} ^ {(t)} + w _ {b b} ^ {(t)}\right) \left(\sum_ {p \in \mathcal {R}} w _ {p r} ^ {(t - 1)} - \sum_ {p \in \mathcal {R}} w _ {p b} ^ {(t - 1)} + w _ {b r} ^ {(t - 1)} - w _ {b b} ^ {(t - 1)}\right) \\ + \sum_ {p \in \mathcal {B} \backslash \{b \}} \left(\sum_ {n \in \mathcal {R}} w _ {n p} ^ {(t)} + w _ {b p} ^ {(t)}\right) \left(w _ {p r} ^ {(t - 1)} - w _ {p b} ^ {(t - 1)}\right) \\ \end{array}
$$
$$
\begin{array}{c} (i v) \\ \geq 0 \end{array}
$$
(iii) holds because for  $p$  a regular node, we have  $\sum_{n \in \mathcal{R}} w_{np}^{(t)} + w_{bp}^{(t)} > \sum_{n \in \mathcal{R}} w_{nb}^{(t)} + w_{bb}^{(t)}$ , which follows from condition (2), and  $w_{pr}^{(t-1)} - w_{pb}^{(t-1)} \geq 0$  for  $p \neq b$ , following from condition (3).
(iv) holds because of conditions (2) and (3).
Verification of (3): for  $n \neq b$ , we want to show the trust towards a regular node  $r$  is bigger than towards a low-quality node  $b$ , that is  $\sum_{p} w_{np}^{(t)} w_{pr}^{(t)} > \sum_{p} w_{np}^{(t)} w_{pb}^{(t)}$
$$
\begin{array}{l} \sum_ {p} w _ {n p} ^ {(t)} w _ {p r} ^ {(t)} - \sum_ {p} w _ {n p} ^ {(t)} w _ {p b} ^ {(t)} \\ = \sum_ {p \in \mathcal {R}} w _ {n p} ^ {(t)} \left(w _ {p r} ^ {(t)} - w _ {p b} ^ {(t)}\right) + \sum_ {p \in \mathcal {B} \backslash \{b \}} w _ {n p} ^ {(t)} \left(w _ {p r} ^ {(t)} - w _ {p b} ^ {(t)}\right) + w _ {n b} ^ {(t)} \left(w _ {b r} ^ {(t)} - w _ {b b} ^ {(t)}\right) \\ \stackrel {(v)} {\geq} \sum_ {p \in \mathcal {R}} w _ {n b} ^ {(t)} \left(w _ {p r} ^ {(t)} - w _ {p b} ^ {(t)}\right) + w _ {n b} ^ {(t)} \left(w _ {b r} ^ {(t)} - w _ {b b} ^ {(t)}\right) + \sum_ {p \in \mathcal {B} \backslash \{b \}} w _ {n p} ^ {(t)} \left(w _ {p r} ^ {(t)} - w _ {p b} ^ {(t)}\right) \\ = w _ {n b} ^ {(t)} \left(\sum_ {p \in \mathcal {R}} w _ {p r} ^ {(t)} - \sum_ {p \in \mathcal {R}} w _ {p b} ^ {(t)} + w _ {b r} ^ {(t)} - w _ {b b} ^ {(t)}\right) + + \sum_ {p \in \mathcal {B} \backslash \{b \}} w _ {n p} ^ {(t)} \left(w _ {p r} ^ {(t)} - w _ {p b} ^ {(t)}\right) \\ \begin{array}{c} (v i) \\ > 0 \end{array} \\ \end{array}
$$
(v) holds because for  $n \neq b$ , we have  $w_{np}^{(t)} \geq w_{nb}^{(t)}$ , following from condition (3), and  $w_{pr}^{(t)} - w_{pb}^{(t)} \geq 0$  for  $p \neq b$ .
(vi) holds following from conditions (2) and (3).
It follows that in the product  $\mathbf{W}^{(\tau)}\mathbf{W}^{(\tau -1)}\ldots \mathbf{W}^{(1)}$ , a low-quality node will still have a lower column sum than any regular node. Because conditions (1)(2)(3) holds for any product of  $\mathbf{W}^{(t)}$ 's as long as each of the  $\mathbf{W}^{(t)}$  share the conditions listed by (1)(2)(3).
After  $t > \tau$ , multiplying with a naive weight matrix does not change the column sum order, we will have all low-quality nodes have lower importance in the consensus than the regular nodes.
F Reasoning for confidence weighting factor  $\beta_{i}^{(t)}$
In this section, we justify our choice of  $\beta_i^{(t)}(\pmb{x})$  in Section 4.3, i.e. we show via adding such a term, we are able to downweight a regular node's trust towards a bad node.
$\Phi^{(t)}$  is a row-normalized pairwise cosine similarity matrix, with  $(i,j)$ -th entry before row normalization as
$$
\frac {1}{n ^ {\star}} \sum_ {\boldsymbol {x} ^ {\prime} \in \boldsymbol {X} ^ {*}} \frac {\left\langle f _ {\boldsymbol {\theta} _ {i} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) , f _ {\boldsymbol {\theta} _ {j} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle}{\| f _ {\boldsymbol {\theta} _ {i} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) \| _ {2} \| f _ {\boldsymbol {\theta} _ {j} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) \| _ {2}} \tag {18}
$$
After adding a  $\beta_i^{(t)}(\pmb{x}) = 1 / \mathcal{H}(f_{\pmb{\theta}_i^{(t - 1)}}(\pmb{x}))$ , we have  $\pmb{W}^{(t)}$  with  $(i,j)$ -th entry before row normalization as
$$
\frac {1}{n ^ {\star}} \sum_ {\boldsymbol {x} ^ {\prime} \in \boldsymbol {X} ^ {*}} \frac {1}{\mathcal {H} \left(f _ {\boldsymbol {\theta} _ {i} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right)\right)} \frac {\left\langle f _ {\boldsymbol {\theta} _ {i} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) , f _ {\boldsymbol {\theta} _ {j} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) \right\rangle}{\| f _ {\boldsymbol {\theta} _ {i} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) \| _ {2} \| f _ {\boldsymbol {\theta} _ {j} ^ {(t - 1)}} \left(\boldsymbol {x} ^ {\prime}\right) \| _ {2}} \tag {19}
$$
We want to show that the weighting scheme down-weights a regular node  $i$ 's trust towards a low-quality node  $b$ , that is
$$
\phi_ {i b} ^ {(t)} > w _ {i b} ^ {(t)}
$$
As the comparison is made with respect to the same time step  $t$ , we drop the  $t$  notation from now on. Let  $\{a_0,..,a_{N - 1}\}$  be the cosine similarity between a regular agent  $i$  and others inside agent  $i$ 's confident region, and  $\{b_0,..,b_{N - 1}\}$  be the cosine similarity between  $i$  and others outside agent  $i$ 's confident region. By confident region, we mean region with low entropy in class probabilities, i.e. the model is more sure about the prediction. Further, we make the following assumptions:
a) for  $\pmb{x}^{\prime}$  in agent  $i$ 's confident region, we have low entropy of predicted class probabilities:  $\mathcal{H}(f_{\pmb{\theta}_i^{(t - 1)}}(\pmb{x}')) = 1 / c_1$ ; while for  $\pmb{x}^{\prime}$  outside agent  $i$ 's confident region, we have  $\mathcal{H}(f_{\pmb{\theta}_i^{(t - 1)}}(\pmb{x}')) = 1 / c_2$ . We further assume  $0 < c_2 < c_1$ .
b) inside a regular node  $i$ 's confident region,  $i$  has a better judgment of the alignment score produced by cosine similarity, such that the cosine similarity with low quality  $b$  is weighted lower inside:
$$
\frac {a _ {b}}{\sum_ {j} a _ {j}} <   \frac {b _ {b}}{\sum_ {j} b _ {j}} \tag {20}
$$
to claim  $w_{ib} < \phi_{ib}$ , we need to show
$$
\frac {c _ {1} a _ {b} + c _ {2} b _ {b}}{\sum_ {j} \left(c _ {1} a _ {j} + c _ {2} b _ {j}\right)} <   \frac {a _ {b} + b _ {b}}{\sum_ {j} \left(a _ {j} + b _ {j}\right)} \tag {21}
$$
Proof. Re-arrange Equation 20, we get
$$
b _ {b} \sum_ {j} a _ {j} > a _ {b} \sum_ {j} b _ {j} \tag {22}
$$
Multiply with  $c_{2} - c_{1}$  on both sides, we have
$$
\left(c _ {2} - c _ {1}\right) b _ {b} \sum_ {j} a _ {j} <   \left(c _ {2} - c _ {1}\right) a _ {b} \sum_ {j} b _ {j} \tag {23}
$$
$$
c _ {2} b _ {b} \sum_ {j} a _ {j} + c _ {1} a _ {b} \sum_ {j} b _ {j} <   c _ {1} b _ {b} \sum_ {j} a _ {j} + c _ {2} a _ {b} \sum_ {j} b _ {j} \tag {24}
$$
Now add  $c_{1}a_{b}\sum_{j}a_{j} + c_{2}b_{b}\sum_{j}b_{j}$  to both sides, we have
$$
c _ {1} a _ {b} \sum_ {j} a _ {j} + c _ {2} b _ {b} \sum_ {j} a _ {j} + c _ {1} a _ {b} \sum_ {j} b _ {j} + c _ {2} b _ {b} \sum_ {j} b _ {j} <   \tag {25}
$$
$$
c _ {1} a _ {b} \sum_ {j} a _ {j} + c _ {1} b _ {b} \sum_ {j} a _ {j} + c _ {2} a _ {b} \sum_ {j} b _ {j} + c _ {2} b _ {b} \sum_ {j} b _ {j}
$$
Combining the terms we have
$$
\left(c _ {1} a _ {b} + c _ {2} b _ {b}\right) \left(\sum_ {j} \left(a _ {j} + b _ {j}\right)\right) <   \left(\sum_ {j} \left(c _ {1} a _ {j} + c _ {2} b _ {j}\right)\right) \left(a _ {b} + b _ {b}\right) \tag {26}
$$
following which, we directly have
$$
\frac {c _ {1} a _ {b} + c _ {2} b _ {b}}{\sum_ {j} \left(c _ {1} a _ {j} + c _ {2} b _ {j}\right)} <   \frac {a _ {b} + b _ {b}}{\sum_ {j} \left(a _ {j} + b _ {j}\right)} \tag {27}
$$
G Complementary details
G.1 Details regarding model training
All the model training was done using a single GPU (NVIDIA Tesla V100). For each local iteration, we load local data and shared unlabeled data with batch size 64 and 256 separately. We empirically observed that a larger batch size for unlabeled data is necessary for the training to work well. The optimizer used is Adam with a learning rate 5e-3. For Cifar10 and Cifar100, as the base model is not pretrained, we do 50 global rounds with 5 local training epochs for each agent per global round. For Fed-ISIC-2019 dataset, as the base model is pretrained EfficientNet, we do 20 global rounds. For the first 5 global rounds, we set  $\lambda = 0$  to arrive at good local models, such that every agent can evaluate trust more fairly. After that,  $\lambda$  is fixed as 0.5. Dynamic trust is computed after each global round, while static trust denotes the utilization of the initially calculated trust value throughout the whole experiment.
For Cifar10 and Cifar100, we use  $5\%$  of the whole dataset to constitute  $\mathbf{X}^*$ , where each class has equal representation. For the rest, we spread them into 10 clients using Dirichlet distribution with  $\alpha = 1$ . For Fed-ISIC-2019 dataset, we follow the original splits as in du Terrail et al. [34], and we let each client contribute 50 data samples to constitute  $\mathbf{X}^*$ .
We employ a fixed  $\lambda$  for all our experiments. To select  $\lambda$ , we randomly sample  $10\%$  of the full Cifar10 dataset, which we then split into local training data  $(95\%)$  and  $\mathbf{X}^{*}$ $(5\%)$ . The local training data is then spread into 10 clients using Dirichlet distribution with  $\alpha = 1$ . The test global accuracy and value of  $\lambda$  is plotted out in Fig. 8. We thus choose  $\lambda = 0.5$  for all our experiments, and it is always able to give stable performances according to our experiments.
Figure 8:  $\lambda$  versus algorithm performance
G.2 Limitations of the work
The main limitation of this work is the requirement of an extra shared unlabelled dataset, like in other knowledge distillation-based decentralized learning works. Moreover, each agent needs to calculate their trust towards all other nodes locally. The extra computational complexity is  $\mathcal{O}(N^2 \times n^\star \times C)$  per global round, where  $N$  stands for the number of agents,  $n^\star$  stands for the size of the shared dataset and  $C$  denotes the number of classes. The computation can be heavy if the number of clients gets large. But as we focus on cross-silo setting,  $N$  usually does not tend to be a big number.