Test-time Collective Prediction
Authors: Celestine Mendler-Dünner
Wenshuo Guo
Stephen Bates
Michael Jordan
Venue: NeurIPS 2021 Poster (2021)
Keywords: degroot
consenesus finding
combining expert opinion
opinion pool
adaptive model weighting
data heterogeneity
social sciences
federated learning
ensemble

Abstract: An increasingly common setting in machine learning involves multiple parties, each with their own data, who want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents to make better predictions than they would individually, but may not be willing to release labeled data or model parameters. In this work, we explore a decentralized mechanism to make collective predictions at test time, that is inspired by the literature in social science on human consensus-making. Building on a query model to facilitate information exchange among agents, our approach leverages each agent’s pre-trained model without relying on external validation, model retraining, or data pooling. A theoretical analysis shows that our approach recovers inverse mean-squared-error (MSE) weighting in the large-sample limit which is known to be the optimal way to combine independent, unbiased estimators. Empirically, we demonstrate that our scheme effectively combines models with differing quality across the input space: the proposed consensus prediction achieves significant gains over classical model averaging, and even outperforms weighted averaging schemes that have access to additional validation data. Finally, we propose a decentralized Jackknife procedure as a tool to evaluate the sensitivity of the collective predictions with respect to a single agent's opinion.

Test-Time Collective Prediction
Celestine Mendler-Dünnee MPI for Intelligent Systems, Tübingen cmendler@tuubingen .pp.. de
Wenshuo Guo University of California, Berkeley wguo@cs.berken berkeley.
Stephen Bates University California, Berkeley stephenbates@cs. berkeley. edu
Michael I. Jordan University California, Berkeley j jordanecs. berkeley. edu edu
Abstract
An increasingly common setting in machine learning involves multiple parties, each with their own data, who want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents make better predictions than they would individually, but may not be willing to release labeled data or model parameters. In this work, we explore a decentralized mechanism to make collective predictions at test time, that inspired by the literature in social science on human consensus-making Building on a query model to facilitate information exchange among agents, our approach leverages each agent's pre- tre-trained model without relying external validation, model retraining, or data pooling. A theoretical analysis shows that our approach recovers inverse mnean-squared-ee which is known to be the optimal way to combine independent, unbiased estimators. Empirically, we demonstrate that our scheme effectively combines models with differing quality across the input space: the proposed consensus prediction achieves significant gains over classical model averaging, and even outperforms weighted averaging schemes that have access to additional validation data. Finally, we propose decentralized Jackknife procedure tool evaluate the sensitivity of the collective predictions with respect to single agent's opinion.
(MSE) weighting in the large-sample limit
Introduction
Large-scale datasets often collected from diverse sources, by multiple parties, and stored across different machines. In many scenarios centralized pooling of data not possible due to privacy concerns, data ownership, storage constraints. The challenge of doing machine learning in such distributed and decentralized settings has motivated research areas of federated learning [Konečný al., 2015, McMahan et al., 2017], distributed learning [Dean et al., 2012, Gupta and Raskar, 2018), as well as hardware and software design [Zaharia et al., 2016, Moritz et al., 2018].
While the predominant paradigm in distributed machine learning is collaborative learning of one centralized model, this level of coordination across machines at the training stage is sometimes not feasible. In this work, instead aim for collective prediction test time without posing any specific requirement at the training stage. Combining the predictions pre-trained machine learning models has been considered in statistics and machine learning in the context of ensemble methods, including bagging [Breiman, 1996a] and stacking [Wolpert, 1992]. In this work, we explore a new perspective on this aggregation problem and investigate whether insights from the social sciences how humans reach a consensus can help us design more effective aggregation schemes that fully take advantage of
"ost of work conducted while Berkeley.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

weights
1.0
agent
agent 2
MSE
agent
agent 4
0.0
agent
Projection of
Figure 1: (Test-time aggregation of heterogeneous models). The proposed method upweights more accurate models (lower MSE) adaptively in different regions of the input space. Constructed from regression task described in Section 4.1.
each model''s individual strength. At high level, when a panel of human experts come together to make collective decisions, they exchange information and share expertise through discourse to then weigh their subjective beliefs accordingly. Experts have a conforming influence on each other which gives rise to dynamic process of consensus finding that is decentralized and does not rely on any external judgement. We map this paradigm to machine learning setting where experts correspond to pre-trained machine learning models, and show that it leads to an appealing mechanism for test-time collective prediction.
1.1 Our work
We build on a model for human consensus finding that defines an iterative process of opinion pooling based mutual frust scores among agents [DeGroot, 1974]. Informally speaking, a mutual trust score reflects an agent's willingness to adapt another agent's opinion. Thus, each individual's impact on the final judgment depends on how much it is trusted by the other agents.
Mapping the DeGroot model to our learning scenario, we develop a scheme where each agent uses its local training data assess the predictive capabilities of other agents' models and determine how much they should be trusted. This assessment of mutual trust is designed to be adaptive to the prediction task at hand, by using only subset of the local data in the area around the current test point. Then, building on these trust scores, a final judgment reached through an iterative procedure, where agents repeatedly share their updated predictions.
This aggregation scheme conceptually appealing as mimics human consensus finding, and exhibits several practical advantages over existing methods. First, is decentralized and does not require agents to release their labeled data or model parameters. This can be beneficial terms of communication costs and attractive from privacy perspective, as it allows agents to control access to their data and model at any time. Second, does not require hold-out validation data to construct or train the aggregation function. Thus, there no need to collect additional labeled data and coordinate shared access to a common database. Third, it does not require any synchronization across agents during the training stage, agents are free to choose their preferred model architecture, and train their models independently.
A crucial algorithmic feature of our procedure is its adaptivity to the test point at hand, which allows to deal with inherent variations the predictive quality of models in different regions of the input space. Figure 1 illustrates how mechanism adaptively upweights models with lower error depending on the test point's location the input space. fact, we prove theoretically that our aggregation procedure can recover inverse-MSB weighting in the large-sample limit, which is known to be optimal for variance reduction of unbiased estimators [Bates and Granger, 1969].
To assess our procedure's sensitivity to the predictions of individual agents we propose a decentralized Jackknife algorithm to compute error bars for the consensus predictions with respect to the collection of observed agents. These error bars offer attractive target of inference since they are agnostic to how individual models have been trained, can evaluated without additional model evaluations, and allow to diagnose the vulnerability of the algorithm to malicious agents.

---

On the empirical side, we demonstrate the efficacy of our mechanism through extensive numerical ex- periments across different learning scenarios. In particular, we illustrate the mechanism's 's advantages over model averaging as well as model selection, and demonstrate that it consistently outperforms alternative non-uniform combination schemes that have access to additional validation data across a wide variety of models and datasets.
1.2 Related work
The most common approach for combining expert judgment is to aggregate and average individ- ual expert opinions [Hammitt and Zhang, 2013]. In machine learning this statistical approach of equally-weighted averaging is also known the "mean-rule" and underlies the popular bagging technique [Breiman, 1996a, Buja and Stuetzle, 2006, Efron, 2014]. It corresponds to an optimal aggregation technique for dealing with uncertainty in settings where individual learners are ho- mogeneous and the local datasets are drawn independently from the same underlying distribution. While principled, and powerful due to its simplicity, an unweighted approach cannot account for heterogeneity in the quality, or the expertise, of the base learners.
To address this issue, performance- averaging schemes have been proposed. For determining an appropriate set of weights, these approaches typically rely on calibrating variables [Aspinall, 2010] to detect differences predictive quality among agents. In machine learning these calibrating variables correspond to labeled cross-validation samples, while in areas such as expert forecasting and risk analysis these variables take the form of set of agreed-upon seed questions [Cooke, 1991]. These weights can be updated over time using statistical information about the relative predictive performance [Rogova, 2008] or the posterior model probability [Wasserman et al., 2000]. More recently there have been several applications to classification that focus on dynamic selection techniques [Cruz et al., 2018b), where the goal is to select the locally best classifier from pool of classifiers on the fly for every test point. At their core, these approaches all rely on constructing good proxy for local accuracy in a region of the feature space around the test point and to use this proxy to weight or rank the candidate models. Various versions of distance and accuracy metrics have been proposed in this context [eg. Woods et al., 1997, Didaci et al., 2005, Cruz et al., 20 8a]. However, all these methods rely crucially on shared access to labeled validation data to determine the weights. Similarly, aggregation rules such as stacking [Breiman, 1996b, Coscrato et al., 2020] or meta-models [Albardan et al., 2020], both of which have proved their value empirically, require central trusted agency with additional data to train an aggregation function on top of the predictor outputs. A related model-based aggregation method, the mixture-of-expert. model [Jacobs et al., 1991], requires joint training of all models as well as data pooling.
To the best of our knowledge there is no approach in the literature that can perform adaptive performance weighting without relying on a central agent to validate the individual models. Notably, approach differs from general frameworks such as model parameter mixing [Zhang et al., 2013], model fusion [Leontev et al., 2018, Singh and Jaggi, 2020] in that agents are not required to release their models.
Outside machine learning, the question of how to combine opinions from different experts to arrive at a better overall outcome has been studied for decades in risk analysis [Hanea et al., 2018, Clemen and Winkler, 1999] and management science [Morris, 1977]. Simplifying the complex mechanism of opinion dynamics, the celebrated DeGroot model [DeGroot, 1974] offers an appealing abstraction to reason mathematically about human consensus finding- It has been popular in the analysis of convergent beliefs in social networks and it can be regarded as formalization of the Delphi technique, which has been used to forecast group judgments related to the economy, education, healthcare, and public policy [Dalkey, 1972]. Extensions of the DeGroot model have been proposed to incorporate more sophisticated pocio-psychologica features that may be involved when individuals interact, such homophily [Hegselmann and Krause, 2002], competitive interactions [Altafini, 2013] and stubbornness [Friedkin and Bullo, 2017].
2 Preliminaries
We are given K independently trained models and aim to aggregate their predictions at test time. Specifically, we assume there are K agents, each of which holds local training dataset, DR. of size
3

---

ne, and model fo : - V. We do not make any prior assumption about the quality or structure of the individual models, other than that they aim solve the same prediction task.
At test time, let x' E X denote the data point that we would like to predict. Fach agent l E [K] has her own prediction for the test point corresponding fx(x'). Our goal is to design mechanism. M that combines these predictions into an accurate single prediction, p" (a') ) (1ff(x')),... fx (z')).
In the following we focus on regression and use the squared loss measure of predictive quality,
The requirements we pose on the aggregation mechanism M are: (i) agents should not be required to share their labeled data model parameters with other agents, any external party; (ii) the procedure should not assume access to additional training data for an independent performance assessment or training of a meta-model.
2.1 The DeGroot consensus model
The consensus model proposed by DeGroot [1974] specifies how agents in group influence each other's opinion on the basis of mutual trust. Formally, we denote the trust of agent i towards the belief of agent j as Tiy E (0, 1], with ΣjE[K] Tip for every Then, DeGroot defines an iterative procedure whereby each agent i repeatedly updates her belief pt as
(++1) P3
(0) Tia Pj
The procedure initialized with initial beliefs p ( and run until -/0 -p for every pair i,j E [K]. which point consensus is deemed to have been reached. We denote the consensus belief by p".
The DeGroot model has a family resemblance to the influential PageRank algorithm [Page et al., 1999] that is designed to aggregate information across webpages in decentralized manner. Similar to the consensus in DeGroot, the page rank corresponds to steady state of equation (2), where the weights Tiy computed based on the fraction of outgoing links pointing from page to page j.
2.2 Computing a consensus
The DeGroot model can equivalently be thought of as specifying a weight for each agent's prediction in a linear opinion pool [Stone, 1961]. This suggests another way to compute the consensus prediction. Let us represent the mutual trust scores in the form of a stochastic matrix, T = ) which defines the state transition probability matrix of a Markov chain with K states. The stationary state w of this Markov chain, satisfying wT w, defines the relative weight of each agent's 's initial belief in the final consensus
(0) W Pj
(2)
In our method the Markov chain is guaranteed be irreducible and aperiodic since all entries of T'' are positive. Therefore, unique stationary distribution w exists, and consensus will be reached from any initial state [Chatterjee and Seneta, 1977]. We refer to the weights w as consensus weights.
To compute the consensus prediction pd one can solve the eigenvalue problem numerically for w via power iteration. Letting T = limm- 7 the rows of 7" identify, and correspond the weights w that DeGroot assigns to the individual agents [Durrett, 2019, Theorem 5.6.6). The consensus (2) is computed weighted average over the initial beliefs.
3 DeGroot aggregation for collective test-time prediction
In this section we discuss the details of how we implement the DeGroot model outlined in Section 2.1 for defining a mechanism M to aggregate predictions across machine learning agents. We then analyze the resulting algorithm theoretically, proving its optimality in the large-sample limit, and we outline how the sensitivity of the procedure with respect to individual agents can be evaluated in decentralized fashion using Jackknife method.

---

Algorithm DeGroot Aggregation
1: Input: K agents with pre-trained models , I'x and local data DR, k [K]; neighborhood size N; test point x'.
2: construct trust scores (based on local accuracy):
3: for ..... do
4: Construct local validation dataset (a("") using N-nearest neighbors of x' in D;-
5: Compute accuracy MSE,j (z') ) of other agents j 1. .... K on D2(x') according to (3).
6: Evaluate local trust scores {Ti))sE(K) by normalizing MSE,, in (4).
7: end for
8: find consensus:
9: Run pooling iterations (2), initialized at Pj
(0) = fa(" Vj, until consensus p" is reached.
10: Return: Collective prediction p" (z')
3.1 Local cross-validation and mutual trust score
key component of DeGroot's s model for consensus finding is the notion of mutual trust between agents, built by information exchange in form of discourse. Given machine learning setup with K agents each of which holds a local dataset and a model fk. we simulate this discourse by allowing query access to predictions from other models. This enables each agent to validate another agent's predictive performance on its own local data. This operationalizes a proxy for trustworthiness.
An important requirement for our approach is that it should account for model heterogeneity, and define trust light of the test sample x' we want predict. Therefore, we design an adaptive method, where the mutual trust Túj is reevaluated for every query via local vols-validation procedure. Namely, agent evaluates all other agents' predictive accuracy using a subset (x C D, of the local data points that are most similar to the test point x' [ef. Woods et al., 1997]. More preciscly, agent evaluates
MSE,;(2')
Σ
(f)(z))))
3
Di(x')
(D.)))
locally for every agent j [K], and then performs normalization to obtain the trust scores:
1/MSE
Tij = ]j((K] 1/MSE.j
4)
There are various ways one can define the subset Di(x') ( see, c.g., Cruz et al. [2018b] for a discussion of relevant distance metries in the context of dynamic classifier selection. Since we focus tabular data in our experiments we use Euclidean distance and assume (z')) has fixed size. Other distance metrics, alternatively kernel-based approach, could readily be accommodated within the DeGroot aggregation procedure.
3.2 Algorithm procedure and basic properties
We now describe our overall procedure. We take the trust scores from the previous section, and use them to aggregate the agents' predictions via the DeGroot consensus mechanism. See Algorithm for a full description of the DeGroot aggregation procedure. In words, after each agent evaluates her trust in other agents, she repeatedly pools their updated predictions using the trust scores as relative weights. The consensus which this procedure converges is returned as the final collective prediction, denoted p (x'). ) the following we discuss some basic properties of the consensus prediction found by DeGroot aggregation.
In the context of combining expert judgement, the seminal work by Lehrer and Wagner [1981] on social choice theory has defined unanimity as general property any such aggregation function should satisfy. Unanimity requires that when all agents have the same subjective opinion, the combined prediction should be no different. Algorithms naturally satisfies this condition.
Proposition 3.1 (Unanimity). If all agents agree on the prediction, then the consensus prediction from Algorithm I agrees with the prediction of each agent: p" (a') f((x') for every [K].
5

---

In addition, our algorithm naturally preserves a global ordering of the models, and the consensus weight of each agent is bounded from below and above by the minimal and maximal trust she receives from any of the other agents.
Proposition 3.2 (Ranking preeiingggeeeerggg...... Suppose all agent rankings have the same order: for all J1.J2 [K], if Téfa Tapz for some E [ then Ty 2 Tv for all [K]. Then, Algorithm I
finds a consensus
2; wif((x') where for pairs J1, such that Tij. 2 Tafz we have
Wi 3 Wj
Proposition 3.3 (Bounded by min and max trust). The final weight assigned to agent is between the minimal and maximal trust assigned to it by the set of agents:
max Tjx 2 min Tja-
Finally, Algorithm recovers weighhed averaging whenever each agent receives a constant amount of combined total trust from the other agents. This is satisfied if all agents perform equally well on the different validation sets. However, the trust necessarily needs to allocated uniformly, there may be multiple T that result in uniform stationary distribution.
Proposition 3.4 (Averaging as a special case). If the columns of I sum to 1, then Algorithm 1 returns an equal weighting of the agents: w, 1/K for 1.....
Together these properties serve basic sanity check that Algorithm I implements reasonable consensus-finding procedure, from a decision-theoretic as well as algorithmic perspective.
3.3 Optimality in a large-sample limit
In this section we analyze the large-sample behavior of the DeGroot consensus mechanism, showing that it recovers the optimal weighting scheme in the scenario where agents are independent.
For our Harge-sample analysis, we suppose that the agents' local datasets Dj are drawn independently from (different) distributions Ps over X' x y. In order for our prediction task be well-defined, we assume that the conditional distribution of Y given X is the same for all Pk. In other words, the distributions Ps are all covariate shifts of each other. For simplicity, we assume the distributions Pk are all continuous and have the same support. We will consider the large-sample regime in which each agent has a growing amount of data, growing the same rate. That is, if n = |D1 + +.... [De] is the number of total training points, then 'Dzl/n- n - Cj > 0, as - co for each agent k = 1,.... K. Lastly, we require basic consistency condition: for any compact set A X. fa + fž uniformly over E A as + for some continuous function f
Theorem 3.5 (DeGroot converges to inverse-MSE weighting). Let E X be some test point. Assume that Pk supported in some ball of radius dg centered at x', and that the first four conditional moments of Y given X = are continuous functions of x in this neighborhood. Next, suppose we run Algorithm choosing N [n[] neurest neighbors for some e E (0, 1). Let
MSE [E -FE(2)))
denote the asymptotic MSE of model l at the test point x', where the expectation is over draw of Y from the distribution of Y x'. Then, the DeGroot consensus mechanism yields weights
1/MSE;
Ws - w = 1/MSE; + ... I/MSE
In other words, in the large-sample limit the DeGroot aggregation yields inverse-MSE weighting and thus recovers inverse-variance weighting for unbiased estimators a canonical weighting scheme from classical statisties [Hartung al., 2008] that is known to be the optimal way to linearly combine independent, unbiased estimates. This can be extended to our setting to show that DeGroot averaging leads to the optimal aggregation when we have independent' agents, stated next.
Whiie independence is a condition that we can't readily check in practice, it may hold approximately the agents use different models constructed from independent data. In any case, the primary takeaway is that DeGroot is acting reasonably in this tractable case.

---

Theorem 3.6 (DeGroot is optimal for independent, unbiased agents), Assume that the conditional
mean and variance of given X x are continuous functions of x. For some ö > O. consider drawing X uniformly from ball of radius 8 centered x' and Y from the distribution of X X. Under this sampling distribution, suppose the residuals Y Y-J [x(X) from the agents' predictions have mean zero and each pair has correlation zero. Then the optimal weights,
(i-
W = arg min E B(x,9S
Σ KE[K]
wxff
=ER*:||[|l==
approach the DeGroot weights: i + w as +
In summary, Theorem 3.5 shows that the DeGroot weights asymptotically converging meaningful (sometimes provably optimal) aggregation the models - locally for each test point x'- This is stronger notion of adaptivity than that usually considered in the model-averaging literature.
3.4 Error bars for predictions
Finally, we develop a decentralized Jackknife algorithm [Quenouille, 1949, Efron and Tibshirani, 1993] to estimate the standard eπror of the consensus prediction p" (x'). Our proposed procedure measures the impact of excluding a random agent from the ensemble and returns error bars that measure how stable the consensus prediction is the observed collection of agents. Formally, let (x"("") denote the consensus reached by the ensemble after removing agent i. Then, the Jackknife estimate of standard error corresponds
SE(z')
Σ (p2((")--(xx")) KE[K]
(5)
K
where ('()') 1 EK P=,(') is the average delete-one prediction.
In the collective prediction setting, this an attractive target of inference because it can be computed in decentralized manner, is entirely agnostic to the data collection or training mechanism employed by the agents, and requires no additional model evaluations above those already carried out in Algorithm 1. Furthermore, it allows one to diagnose the impact of single agent the collective prediction and thus assess the vulnerability of the algorithm to malicious agents. A detailed description of the DeGroot Jackknife procedure can be found in Algorithm 2 in Appendix B. Finally, want to note that this measure is not specific to how the consensus is reached via the DeGroot method and readily be applied as diagnosis tool other collective prediction algorithms.
3.5 Privacy-preserving properties
an appealing property of the DeGroot algorithm that it does not require agents to share their raw training data or model parameters upfront. Information is only exchanged needed through requesting and answering prediction queries of other agents. This implics that agents can do privacy accounting and remain in control of their data at any time. Sensitive information about the training data can only by leaked in Step of Algorithm 1 by revealing unlabeled data points and prediction queries. For privacy-critics applications this information could further be protected by augmenting DeGroot aggregation with techniques from differential privacy [Dwork and Roth, 2014], or tools from cryptography [Bonawitz et al., 2017]. While we are providing a rigorous privacy analysis of our method in this work, it could be an interesting extension for future work.
4 Experiments
We investigate the efficacy of the DeGroot aggregation mechanism empirically for various datasets, partitioning schemes and model configurations. We start with synthetic setup to illustrate the strengths and limitations our method. Then, we focus on real datasets to see how these gains surface in more practical applications with natural challenges, including data scarcity, heterogeneity of different kinds, and scalability. We compare our Algorithm (DeGroot) to the natural baseline of equally-weighte model averaging (M-avg) and also compare to the performance of individual
7

---

MSE avg
Indivoual agents K-avg
10-4
1.5
10
DeGrost
gessiction
1.0
2 10
20--
0.5
19
10-1
DeGroot
-10
10 "uumber of pooling iterations
-5
0
E- 0'x
E-
(a) individual test points
(b) consensus finding
DeGroot jackknitee
Figure 2: (Synthetic data experiment). Comparison of DeGroot to M-avg. (a) Predictive accuracy on individual test points. (b) Performance for varying number of pooling iterations during DeGroot consensus finding. (c) Confidence intervals for individual test points returned by DeGroot Jackknife.
models composing the ensemble. In Section 4.2 we include an additional comparison with two reference schemes that have access to shared validation data and central agent that determines an (adaptive) weighting inmeme-informaaioon DeGroot does not have.
4.1 Synthetic data
First, we investigate a synthetic two-dimensional setting where each agent's features are drawn from multivariate Gaussian distribution, x (un: Σx). The true labeling function given as y [1 exp(a x>]- and we add white noise of variance of to the labels in the training data. Unless stated otherwise, we use = 5 agents and let each agent fit a linear model to her local data. See Figure 4 in the Appendix for visualization of the models learned by the individual agents, and how they accurately approximate the true labeling function in different regions of the input space."
To evaluate the DeGroot aggregation procedure randomly sample 200 test points from the uniform mixture distribution spanned by the K training data distributions, and compare the prediction of DeGroot to M-avg. We visualize the squared error for the collective prediction on the individual test points Figure 2a where we use { == a x as a summary for the location of the test point on the logistic curve. We observe that DeGroot consistently outperforms M-avg across the entire range of the feature space. Exceptions are three singularities, where the errors of the individual agents happen to cancel out by averaging. Overall DeGroot achieves MSE of 4.6e which is impressive reduction of over 50x compared to M-avg. Not shown in the figure the performance of the best individual model which achieves an MSE of -2 and performs worse than the M-avg baseline. Thus, DeGroot outperforms an oracle model selection algorithm and every agent strictly benefits from participating in the ensemble. The power of adaptive weighting such as used in DeGroot is that can trace out a nonlinear function, given only linear models, whereas any static weighting (or model selection) scheme will always be bound the best linear fit.
In Figure 2b we show how individual agent's predictions improve with the number of pooling iterations performed in Step of Algorithm 1. The influence of cach local model on the consensus prediction of z' and how it changes depending on the location of the test point is illustrated in Figure in the introduction.
An interesting finding that we expand on Appendix D. that for the given setup the iterative approach of DeGroot is superior to more naive approach of combining the individual trust scores or local MSE values into a single weight vector. In particular, DeGroot reduces the MSE by 20x compared to using the average trust scores as weights. A theoretical understanding of this remarkable phenomenon remains to be developed.
In Figure 2c we visualize the error bars returned by our DeGroot Jackknife procedure. As expected, the intervals are large in regions where only one a small number of agents possess representative training data, and small in regions where there is more redundancy across agents.
II' not stated otherwise we spread the means as = [[-3, -4, [-2, -2, [-1,-1], 0), [3, 2 and let Σ, L We use 200 training samples on each agent, for the labeling function we use = [1, and for the label noise in the training data set σy O.I. We use N = for local crossvvaliattin in DeGroot.
8

---

30%
M-arg
30%
Makg DeGreot
DeGroot
20%
gan avaraging
20%
individual lagrrks
20%
irdivicual agents
gain aver sveraging
2056
10%
205
10%
-0%
-0%
M-avg
-30%
40%
individual agents
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.5
1.0
(a) label heterogeneity
(b) feature heterogeneity
(c) regularizer heterogeneity
Figure 3: (Data and model heterogeneity for the abalone data). Relative gain over M-avg. Confidence intervals of DeGroot are over randomness in data partitioning and for the individual agents the shaded area spans the performance from the worst to the best model, the line indicating average performance.
Finally, in Appendix D.1 we conduct further qualitative investigations of the DeGroot algorithm. First, we vary the covariance in the local data feature distribution. We find that the gain of DeGroot over M-avg becomes smaller as the covariance is increased; models become less specialized and there is little for DeGroot to exploit. On the other extreme, if the variance in the data is extremely small, the adaptivity mechanism of DeGroot becomes less effective, as the quality of individual models can no longer be reliably assessed. However, we need to go extreme values in order to observe these phenomena. a second experiment we explore DeGroot's 's sensitivity to choices of N. We find that when N is chosen too large adaptivity can suffer, and for very small values of N performance can degrade if the labels are highly noisy. However, there is a broad spectrum values for which the algorithm performs well, between 1% and 10% of the local data usually good choice.
4.2 Real datasets
Turning to real data sets, we first investigate how DeGroot deals with three different sources of heterogeneity. We work with the abalone dataset [Nash et al., 1994] and train lasso model on each agent² with regularization parameter A X that achieves sparsity of ~0.88. For our first experiment, shown in Figure 3a, we follow a popular setup from federated learning to control heterogeneity in the data partitioning [see, e.g., Shen et al., 2021]. We partition the training data in the following manner: a fraction 1 1- p the training data is partitioned randomly, and a fraction p is first sorted by the outcome variable y and then partitioned sequentially. For the second experiment, shown in Figure 3b, we use a similar partitioning scheme, but sort the data along an important feature dimension, instead of y. This directly leads to heterogeneity in the input domain. Finally, investigate model heterogeneity arising from different hyper- parameter choices of the regularization hyper-parameter parameter X, given random partitioning. To control the level of heterogeneity, we start with À X on each agent, and let the parameters diverge increasingly, such that Åς X' [1 + k=ku] with kn = 3. The results are depicted in Figure 3c. In all three experiments the gain of DeGroot over M-avg is more pronounced with increasing degree of heterogeneity. When there is heterogeneity in the partitioning, DeGroot clearly outperforms the best individual model, by taking advantage of models' local areas of expertise. For the third experiment where the difference is in model quality, DeGroot manages to almost recover the performance of the best single model.
Next, conduct broad comparison of DeGroot additional reference schemes across various datasets and models, reporting the results in Table 1. To the best of our knowledge there is no existing generic adaptive aggregation scheme that does not rely external validation data to either build the aggregation function or determining the weights w. Thus, we decided to include a comparison with an approach that uses validation dataset to compute the weighting. where the external validation data has the same size as the local partitions. Our two references are static inverse-MSIE weighting (CV- static) where the weights are determined based on the model's average performance on the validation data, and adaptive inverse-MSE weighting (CV-adaptive), where the performance is evaluated only in the region around the current test point z'. We achieve heterogeneity across local datasets by
Similar phenomena can be observed for other datasets and models (see Appendix D.2)

---

> x=1 max_depth Table (Benchmark comparison). Performance for different dataset/model combinations. Datasets have been downloaded from [Fan, 2011]. We use ridge and lasso as a baseline and train a decision tree regressor (DTR) for discrete outcome variables, and a neural net (NN) with two hidden layers for continuous outcomes. A positive relative gain means a reduction in MSE compared to DeGroot.
MSE DeGroot
gain relative to DeGroot [%]
hyper- parameters
M-avg
CV-static
CV-adaptive
Boston
Ridge
25.23
-12.45+22.99
-10.24-+1.90
-2.80
À le À = 5e -8
Lasso
24.17
113.70#1.72
10.35-:1.16
-3,46:11.44
NN
14.18
-15.1813,433
-11.0113.08
-5.81133.20
layer= (7,7)
E2006
Ridge
0.15
-0.1244-0.66
-0.21+00.06
0.13+-0.11
À = 5e X 5e
Lasso
0.097
-0.92:10.15
-0,90::0113
-0.08 0.07
NN
0.11
14,58:13.48
5.69:10.97
-0.16:0..3
layer= (9,9)
Abalone
Ridge
3.18
-3.67 + 0,55
-3.52:0.47
0049:10.46
de
Lasso
4.93
-1,099 0.83
10.055+0.80
-0.39:0.75
A = 5c
DTR
4.85
-2.531 0.69
-2.55::0.70
00.7910.88
max depth
epusmall
Ridge 59.00
-96.231:23446
15.881=15.58
0.78
1e- X = le dep_depth=7
Lasso
53.82
14,408=14,45
17..81+
1.80
DTR
11.16
-4.05+ 1.88
+3.90++..11
1.65++2.20
YearPrediction
Ridge
95.02
-0.63::.24
-0.4410.11
0.21
A ==1
I Lasso
91.21
-1.73++0.47
-1.98+0..5
-0.22++0.9
DTR
62.11
-3.13+11.3
-2.26+11.22
-0.54+00.3
partially sorting a fraction p= = 0.5 of the labels, and we choose N to be 1% of the data partition for all schemes (with a had lower bound at 2). Each scheme evaluated for the same set of local models and confidence intervals are over the randomness of the data splitting into partitions, test and validation data across evaluation runs. The results demonstrate that DeGroot can effectively take advantage of the proprietary data to find good aggregate prediction, significantly outperforming M-avg and CV-static, and achieving better or comparable performance to CV-adaptive that works with additional data.
To complement our study, we verify in Appendix D.2 that our aggregation mechanism scales robustly with the number of agents in the ensemble and the simultaneously decreasing partitioning size: DeGroot consistently outperforms model averaging on two different datasets on the full range from 2 up to 128 agents. Finally, for the reader who is interested in applying the DeGroot procedure to classification tasks we have outlined a classification example in Appendix D.3.
Discussion
We have shown that insights from the literature humans consensus via discourse suggest effective aggregation scheme for machine learning. In settings where there no additional validation data available for assessing individual models' quality, our approach offers appealing mechanism to take advantage of individual agents' proprictary data to arrive at an accurate collective prediction. Our mechanism relies on information exchange between agents formalized through query model. This emulation of a decentralized discussion phase that allows agents to influence each other's opinion offers an appealing new perspective on decentralized learning. We are convinced that building on this idea, developing richer models of discourse, and refining mutual trust through experience could offer a promising avenue for future research in federated learning.
Acknowledgments and funding statement
The authors would like to thank Jacob Steinhardt and Alex Wei for feedback on this manuscript. In terms of funding; CM was supported by the Swiss National Science Foundation Early Postdoc Mobility fellowship program, WG was supported by Google PhD fellowship. and MJ acknowledges support from the Vannevar Bush Faculty fellowship program under grant number N00014-21-1-291)
10

---

References
Mahmoud Albardan, Klein, and O. Colot. Spoce: Scalable possibilistic classifier combination toward robust aggregation of classifiers. Expert Syst. Appl., 150:113332, 2020.
Claudio Altafini. Consensus problems networks with antagonistic interactions. IEEE Transactions on Automatic Control, 58(4):335 946, 2013.
W. Aspinall. A route to more tractable expert advice. Nature, 463(7779))294 295, 2010.
M. Bates and C. W. J. Granger. The combination of forecasts. Operational Research, 20(4): 451-468, 1969.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference Computer and Communications Security, CCS 17, page 1175-1191, 2017. ISBN 9781450349468, doi: 10.1 10.1144531339556313333
Leo Breiman. Bagging predictors. Mach. Learn., 24(2):123-140, August 1996a.
Leo Breiman. Stacked regressions. Mach. Learn., 24(1):49 64, July 1996b.
Andreas Buja and Werner Stuetzle. Observations on bagging. Statistica Sinica, 16(2):32--351, 2006.
Chatterjee and Seneta. Towards consensus: Some convergence theorems on repeated averaging.
Journal of Applied Probability, 14(1):89-97, 1977.
Robert T. Clemen and Robert L. Winkler. Combining probability distributions from experts in risk analysis. Risk Analysis, 19(2):187-203, 1999.
Roger M. Cooke. Experts Uncertainty: Opinion and Subjective Probability in Science. Oxford University Press, 1991.
Victor Coscrato, Marco Henrique de Almeida Inacio, and Rafael Izbicki. The NN-Stacking: Feature weighted linear stacking through neural networks. Neurocomputting. 399:141-152, 2020.
Rafael M. Cruz, Robert Sabourin, and George D. Cavalcanti. Prototype selection for dynamic classifier and ensemble selection. Neural Comput. Appl., 29(2) 2018a.
Rafael M. Cruz, Robert Sabourin, and George D.C. Cavalcanti. Dynamic classifier selection: Recent advances and perspectives. Information Fusion, 41:195 2018b.
N. Dalkey. Studies in the quality of life; DELPHI and decision-making. 1972.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang. Quoc Le, and Andrew Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems, volume 25, 2012.
Morris H. DeGroot. Reaching a consensus. Journal of the American Statistical Association, 69(345): 118-121, 1974.
Luca Didaci, Giorgio Giacinto, Fabio Roli, and Gian Luca Marcialis. A study on the performances of dynamic classifier selection based on local accuracy estimation. Pattern Recognition, 38(11): 2188-2191, 2005.
Dheeru Dua and Casey Graff. UCI machine learning repository. 2017. URL http:////arrrcii....ss aci.edd/ml..
Rick Durrett. Probability: theory and examples, volume 49, Cambridge university press, 2019,
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. 9(3 4):211-4409 2014. ISSN 1551-305X.
Bradley Efron. Estimation and accuracy after model selection. Journal of the American Statistical Association, 109(507)9911-000)) 2014.

---

Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, 1993.
Rong-En Fan. Libsvm data: Classification, regression, and multi-label, 2011. Datasets available at https:/ :/www////wwwww// http://www..iiieeooeoeeoeeeoeeet csie. ntu. edu. wwtp:/ddwww/////due/.ee/eeo///// wwwlin/lissttttollll.... /ata///aats/////////
Noah E. Friedkin and Francesco Bullo. How truth wins in opinion dynamics along issue sequences.
Proceedings of the National Academy of Sciences, 1 114(43): (380--1385) 2017.
Otkrist Gupta and Ramesh Raskar. Distributed learning of deep neural network over multiple agents.
Journal of Network and Computer Applications, 116:1-8, 2018.
James K. Hammitt and Yifan Zhang. Combining experts' judgments: Comparison of algorithmic methods using synthetic data. Risk Analysis, 33(1):109-11) 2013.
Anca Hanea, Marissa McBride, Mark Burgman, and Bonnie Wintle. The value of performance weights and discussion in aggregated expert judgments. Risk Analysis, 38, 03 2018.
J. Hartung, G. Knapp, and B.K. Sinha. Statistical Meta-Analysis with Applications. Wiley Series in Probability and Statistics. Wiley, 2008. ISBN 9780470290897.
Rainer Hegselmann and Ulrich Krause. Opinion dynamics and bounded confidence: Models, analysis and simulation. Journal of Artificial Societies and Social Simulation, 5:1-24, 2002.
Olle Häggström. Finite Markov Chains and Algorithmic Applications. London Mathematical Society Student Texts. Cambridge University Press, 2002.
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.
Shimon Kogan, Dimitry Levin, Bryan R. Routledge, Jacob Sagi, and Noah A. Smith. Predicting risk from financial reports with regression. NAACL '09, page 272-2822 Association for Computational Linguistics, 2009.
Jakub Konečný, Brendan McMahan, and Daniel Ramage. Federated optimization: Distributed optimization beyond the datacenter. Arxiv, arxiv: 2015.
Keith Lehrer and Carl Wagner. Rational Consensus in Science and Society. Boston: D. Reidel, 1981.
Mikhail Iu. Leontev, Viktoriia Islenteva, and Sergey V. Sukhov. Non-iterative knowledge fusion in deep convolutional neural networks. Arxiv, Mixiv:800.009399 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference Artificial Intelligence and Statistics, volume 54, pages 1273-1282. PMLR, 2017.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael I. Jordan, and lon Stoica. Ray: A distributed framework for emerging applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 561-577, 2018. Software available at https: /raa/aaaa/a////aa// .io.
Peter A. Morris. Combining expert judgments: A bayesian approach. Management Science, 23(7): 679-693, 1977.
Warwick Nash, T.L. Sellers, S.R. Talbot, A.J. Cawthorn, and W.B. Ford. 7he population biology of abalone (haliotis species) in tasmania. i. blacklip abalone (h. rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Technical Report No, 48, 01 1994.
Lawrence Page. Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, November 1999.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825-2882 2011.
12

---

M. H. Quenouille. Approximate tests of correlation in time-series. Journal of the Royal Statistical Society. Series B (Methodological), 11(1):68-4 84, 1949.
Galina Rogova, Combining the Results of Several Neural Network Classifiers, pages 683-692.
Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 978-3-540-44792-4
Zebang Shen, Hamed Hassani, Satyen Kale, and Amin Karbasi. Federated functional gradient boosting. Arxiv, auxiv:2103.06999 2021.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22045-22055, 2020.
M. Stone. The Opinion Pool. The Annals of Mathematical Statistics, 32(4):1339 - 1342, 1961.
Larry Wasserman et al. Bayesian model selection and model averaging. Journal of mathematical psychology, 44(1);92-107, 2000.
David H. Wolpert. Stacked generalization. Neural Networks, 5(2):241-259, 1992.
K. Woods, W.P. Kegelmeyer, and K. Bowyer. Combination multiple classifiers using local accuracy estimates. IEEE Transactions Pattern Analysis and Machine Intelligence, 1944):05-410, 1997.
Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, and Ion Stoica. Apache Spark: Unified Engine for Big Data Processing. Commun. ACM, 59(1 1):56-65. 65, 2016. Software available at https://sparkk apache org/
Yuchen Zhang. John C. Duchi, and Martin J. Wainwright. Communication--fficceenn algorithms for statistical optimization. Journal of Machine Learning Reseurch, 14(68):3321-3363 2013.
13


Reviews:
[{"review_id": "QeGvbh59FeP", "reviewer": "Reviewer_tLXB", "summary": "This paper proposes an interesting approach based on DeGroot’s consensus model for combining predictions of multiple pre-trained models to collectively make a final prediction. The proposed method is useful in cases where data pooling, model sharing, or external validation is difficult. Experiments show that the proposed method is better than classical baselines of model averaging under heterogeneous conditions, without assuming access to additional labeled data for validation or learning a meta-model. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The motivation of collective prediction from multiple independent models without model/weight sharing has a lot of practical relevance. Thus, this paper might be of interest to a broader community.\n- The proposed method does not require additional validation data for learning aggregation to do collective predictions and scales well with the number of models in the ensemble.\n- Sufficient and convincing experiments in the main paper and appendix demonstrating the usefulness of the proposed method for regression tasks. The authors also analyze the conditions when the proposed method may not be significantly better than simple ensembling approaches.\n\nLimitations:\n- Experiments are limited to the task of regression and involve only simple record-based datasets. The experimental section could be made stronger with more complex regression/classification tasks. \n- Computing trust scores for each test example might affect the latency in comparison to other simpler baselines. \n\nQuestions:\n- Is there any specific reason why classification tasks were not considered?\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "OrIRd0Lm9PR", "reviewer": "Reviewer_rCjs", "summary": "The authors present a decentralized algorithm for test-time model combination (collective prediction) wherein each party has an private model and dataset, and only has query access to other party's models. The algorithm is a novel application of DeGroot's consensus model which models how several agents iteratively influence each other's predictions based on mutual trust. The authors use the mean squared error on a local set of points as a proxy for mutual trust: each party queries each other party's models using points neighboring the test instance from its private dataset and measures the MSE on this set. The authors also show that the algorithm easily lends itself to a decentralized jackknife algorithm.\n\nThe authors prove several intuitive properties of their algorithm and prove that, asymptotically, it recovers an optimal inverse-MSE weighting. Empirically, they show that the method outperforms a naive equally-weighted average of the classifier predictions on synthetic and real data. Through synthetic experiments, the authors show that the proposed algorithm is particularly beneficial when there is significant heterogeneity in the private data or learners because of the adaptively computed weighting (based on MSE in a neighborhood around the test instance).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written, well-executed paper. The authors present a novel algorithm and characterize it both theoretically and through experiments. The experiments highlight where the model is expected to work well and where it reduces to existing baselines. The authors also provide an excellent survey or related work.\n\nMy main concern is the motivation for this learning setup: the key motivation for decentralized collective prediction is to keep allow parties to keep their data and models private. However, as parties use their own data to query other models, they are in affect sharing their dataset piece by piece with others. Is it possible to share data between parties in a differentially private way?\n\nMinor typos:\n\nLines 116-119: recommend using \\(x\\) instead of \\(x'\\).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author should mention or address the limitation regarding agents leaking private data by providing query access mentioned above.\n\nI do not foresee any negative societal impact from this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "FDahmc9qOBP", "reviewer": "Reviewer_gYkb", "summary": "This paper considers collective prediction at test time, and proposes a novel decentralized mechanism by leveraging each agent's pretrained model and data without sharing them with other agents, inspired by human consensus in social science. In particular, for each test example, the proposed approach computes trust scores between two agents and aggregate the scores for a final collective prediction via DeGroot aggregation. In theory, this paper proves that the proposed DeGroot consensus mechanism is asymptotically optimal under some conditions. The efficacy of the proposed approach is empirically demonstrated by showing that the proposed scheme is better than an un-weighted consensus baseline. \n\nThe major contributions include the following: \n(1) this paper introduces the concept of \"mutual trust scores among agents\" in social science (which is similar to PageRank algorithm in data science) for collective prediction at test time, \n(2) the proposed approach is advantageous over existing methods since it does not require agents to share data or model parameters, which are beneficial in terms of communication cost and privacy perspective, \n(3) the proposed approach is advantageous over existing methods since it does not require additional hold-out validation set to aggregate predictions of each agent, \n(4) the proposed approach is advantageous over existing methods since it does not have any restriction on training procedure and a trained model of each agent, \n(5) this paper theoretically proves the optimality of the proposed scheme (i.e., DeGroot consensus mechanism) is optimal under some conditions, and\n(6) this paper empirically demonstrates that the proposed approach is superior than the baseline approach (which does not weight the predictions of agents). ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:**\nThis paper introduces, for the first time, the known DeGroot consensus model in social science to address collective prediction at test time in the \"decentralized\" environment. PageRank can be considered as one application of the DeGroot consensus model, but it is not specifically used for collective predictions at test time. \n\nI think the key claim is that the proposed approach has its \"decentralization\" properties, e.g., each agent does not need to share its data and model to conclude the final, collective prediction via predictions from multiple agents. But, one concern in computing mutual scores in Eq (2) is that each agent actually needs to see other agents' local data (assuming models are not shared); in particular, Eq (2) means that the i-th agent computes the mutual scores of other agents by evaluating the j-th agent prediction over the i-th agent data, which implies the i-th agent data needs to be transferred to the j-th agent for the prediction. \n- Considering this, the paper's claim that each agent's data is not shared (e.g., line 35) could be overstated, and its unclear that the proposed approach addresses learning/inference issues in decentralized settings (since actually data sharing happens).\n- Assuming each agent need to transmit 1% of its data (as mentioned in line 611), i.e., |D_i(x')| = 0.01|D_i|. in Eq (2), each agent can obtain most of other agent's covariate data about 100 inference iterations. This is equivalent to having one agent all data without labels across agents, then what's the benefit of the proposed approach compared to conventional semi-supervised learning approaches, i.e., one agent just trains a model using its labeled data and unlabeled data from other agents, or the ensemble of agent's models, each of which is trained using semi-supervised learning?\n\n\n**Quality:**\nThis paper looks technically sound, and empirically well-justified on the importance of estimating \"weight\" of agent's predictions in aggregation.\n\n\n**Clarity:**\nThis paper is clearly-written. \n- line 609: is \"1 − 10%\" typo?\n\n**Significance:**\nI guess the DeGroot consensus idea in collective prediction could fertilize interesting ideas in other domains (e.g., classification or uncertainty estimation under distribution shift, where ensemble of models could be effective, but the models are combined with equal-weights).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "2sGJKrr71Hg", "reviewer": "Reviewer_9CMX", "summary": "This paper addresses test-time collective prediction. As far as I know, this is an underexplored topic, yet it is interesting. The paper proposes a decentralized consensus algorithm for multi-agent model aggregation. The paper includes both a nice theoretical analysis and empirical validation. Although I did not check the proofs, the theoretical analysis seems sound.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I tend to favor accepting this paper, so I am tentatively voting with a 7. However, I have some concerns below that I would like the authors to clarify.\n\nKEY CONCERN:\nI would like the authors to clarify one aspect of the proposed algorithm. It seems like the trust model relies on the agent's querying each other on a validation subset of their private data. How does this not violate the premise that the agents are not willing to share data: \"[the mechanism] does not require any data sharing, communication of model parameters, or labeled validation data.\"? \n\nWith respect to the above, I might recommend that the authors modify the protocol such that the collaborative agents use cryptographic tools like secure 2-party communication or FHE in order for agent i and agent j to agree on MSE_ij without agent i learning what agent j's model predicts on any particular data point in D_i(x') and without agent j learning what any particular data point in D_i(x') is. By only agreeing on MSE_ij, agent i and j are sharing the minimum amount of information necessary for the protocol to work. \n\nThe authors should either (1) clarify why this type of data sharing still is acceptable given their own premise,  (2) modify the protocol as described above (or any other way that achieves the objective laid out in their premise), or (3) clarify how I am misunderstanding the protocol if there is no data sharing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, included social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Strengths:
- Novel approach and application of the DeGroots consensus model
- Thorough theoretical analysis
- Effective empirical demonstration
- As stated by one reviewer: “well-written, well-executed”

Weaknesses (mild):
- Motivation for decentralized collection prediction could be improved
- The way in which privacy and data sharing properties of the approach are described

Summary:

Reviewers were unanimous in their view that this is a strong and complete submission. This was evident both in the original reviews as well as in correspondences with the authors. Authors are encouraged to discuss in the final version issues raised by the reviewers, including (i) how privacy and data sharing are considered, (ii) experimenting with classification (in addition to regression), and (iii) possible future relations to crypto and differential privacy.

Decision: Accept (Poster)